Hi everyone! My name is Bruno Messias currently I'm a Ph.D student at USP/Brazil. In this summer I'll develop new tools and features for FURY-GL Specifically, I'll focus into developing a system for collaborative visualization of large network layouts using FURY and VTK.
Before this commit was not possible to record the positions to have a
smooth animations with IPCLayout approach. See the animation bellow
After this PR now it's possible to tell Helios to store the evolution of the network positions using the record_positions parameter. This parameter should be passed on the start method. Notice in the image bellow how this gives to us a better visualization
Probably, I'll work more on Helios. Specifically I want to improve the memory management system. It seems that some shared memory resources are not been released when using the IPCLayout approach.
Hi all. In the past weeks, I’ve been focusing on developing Helios; the network
visualization library for FURY. I
improved the visual aspects of the network rendering as well as implemented the
most relevant network layout methods.
In this post I will discuss the most challenging task that I faced to implement those new network layout methods and
how I solved it.
The problem: network layout algorithm implementations with a blocking behavior
Case 1: Suppose that you need to monitor a hashtag and build a
social graph. You want to interact with
the graph and at the same time get insights about the structure of the user
interactions. To get those insights you can
perform a node embedding using any kind of network layout algorithm, such as
force-directed or minimum distortion
embeddings.
Case 2: Suppose that you are modelling a network dynamic such
as an epidemic spreading or a Kuramoto
model. In some of those network dynamics a node can change the state and the
edges related to the node must be deleted. For
example, in an epidemic model a node can represent a person who died due to a
disease. Consequently, the layout of the
network must be recomputed to give better insights.
In described cases if we want a better (UX) and at the same time a more practical
and insightful application of Helios
layouts algorithms shouldn’t block any kind of computation in the main thread.
In Helios we already have a lib written in C (with a python wrapper) which
performs the force-directed layout algorithm
using separated threads avoiding the GIL problem and consequently avoiding the
blocking. But and the other open-source
network layout libs available on the internet? Unfortunately, most of those libs
have not been implemented like Helios
force-directed methods and consequently, if we want to update the network layout
the python interpreter will block the
computation and user interaction in your network visualization. How to solve
this problem?
Why is using the python threading is not a good solution?
One solution to remove the blocking behavior of the network layout libs like
PyMDE is to use the threading module from
python. However, remember the GIL problem: only one thread can execute python
code at once. Therefore, this solution
will be unfeasible for networks with more than some hundreds of nodes or even
less! Ok, then how to solve it well?
IPC using python
As I said in my previous posts I’ve created a streaming system for data
visualization for FURY using webrtc. The streaming system is already working
and an important piece in this system was implemented using the python
SharedMemory from multiprocessing. We can get the same ideas from the streaming
system to remove the blocking behavior
of the network layout libs.
My solution to have PyMDE and CuGraph-ForceAtlas without blocking was to break
the network layout method into two
different types of processes: A and B.
The list below describes the most important behaviors and responsibilities for each process
Process A:
Where the visualization (NetworkDraw) will happen
Create the shared memory resources: edges, weights, positions, info..
Check if the process B has updated the shared memory resource which
stores the positions using the timestamp
stored in the info_buffer
Update the positions inside of NetworkDraw instance
Process B:
Read the network information stored in the shared memory resources:
edges , weights, positions
Execute the network layout algorithm
Update the positions values inside of the shared memory resource
Update the timestamp inside of the shared memory resource
I used the timestamp information to avoid unnecessary updates in the FURY/VTK
window instance, which can consume a lot
of computational resources.
How have I implemented the code for A and B?
Because we need to deal with a lot of different data and share them between
different processes I’ve created a set of
tools to deal with that, take a look for example in the
ShmManagerMultiArrays Object
,
which makes the memory management less
painful.
I'm breaking the layout method into two different processes. Thus I’ve
created two abstract objects to deal with
any kind of network layout algorithm which must be performed using
inter-process-communication (IPC). Those objects are:
NetworkLayoutIPCServerCalc
;
used by
processes of type B and
NetworkLayoutIPCRender
;
which
should be used by processes of type A.
I’ll not bore you with the details of the implementation. But let’s take a look
into some important points.
As I’ve said saving the timestamp after each step of the network layout
algorithm. Take a look into the method
_check_and_sync from NetworkLayoutIPCRender
here.
Notice that the update happens only if the stored timestamp has been changed.
Also, look at this line helios/layouts/mde.py#L180, the IPC-PyMDE implementation
This line writes a value 1 into the second element of the info_buffer. This value
is used to inform the process A that everything worked well. I used that info
for example in the tests for the network layout method, see the link
helios/tests/test_mde_layouts.py#L43
Results
Until now Helios has three network layout methods implemented: Force Directed ,
Minimum Distortion Embeddings and Force Atlas 2.
Here
docs/examples/viz_helios_mde.ipynb
you can get a jupyter notebook that I’ve a created showing how to use MDE with
IPC in Helios.
In the animation below we can see the result of the Helios-MDE application into
a network with a set of anchored nodes.
Next steps
I’ll probably focus on the Helios network visualization system. Improving the
documentation and testing the ForceAtlas2 in a computer with cuda installed. See
the list of opened
issues
Before the
8c670c2
commit, for some versions of MacOs the streaming system was
falling in a silent bug. I’ve spent a lot of time researching to find a
cause for this. Fortunately, I could found the cause and the solution.
This troublesome MacOs was falling in a silent bug because the
SharedMemory Object was creating a memory resource with at least 4086
bytes indepedent if I've requested less than that. If we look into the
MultiDimensionalBuffer Object (stream/tools.py) before the 8c670c2
commit we can see that Object has max_size parameter which needs to be updated
if the SharedMemory was created with a "wrong" size.
In the past week I've made a lot of improvements in this PR,
from performance improvements
to visual effects. Bellow are the list of the tasks
related with this PR:
- Code refactoring.
- Visual improvements: Using the UniformTools from my pull request
#424 now is
possible to control all the visual characteristics at runtime.
- 2D Layout: Meanwhile 3d network representations are very usefully for
exploring a dataset is hard to convice a group of network scientists to use a visualization
system which dosen't allow 2d representations. Because of that I started
to coding the 2d behavior in the network visualization system.
- Minimum Distortion Embeddings examples: I've created some examples
which shows how integrate pymde (Python Minimum Distortion Embeddings) with
fury/helios. The image bellow shows the result of this integration: a "perfect" graph embedding
What is coming up next week?
I'll probably focus on the
heliosPR#1. Specifically,
writing tests and improving the minimum distortion
embedding layout.
Post #2: SOLID, monkey patching a python issue and network layouts through WebRTC
demvessias
Published: 06/28/2021
Hi everyone! My name is Bruno Messias and I'm a PhD student working with graphs and networks. This summer I'll develop new tools and features for FURY-GL Specifically, I'll focus on developing a system for collaborative visualization of large network layouts using FURY and VTK.
These past two weeks I’ve spent most of my time in the
Streaming System PR
and the
Network Layout PR
In this post I’ll focus on the most relevant things I’ve made for those PRs
The past weeks I've spent some time refactoring the code to see what I’ve done let’ s take a look into this
fury/blob/b1e985.../fury/stream/client.py#L20, the FuryStreamClient Object before the refactoring.
The code is a mess.
To see why this code is not good according to SOLID principles let’s just list all the responsibilities of FuryStreamClient
Creates a RawArray or SharedMemory to store the n-buffers
Creates a RawArray or SharedMemory to store the information about each buffer
Cleanup the shared memory resources if the SharedMemory was used
Write the vtk buffer into the shared memory resource
Creates the vtk callbacks to update the vtk-buffer
That’s a lot and those responsibilities are not even related to each other. How can we be more SOLID[1]? An obvious solution is to create a specific object to deal with the shared memory resources. But it's not good enough because we still have a poor generalization since this new object still needs to deal with different memory management systems: rawarray or shared memory (maybe sockets in the future). Fortunately, we can use the python Abstract Classes[2] to organize the code.
To use the ABC from python I first listed all the behaviors that should be mandatory in the new abstract class. If we are using SharedMemory or RawArrays we need first to create the memory resource in a proper way. Therefore, the GenericImageBufferManager must have a abstract method create_mem_resource. Now take a look into the ImageBufferManager inside of stream/server/server.py, sometimes it is necessary to load the memory resource in a proper way. Because of that, the GenericImageBufferManager needs to have a load_mem_resource abstract method. Finally, each type of ImageBufferManager should have a different cleanup method. The code below presents the sketch of the abstract class
Now we can look for those behaviors inside of FuryStreamClient.py and ImageBufferManger.py that does not depend if we are using the SharedMemory or RawArrays. These behaviors should be methods inside of the new GenericImageBufferManager.
With the
GenericImageBufferManager the
RawArrayImageBufferManager
and SharedMemImageBufferManager is now implemented with less duplication of code (DRY principle). This makes the code more readable and easier to find bugs. In addition, later we can implement other memory management systems in the streaming system without modifying the behavior of FuryStreamClient or the code inside of server.py.
I’ve also applied the same SOLID principles to improve the CircularQueue object. Although the CircularQueue and FuryStreamInteraction was not violating the S from SOLID the head-tail buffer from the CircularQueue must have a way to lock the write/read if the memory resource is busy. Meanwhile the multiprocessing.Arrays already has a context which allows lock (.get_lock()) SharedMemory dosen’t[2]. The use of abstract class allowed me to deal with those peculiarities.
commit 358402e
Using namedtuples to grant immutability and to avoid silent bugs
The circular queue and the user interaction are implemented in the streaming system using numbers to identify the type of event (mouse click, mouse weel, ...) and where to store the specific values associated with the event , for example if the ctrl key is pressed or not. Therefore, those numbers appear in different files and locations: tests/test_stream.py, stream/client.py, steam/server/app_async.py. This can be problematic because a typo can create a silent bug. One possibility to mitigate this is to use a python dictionary to store the constant values, for example
But this solution has another issue, anywhere in the code we can change the values of EVENT_IDS and this will produce a new silent bug. To avoid this I chose to use namedtuples to create an immutable object which holds all the constant values associated with the user interactions. stream/constants.py
The namedtuple has several advantages when compared to dictionaries for this specific situation. In addition, it has a better performance. A good tutorial about namedtuples it’s available here
https://realpython.com/python-namedtuple/
Testing
My mentors asked me to write tests for this PR. Therefore, this past week I’ve implemented the most important tests for the streaming system:
/fury/tests/test_stream.py
Most relevant bugs
As I discussed in my third week check-in there is an open issue related to SharedMemory in python. This"bug" happens in the streaming system through the following scenario
1-Process A creates a shared memory X
2-Process A creates a subprocess B using popen (shell=False)
3-Process B reads X
4-Process B closes X
5-Process A kills B
4-Process A closes X
5-Process A unlink() the shared memory resource
In python, this scenario translates to
from multiprocessing import shared_memory as sh
import time
import subprocess
import sys
shm_a = sh.SharedMemory(create=True, size=10000)
command_string = f"from multiprocessing import shared_memory as sh;import time;shm_b = sh.SharedMemory('{shm_a.name}');shm_b.close();"
time.sleep(2)
p = subprocess.Popen(
[sys.executable, '-c', command_string],
stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=False)
p.wait()
print("\nSTDOUT")
print("=======\n")
print(p.stdout.read())
print("\nSTDERR")
print("=======\n")
print(p.stderr.read())
print("========\n")
time.sleep(2)
shm_a.close()
shm_a.unlink()
Fortunately, I could use a monkey-patching[3] solution to fix that; meanwhile we're waiting for the python-core team to fix the resource_tracker (38119) issue [4].
Finally, the first version of FURY network layout is working as can you see in the video below
In addition, this already can be used with the streaming system allowing user interactions across the internet with WebRTC protocol.
One of the issues that I had to solve to achieve the result presented in the video above was to find a way to update the positions of the vtk objects without blocking the main thread and at the same time allowing the vtk events calls. My solution was to define an interval timer using the python threading module:
/fury/stream/tools.py#L776,
/fury/stream/client.py#L112/fury/stream/client.py#L296
Refs:
[1] A. Souly,"5 Principles to write SOLID Code (examples in Python)," Medium, Apr. 26, 2021. https://towardsdatascience.com/5-principles-to-write-solid-code-examples-in-python-9062272e6bdc (accessed Jun. 28, 2021).
[2]"[Python-ideas] Re: How to prevent shared memory from being corrupted ?" https://www.mail-archive.com/python-ideas@python.org/msg22935.html (accessed Jun. 28, 2021).
Hi everyone! My name is Bruno Messias. In this summer I'll develop new tools and features for FURY-GL Specifically, I'll focus into developing a system for collaborative visualization of large network layouts using FURY and VTK.
PR fury-gl/fury#432
I've made some improvements in my PR which can be used to fine tuning the opengl state on VTK
PR fury-gl/fury#437
I've made several improvements in my streamer proposal for FURY. most of those improvements
it's related with memory management. Using the SharedMemory from python 3.8 now it's possible to
use the streamer direct on a jupyter without blocking
The SharedMemory from python>=3.8 offers new a way to share memory resources between unrelated process. One of the advantages of using the SharedMemory instead of the RawArray from multiprocessing it’s that the SharedMemory allows to share memory blocks without those processes be related with a fork or spawm method. The SharedMemory behavior allowed to achieve our jupyter integration
and
simplifies the use of the streaming system. However, I saw a issue in the shared memory implementation.
Let’s see the following scenario:
1-Process A creates a shared memory X
2-Process A creates a subprocess B using popen (shell=False)
3-Process B reads X
4-Process B closes X
5-Process A kills B
4-Process A closes X
5-Process A unlink() the shared memory resource X
This scenario should work well. unlink() X in it's the
right way as discussed in the python official documentation. However, there is a open issue
which a think it's related with the above scenario.
