Articles on joaodellagli's Blog

Google Summer of Code Final Work Product

joaovdafloriano@usp.br (joaodellagli) — Thu, 31 Aug 2023 14:40:56 +0000

Google Summer of Code Final Work Product

Name: João Victor Dell Agli Floriano
Organisation: Python Software Foundation
Sub-Organisation: FURY
Project: FURY - Project 2. Fast 3D kernel-based density rendering using billboards.

Abstract

This project had the goal to implement a 3D Kernel Density Estimation rendering to Fury. Kernel Density Estimation, or KDE, is a statistical method that uses kernel smoothing for modeling and estimating the density distribution of a set of points defined inside a given region. For its graphical implementation, it was used post-processing techniques such as offscreen rendering to framebuffers and colormap post-processing as tools to achieve the desired results. This was completed with a functional basic KDE rendering result, that relies on a solid and easy-to-use API, as well as some additional features.

Proposed Objectives

First Phase : Implement framebuffer usage in FURY
- Investigate the usage of float framebuffers inside FURY's environment.
- Implement a float framebuffer API.
Second Phase : Shader-framebuffer integration
- Implement a shader that uses a colormap to render framebuffers.
- Escalate this rendering for composing multiple framebuffers.
Third Phase : KDE Calculations
- Investigate KDE calculation for point-cloud datasets.
- Implement KDE calculation inside the framebuffer rendering shaders.
- Test KDE for multiple datasets.

Objectives Completed

Implement framebuffer usage in FURY

The first phase, addressed from May/29 to July/07, started with the investigation of VTK's Frambuffer Object, a vital part of this project, to understand how to use it properly.

Framebuffer Objects, abbreviated as FBOs, are the key to post-processing effects in OpenGL, as they are used to render things offscreen and save the resulting image to a texture that will be later used to apply the desired post-processing effects within the object's fragment shader rendered to screen, in this case, a billboard. In the case of the Kernel Density Estimation post-processing effect, we need a special kind of FBO, one that stores textures' values as floats, different from the standard 8-bit unsigned int storage. This is necessary because the KDE rendering involves rendering every KDE point calculation to separate billboards, rendered to the same scene, which will have their intensities, divided by the number of points rendered, blended with OpenGL Additive Blending, and if a relative big number of points are rendered at the same time, 32-bit float precision is needed to guarantee that small-intensity values will not be capped to zero, and disappear.

After a month going through VTK's FBO documentation and weeks spent trying different approaches to this method, it would not work properly, as some details seemed to be missing from the documentation, and asking the community haven't solved the problem as well. Reporting that to my mentors, which unsuccessfully tried themselves to make it work, they decided it was better if another path was taken, as the 32-bit float FBO would result in a much more complicated pipeline, using VTK's WindowToImageFilter method as a workaround, described in this blogpost. This method helped the development of three new functions to Fury, window_to_texture(), texture_to_actor() and colormap_to_texture(), that allow the passing of different kinds of textures to Fury's actor's shaders, the first one to capture a window and pass it as a texture to an actor, the second one to pass an external texture to an actor, and the third one to specifically pass a colormap as a texture to an actor. It is important to say that WindowToImageFilter() is not the ideal way to make it work, as this method does not seem to support float textures. However, a workaround to that is currently being worked on, as I will describe later on.

Pull Requests:
- KDE Rendering Experimental Program (needs major revision): fury-gl/fury#804
  
  The result of this whole FBO and WindowToImageFilter experimentation is well documented in PR #804 that implements an experimental version of a KDE rendering program. The future of this PR, as discussed with my mentors, is to be better documented to be used as an example for developers on how to develop features in Fury with the tools used, and it shall be done soon.
Shader-framebuffer integration

The second phase, which initially was thought of as "Implement a shader that uses a colormap to render framebuffers" and "Escalate this rendering for composing multiple framebuffers" was actually a pretty simple phase that could be addressed in one week, July/10 to July/17, done at the same time as the third phase goal, documented in this blogpost. As Fury already had a tool for generating and using colormaps, they were simply connected to the shader part of the program as textures, with the functions explained above. Below, is the result of the matplotlib viridis colormap passed to a simple gaussian KDE render:

That is also included in PR #804. Having the 2D plot ready, some time was taken to figure out how to enable a 3D render, that includes rotation and other movement around the set rendered, which was solved by learning about the callback properties that exist inside VTK. Callbacks are ways to enable code execution inside the VTK rendering loop, enclosed inside vtkRenderWindowInteractor.start(). If it is desired to add a piece of code that, for example, passes a time variable to the fragment shader over time, a callback function can be declared:
```
from fury import window
t = 0
showm = window.ShowManager(...)

def callback_function:
    t += 0.01
    pass_shader_uniforms_to_fs(t, "t")

showm.add_iren_callback(callback_function, "RenderEvent")
```
The piece of code above created a function that updates the time variable t in every "RenderEvent", and passes it to the fragment shader. With that property, the camera and some other parameters could be updated, which enabled 3D visualization, that then, outputted the following result, using matplotlib inferno colormap:
KDE Calculations (ongoing)

As said before, the second and third phases were done simultaneously, so after having a way to capture the window and use it as a texture ready, the colormap ready, and an initial KDE render ready, all it was needed to do was to improve the KDE calculations. As this Wikipedia page explains, a KDE calculation is to estimate an abstract density around a set of points defined inside a given region with a kernel, that is a function that models the density around a point based on its associated distribution sigma.

A well-known kernel is, for example, the Gaussian Kernel, that says that the density around a point p with distribution sigma is defined as:

Using that kernel, we can calculate the KDE of a set of points A with associated distributions S calculating their individual Gaussian distributions, summing them up and dividing them by the total number of points p:

So I dove into implementing all of that into the offscreen rendering part, and that is when the lack of a float framebuffer would charge its cost. As it can be seen above, just calculating each point's density isn't the whole part, as I also need to divide everyone by the total number of points p, and then sum them all. The problem is that, if the number of points its big enough, the individual densities will be really low, and that would not be a problem for a 32-bit precision float framebuffer, but that is definitely a problem for a 8-bit integer framebuffer, as small enough values will simply underflow and disappear. That issue is currently under investigation, and some solutions have already been presented, as I will show in the Objectives in Progress section.

Apart from that, after having the experimental program ready, I focused on modularizing it into a functional and simple API (without the n division for now), and I could get a good set of results from that. The API I first developed implemented the EffectManager class, responsible for managing all of the behind-the-scenes steps necessary for the kde render to work, encapsulated inside the ÈffectManager.kde() method. It had the following look:
```
from fury.effect_manager import EffectManager
from fury import window

showm = window.ShowManager(...)

# KDE rendering setup
em = EffectManager(showm)
kde_actor = em.kde(...)
# End of KDE rendering setup

showmn.scene.add(kde_actor)

showm.start()
```
Those straightforward instructions, that hid several lines of code and setup, could manage to output the following result:

And this was not the only feature I had implemented for this API, as the use of WindowToImageFilter method opened doors for a whole new world for Fury: The world of post-processing effects. With this features setup, I managed to implement a Gaussian blur effect, a grayscale effect and a Laplacian effect for calculating "borders":

As this wasn't the initial goal of the project and I still had several issues to deal with, I have decided to leave these features as a future addition.

Talking with my mentors, we realized that the first KDE API, even though simple, could lead to bad usage from users, as the em.kde() method, that outputted a Fury actor, had dependencies different from any other object of its kind, making it a new class of actors, which could lead to confusion and bad handling. After some pair programming sessions, they instructed me to take a similar, but different road from what I was doing, turning the kde actor into a new class, the KDE class. This class would have almost the same set of instructions present in the prior method, but it would break them in a way it would only be completely set up after being passed to the EffectManager via its add function. Below, how the refactoring handles it:
```
from fury.effects import EffectManager, KDE
from fury import window

showm = window.ShowManager(...)

# KDE rendering setup
em = EffectManager(showm)
kde_effect = KDE(...)
em.add(kde_effect)
# End of KDE rendering setup

showm.start()
```
Which outputted the same results as shown above. It may have cost some simplicity as we are now one line farther from having it working, but it is more explicit in telling the user this is not just a normal actor.

Another detail I worked on was the kernel variety. The Gaussian Kernel isn't the only one available to model density distributions, there are several others that can do that job, as it can be seen in this scikit-learn piece of documentation and this Wikipedia page on kernels. Based on the scikit-learn KDE implementation, I worked on implementing the following kernels inside our API, that can be chosen as a parameter when calling the KDE class:
- Cosine
- Epanechnikov
- Exponential
- Gaussian
- Linear
- Tophat
Below, the comparison between them using the same set of points and bandwidths:

Pull Requests:
- First Stage of the KDE Rendering API (will be merged soon) fury-gl/fury#826
  
  All of this work culminated in PR #826, that proposes to add the first stage of this API (there are some details yet to be completed, like the n division) to Fury. This PR added the described API, and also proposed some minor changes to some already existing Fury functions related to callbacks, changes necessary for this and other future applications that would use it to work. It also added the six kernels described, and a simple documented example on how to use this feature.

Other Objectives

Stretch Goals : SDE Implementation, Network/Graph visualization using SDE/KDE, Tutorials
- Investigate SDE calculation for surface datasets.
- Implement SDE calculation inside the framebuffer rendering shaders.
- Test SDE for multiple datasets.
- Develop comprehensive tutorials that explain SDE concepts and FURY API usage.
- Create practical, scenario-based tutorials using real datasets and/or simulations.

Objectives in Progress

KDE Calculations (ongoing)

The KDE rendering, even though almost complete, have the n division, an important step, missing, as this normalization allows colormaps to cover the whole range o values rendered. The lack of a float FBO made a big difference in the project, as the search for a functional implementation of it not only delayed the project, but it is vital for the correct calculations to work.

For the last part, a workaround thought was to try an approach I later figured out is an old one, as it can be check in GPU Gems 12.3.3 section: If I need 32-bit float precision and I got 4 8-bit integer precision available, why not trying to pack this float into this RGBA texture? I have first tried to do one myself, but it didn't work for some reason, so I tried Aras Pranckevičius implementation, that does the following:
```
vec4 float_to_rgba(float value) {
   vec4 bitEnc = vec4(1.,256.,65536.0,16777216.0);
   vec4 enc = bitEnc * value;
   enc = fract(enc);
   enc -= enc.yzww * vec2(1./255., 0.).xxxy;
   return enc;
}
```
That initially worked, but for some reason I am still trying to understand, it is resulting in a really noisy texture:

One way to try to mitigate that while is to pass this by a gaussian blur filter, to try to smooth out the result:

But it is not an ideal solution as well, as it may lead to distortions in the actual density values, depending on the application of the KDE. Now, my goal is to first find the root of the noise problem, and then, if that does not work, try to make the gaussian filter work.

Another detail that would be a good addition to the API is UI controls. Filipi, one of my mentors, told me it would be a good feature if the user could control the intensities of the bandwidths for a better structural visualization of the render, and knowing Fury already have a good set of UI elements, I just neeeded to integrate that into my program via callbacks. I tried implementing an intensity slider. However, for some reason, it is making the program crash randomly, for reasons I still don't know, so that is another issue under investigation. Below, we show a first version of that feature, which was working before the crashes:

Pull Requests:
- UI intensity slider for the KDE rendering API (draft): fury-gl/fury#849
- Post-processing effects for FURY Effects API (draft): fury-gl/fury#850

GSoC Weekly Blogs

My blog posts can be found at FURY website and Python GSoC blog.

Timeline

Date	Description	Blog Link
Week 0 (29-05-2023)	The Beginning of Everything	FURY - Python
Week 1 (05-06-2022)	The FBO Saga	FURY - Python
Week 2 (12-06-2022)	The Importance of (good) Documentation	FURY - Python
Week 3 (19-06-2022)	Watch Your Expectations	FURY - Python
Week 4 (26-06-2022)	Nothing is Ever Lost	FURY - Python
Week 5 (03-07-2022)	All Roads Lead to Rome	FURY - Python
Week 6 (10-07-2022)	Things are Starting to Build Up	FURY - Python
Week 7 (17-07-2022)	Experimentation Done	FURY - Python
Week 8 (24-07-2022)	The Birth of a Versatile API	FURY - Python
Week 9 (31-07-2022)	It is Polishing Time!	FURY - Python
Week 10 (07-08-2022)	Ready for Review!	FURY - Python
Week 11 (14-08-2022)	A Refactor is Sometimes Needed	FURY - Python
Week 12 (21-08-2022)	Now That is (almost) a Wrap!	FURY - Python

Week 12: Now That is (almost) a Wrap!

joaovdafloriano@usp.br (joaodellagli) — Wed, 23 Aug 2023 23:36:19 +0000

Hello everyone, it's time for another GSoC blogpost! Today, I am going to talk about some minor details I worked on my project.

Last Week's Effort

After the API refactoring done last week, I focused on addressing the reviews I would get from it. The first issues I addressed was related to style, as there were some minor details my GSoC contributors pointed out that needed change. Also, I have addressed an issue I was having with the `typed hint` of one of my functions. Filipi, my mentor, showed me there is a way to have more than one typed hint in the same parameter, all I needed to do was to use the Union class from the typing module, as shown below:

from typing import Union as tUnion
from numpy import ndarray
 
def function(variable : tUnion(float, np.ndarray)):
    pass

Using that, I could set the typedhint of the bandwidth variable to float and np.ndarray.

So how did it go?

All went fine with no difficult at all, thankfully.

The Next Steps

My next plans are, after having PR #826 merged, to work on the float encoding issue described in this blogpost. Also, I plan to tackle the UI idea once again, to see if I can finally give the user a way to control de intensities of the distributions.

Wish me luck!

A Refactor is Sometimes Needed

joaovdafloriano@usp.br (joaodellagli) — Sat, 19 Aug 2023 00:50:22 +0000

Hello everyone, it's time for another weekly blogpost! Today I am going to share some updates on the API refactoring I was working on with my mentors.

Last Week's Effort

As I shared with you last week, the first draft of my API was finally ready for review, as I finished tweaking some remaining details missing. I was tasked with finding a goode example of usage of the tools we proposed, and I started to do that, however after testing it with some examples, I figured out some significant bugs were to be fixed. Also, after some reviews and hints from some of my mentors and other GSoC contributors, we realised that some refactoring should be done, mainly focused on avoiding bad API usage from the user.

So how did it go?

Initially, I thought only one bug was the source of the issues the rendering presented, but it turned out to be two, which I will explain further.

The first bug was related to scaling and misalignment of the KDE render. The render of the points being post-processed was not only with sizes different from the original set size, but it was also misaligned, making it appear in positions different from the points' original ones. After some time spent, I figured out the bug was related to the texture coordinates I was using. Before, this is how my fragment shader looked:

vec2 res_factor = vec2(res.y/res.x, 1.0);
vec2 tex_coords = res_factor*normalizedVertexMCVSOutput.xy*0.5 + 0.5;
float intensity = texture(screenTexture, tex_coords).r;

It turns out using this texture coordinates for *this case* was not the best choice, as even though it matches the fragment positions, the idea here was to render the offscreen window, which has the same size as the onscreen one, to the billboard actor. With that in mind, I realised the best choice was using texture coordinates that matched the whole screen positions, coordinates that were derived from the gl_FragCoord.xy, being the division of that by the resolution of the screen, for normalization. Below, the change made:

vec2 tex_coords = gl_FragCoord.xy/res;
float intensity = texture(screenTexture, tex_coords).r;

This change worked initially, although with some problems, that later revealed the resolution of the offscreen window needed to be updated inside the callback function as well. Fixing that, it was perfectly aligned and scaled!

The second bug was related with the handling of the bandwidth, former sigma parameter. I realised I wasn't dealing properly with the option of the user passing only one single bandiwidth value being passed, so when trying that, only the first point was being rendered. I also fixed that and it worked, so cheers!

As I previously said, the bugs were not the only details I spent my time on last week. Being reviewed, the API design, even though simple, showed itself vulnerable to bad usage from the user side, requiring some changes. The changes suggested by mentors was, to, basically, take the kde method out of the EffectManager class, and create a new class from it inside an effects module, like it was a special effects class. With this change, the KDE setup would go from:

em = EffectManager(show_manager)

kde_actor = em.kde(...)

show_manager.scene.add(kde_actor)

To:

em = EffectManager(show_manager)

kde_effect = KDE(...)

em.add(kde_effect)

Not a gain in line shortening, however, a gain in security, as preventing users from misusing the kde_actor. Something worth noting is that I learned how to use the functools.partial function, that allowed me to partially call the callback function with only some parameters passed.

This Week's Goals

Having that refactoring made, now I am awaiting for a second review so we could finally wrap it up and merge the first stage of this API. With that being done, I will write the final report and wrap this all up.

Let's get to work!

Ready for Review!

joaovdafloriano@usp.br (joaodellagli) — Thu, 10 Aug 2023 14:08:14 +0000

Hello everyone, it's time for another weekly blogpost!

Last Week's Effort

After talking with my mentors, I was tasked with getting my API PR #826 ready for review, as it still needed some polishing, and the most important of all, it needed its tests working, as this was something I haven't invested time since its creation. Having that in mind, I have spent the whole week cleaning whatever needed, writing the tests, and also writing a simple example of its usage. I also tried implementing a little piece of UI so the user could control the intensity of the bandwidth of the KDE render, but I had a little problem I will talk about below.

So how did it go?

Fortunately, for the cleaning part, I didn't have any trouble, and my PR is finally ready for review! The most complicated part was to write the tests, as this is something that requires attention to understand what needs to be tested, exactly. As for the UI part, I managed to have a slider working for the intensity, however, it was crashing the whole program for a reason, so I decided to leave this idea behind for now. Below, an example of how this should work:

This Week's Goals

After a meeting with my mentors, we decided that this week's focus should be on finding a good usage example of the KDE rendering feature, to have it as a showcase of the capability of this API. Also, they hinted me some changes that need to be done regarding the API, so I will also invest some time on refactoring it.

Wish me luck!

Week 9: It is Polishing Time!

joaovdafloriano@usp.br (joaodellagli) — Wed, 02 Aug 2023 21:00:50 +0000

Hello everyone, it's time for another weekly blogpost! Today, I am going to update you on my project's latest changes.

Last Week's Effort

After having finished a first draft of the API that will be used for the KDE rendering, and showing how it could be used for other post-processing effects, my goal was to clean the code and try some details that would add to it so it could be better complete. Having that in mind, I invested in three work fronts:

Fixing some bugs related to the rendering more than one post-processing effect actor.
Experimenting with other rendering kernels (I was using the *gaussian* one only).
Completing the KDE render by renormalizing the values in relation to the number of points (one of the core KDE details).

Both three turned out more complicated than it initially seemed, as I will show below.

So how did it go?

The first one I did on monday-tuesday, and I had to deal with some issues regarding scaling and repositioning. Due to implementation choices, the final post-processed effects were rendered whether bigger than they were in reality, or out of their original place. After some time dedicated to finding the root of those problems, I could fix the scaling issue, however I realised I would need to,probably, rethink the way the api was implemented. As this general post-processing effects is a side-project that comes as a consequence of my main one, I decided to leave that investment to another time, as I would need to guarantee the quality of the second.

The second was an easy and rather interesting part of my week, as I just needed to setup new kernel shaders. Based on scikit-learn KDE documentation, I could successfully implement the following kernels:

Gaussian
Tophat
Epanechnikov
Exponential
Linear
Cosine

That outputted the following (beautiful) results for a set of 1000 random points with random sigmas:

The third one is still being a trickier challenge. If you recall from my first blogposts, I spent something around one month trying to setup float framebuffer objects to FURY with VTK so I could use them in my project. After spending all of that time with no results, me and Bruno, my mentor, found a way to do what we wanted to do, but using a different VTK class, vtkWindowToImageFilter. Well, it was a good workaround back then and it lead me all the way here, however now it is costing a price. The float framebuffers were an important part of the project because they would allow us to pass *2-bit float information from one shader to another, which would be important as they would allow the densities to have higher precision and more fidelity to the calculations. When rendering a KDE of a given set of points, we use the below function:

KDE(x, y) = n^-1sum( kernel(x, y) )

If the number of points n is big enough, some KDE results will be really low. This presents a real problem to our implementation because, without the float framebuffers, it is currently only being possible to pass 8-bit unsigned char information, that only allows 256 values. This is far from ideal, as low values would have alone densities low enough to disappear. This presented a problem as to renormalize the densities, I was retrieving the texture to the CPU, calculating its minimum and maximum values, and passing to the fragment shader as uniformsfor the renormalization, which didn't work if the maximum values calculated were zero.

One solution I thought to solve that was a really heavy workaround: if an unsigned float is 32-bit and I have exactly 4 8-bit unsigned chars, why not try to pack this float into these 4 chars? Well, this is an interesting approach which I figured out is already an old one, being reported in GPU Gems's chapter 12. Unfortunately I haven't tried yet this implementation yet, and went for one I thought myself, which haven't exactly worked. I also tried this implementation from Aras Pranckevičius' website, which seems to be working, even though not perfectly:

That looks way better, even though not ideal yet.

This Week's Goals

Talking with my mentors, we decided it was better if I focused on the version without the renormalization for now, as it was already done and running fine. So for this week, I plan to clean my PR to finally have it ready for a first review, and maybe add to it a little UI tool to control the intensity of the densities. That should take me some time and discussion, but I hope for it to be ready by the end of the week.

Let's get to work!

Week 8: The Birth of a Versatile API

joaovdafloriano@usp.br (joaodellagli) — Tue, 25 Jul 2023 15:12:49 +0000

Hello everyone, it's time for another weekly blogpost! Today, I am going to tell you all about how is the KDE API development going, and to show you the potential this holds for the future!

Last Week's Effort

Last week I told you how I managed to render some KDE renders to the screen, both in 2D and 3D, as you may check by my last blogpost. My new task was, as I had this example working, to start the API development. I;n a meeting with Bruno, one of my mentors, we debated on how could this work, reaching two options:

Implement the KDE in a single, simple actor.
Implement a KDE rendering manager, as a class.

The first one would have the advantage of being simple and pretty straightforward, as a user would only need to call the actor and have it working on them hands, having the tradeoff of leaving some important steps for a clean API hidden and static. These steps I mention are related to how this rendering works, as I have previously showed you, it relies on post-processing effects, which need an offscreen rendering, that for example are done by the callback functions.

In short, these functions are instructions the user gives to the interactor to run inside the interaction loop. Inside FURY there are tree types of callbacks passed to the windoe interactor:

Timer Callbacks: Added to the window interactor, they are a set of instructions that will be called from time to time, with interval defined by the user.
Window Callbacks: Added directly to the window, they are a set of instructions called whenever an specific event is triggered.
Interactor Callbacks: Added to the window interactor, they are a set of instructions called whenever an specific interaction, for example a mouse left-click, is triggered.

In this API, I will be using the Interactor Callback, set by the window.add_iren_callback() function, that will be called whenever a Render interaction is detected, and needs to be first passed to the onscreen manager.

These details are more complicated, and would need, for example, for the user to pass the onscreen manager to the actor.kde() function. Also, in the case of a kde actor not being used anymore and being declared, the callback then passed would still exist inside the manager and be called even when the kde actor is not on screen anymore, which is not ideal.

Knowing these problems, we thought of second option, that would have the advantage of not leaving those details and steps behind. It has the tradeoff of maybe complicating things as it would need to be called after calling the effects manager, but as I will show you below, it is not that complicated at all.

I also reviewed my fellow GSoC contributors PR's as well, PR #810 and #803. Bruno told me to take a look as well on Conventional Commits, a way to standardtize commits by prefixes, so I did that as well.

So how did it go?

Well, the implemented manager class is named EffectManager() and to initialize it you only need to pass the onscreen manager:

effects = EffectManager(onscreen_manager)

After that, to render a KDE calculation of points to the screen, you need only to call its kde() function:

kde_actor = effects.kde(center, points, sigmas, scale = 10.0, colormap = "inferno")
# Those last two are optional

Pass it to the onscreen manager scene:

onscreen_manager.scene.add(kde_actor)

And to start it, as usual:

onscreen_manager.start()

As simple as that. This three lines of code output the same result as I showed you last week, this time, with different sigmas for each point:

After having that working, I experimented beyond. See, as I previously said, we are dealing here with post-processing effects, with KDE being only one of the many existing ones, as this Wikipedia Page on post processing shows. Knowing that, I tried one of the first filters I learned, the Laplacian one. This filter is, as its name hints, applying the Discrete Laplace Operator in an image. This filter shows suddend changes of value, a good way to detect borders. The process is the same as the kde actor, requiring only the actor you want to apply the filter to. Below, the result I got from applying that to a box actor:

Something I found important to leave as an option was filter compositing. What if an user wanted to, for example, apply one laplacian filter after another? Well, the example below shows that is possible as well:

It still need some tweaks and suffers from some bugs, but it works! Those represent important progress as it shows the versatility this API may present. I have also already implemented grayscale and 3x3 gaussian blur as well:

This Week's Goals

My plans for this week are to keep working and polishing the API, mainly the KDE part, so it can be ready for a first review. When that is ready, I plan to experiment with more filters and make this more dynamic, maybe implementing a way to apply custom kernel transformations, passed by the user, to the rendering process. This has been a really exciting journey and I am getting happy with the results!

Wish me luck!

Week 7: Experimentation Done

joaovdafloriano@usp.br (joaodellagli) — Tue, 18 Jul 2023 18:41:23 +0000

Hello everyone, welcome to another weekly blogpost! Let's talk about the current status of my project (spoiler: it is beautiful).

Last Week's Effort

Having accomplished a KDE rendering to a billboard last week, I was then tasked with trying a different approach to how the rendering was done. So, to recap, below was how I was doing it:

Render one point's KDE offscreen to a single billboard, passing its position and sigma to the fragment shader as uniforms.
Capture the last rendering's screen as a texture.
Render the next point's KDE, and sum it up with the last rendering's texture.
Do this until the end of the points.
Capture the final render screen as a texture.
Apply post processing effects (colormapping).
Render the result to the screen.

This approach was good, but it had some later limitations and issues that would probably take more processing time and attention to details (correct matrix transformations, etc) than the ideal. The different idea is pretty similar, but with some differences:

Activate additive blending in OpenGL.
Render each point's KDE to its own billboard, with position defined by the point's position, all together in one pass.
Capture the rendered screen as a texture.
Pass this texture to a billboard.
Apply post processing effects (colormapping).
Render the result to the screen.

So I needed to basically do that.

Was it Hard?

Fortunately, it wasn't so hard to do it in the end. Following those steps turned out pretty smooth, and after some days, I had the below result:

This is a 2D KDE render of random 1000 points. For this I used the *"viridis"* colormap from `matplotlib`. Some details worth noting:

For this to work, I have implemented three texture helper functions: `window_to_texture()`, `texture_to_actor()` and `colormap_to_texture()`. The first one captures a window and pass it as a texture to an actor, the second one passes an imported texture to an actor, and the last one passes a colormap, prior passed as an array, as a texture to an actor.
The colormap is directly get from `matplotlib`, available in its `colormaps` object.
This was only a 2D flatten plot. At first, I could not figure out how to make the connection between the offscreen interactor and the onscreen one, so rotating and moving around the render was not happening. After some ponder and talk to my mentors, they told me to use *callback* functions inside the interactor, and after doing that, I managed to make the 3D render work, which had the following result:

After those results, I refactored my PR #804 to better fit its current status, and it is now ready for review. Success!

This Week's Goals

After finishing the first iteration of my experimental program, the next step is to work on an API for KDE rendering. I plan to meet with my mentors and talk about the details of this API, so expect an update next week. Also, I plan to take a better look on my fellow GSoC FURY contributors work so when their PRs are ready for review, I will have be better prepared for it.

Let's get to work!

Week 6: Things are Starting to Build Up

joaovdafloriano@usp.br (joaodellagli) — Wed, 12 Jul 2023 14:06:06 +0000

Hello everyone, time for a other weekly blogpost! Today, I will show you my current progress on my project and latest activities.

What I did Last Week

Last week I had the goal to implement KDE rendering to the screen (if you want to understand what this is, check my last blogpost. After some days diving into the code, I finally managed to do it:

This render may seem clean and working, but the code isn't exactly like that. For this to work, some tricks and work arounds needed to be done, as I will describe in the section below.

Also, I reviewed the shader part of Tania's PR #791, that implement ellipsoid actors inside FURY. It was my first review of a PR that isn't a blogpost, so it was an interesting experience and I hope I can get better at it.

It is important as well to point out that I had to dedicate myself to finishing my graduation capstone project's presentation that I will attend to this week, so I had limited time to polish my code, which I plan to do better this week.

Where the Problem Was

The KDE render basically works rendering the KDE of a point to a texture and summing that texture to the next render. For this to work, the texture, rendered to a billboard, needs to be the same size of the screen, otherwise the captured texture will include the black background. The problem I faced with that is that the billboard scaling isn't exactly well defined, so I had to guess for a fixed screen size (in this example, I worked with 600x600) what scaling value made the billboard fit exactly inside the screen (it's 3.4). That is far from ideal as I will need to modularize this behavior inside a function that needs to work for every case, so I will need to figure out a way to fix that for every screen size. For that, I have two options:

Find the scaling factor function that makes the billboard fit into any screen size.
Figure out how the scaling works inside the billboard actor to understand if it needs to be refactored.

The first seems ok to do, but it is kind of a work around as well. The second one is a good general solution, but it is a more delicate one, as it deals with how the billboard works and already existing applications of it may suffer problems if the scaling is changed. I will see what is better talking with my mentors.

Another problem I faced (that is already fixed) relied on shaders. I didn't fully understood how shaders work inside FURY so I was using my own fragment shader implementation, replacing the already existing one completely. That was working, but I was having an issue with the texture coordinates of the rendering texture. As I completely replaced the fragment shader, I had to pass custom texture coordinates to it, resulting in distorted textures that ruined the calculations. Those issues motivated me to learn the shaders API, which allowed me to use the right texture coordinates and finally render the results you see above.

This Week's Goals

For this week, I plan to try a different approach Filipi, one of my mentors, told me to do. This approach was supposed to be the original one, but a communication failure lead to this path I am currently in. This approach renders each KDE calculation into its own billboard, and those are rendered together with additive blending. After this first pass, this render is captured into a texture and then rendered to another big billboard.

Also, I plan to refactor my draft PR #804 to make it more understandable, as its description still dates back to the time I was using the flawed Framebuffer implementation, and my fellow GSoC contributors will eventually review it, and to do so, they will need to understand it.

Wish me luck!

Week 5: All Roads Lead to Rome

joaovdafloriano@usp.br (joaodellagli) — Tue, 04 Jul 2023 21:00:28 +0000

Hello everyone, time for another weekly blogpost! Today, we will talk about taking different paths to reach your objective.

Last Week's Effort

After having the FBO properly set up, the plan was to finally *render* something to it. Well, I wished for a less bumpier road at my last blogpost but as in this project things apparently tend to go wrong, of course the same happened with this step.

Where the Problem Was

Days passed without anything being rendered to the FBO. The setup I was working on followed the simplest OpenGL pipeline of rendering to

an FBO:

1. Setup the FBO

2. Attach a texture to its color attachment

3. Setup the shader to be used in the FBO render and the shader to render the FBO's Color Attachment

4. Render to the FBO

5. Use the color attachment as texture attached to a billboard to render what was on the screen

But it seems like this pipeline doesn't translate well into VTK. I paired again on wednesday with my mentors, Bruno and Filipi, to try to figure out where the problem was, but after hours we could not find it. Wednesday passed and then thursday came, and with thursday, a solution: Bruno didn't give up on the idea and dug deep on VTK's documentation until he found a workaround to do what we wanted, that was retrieving a texture from what was rendered to the screen and pass it as a texture to render to the billboard. To do it, he figured out we needed to use a different class, vtkWindowToImageFilter , a class that has the specific job of doing exactly what I described above. Below, the steps to do it:

windowToImageFilter = vtk.vtkWindowToImageFilter()
windowToImageFilter.SetInput(scene.GetRenderWindow())
windowToImageFilter.Update()

texture = vtk.vtkTexture()
texture.SetInputConnection(windowToImageFilter.GetOutputPort())
# Bind the framebuffer texture to the desired actor
actor.SetTexture(texture)

This is enough to bind to the desired actor a texture that corresponds to was prior rendered to the screen.

This Week's Goals

Having a solution to that, now its time to finally render some KDE's! This week's plans involve implementing the first version of a KDE calculation. For anyone interested in understanding what a Kernel Density Estimation is, here is a brief summary from this Wikipedia page:

"In statistics, kernel density estimation (KDE) is the application of kernel smoothing for probability density estimation, i.e., a non-parametric method to estimate the probability density function of a random variable based on kernels as weights. KDE answers a fundamental data smoothing problem where inferences about the population are made, based on a finite data sample. In some fields such as signal processing and econometrics it is also termed the Parzen–Rosenblatt window method, after Emanuel Parzen and Murray Rosenblatt, who are usually credited with independently creating it in its current form. One of the famous applications of kernel density estimation is in estimating the class-conditional marginal densities of data when using a naive Bayes classifier, which can improve its prediction accuracy."

This complicated sentence can be translated into the below image:

That is what a KDE plot of 100 random points look like. The greener the area, the greater the density of points. The plan is to implement something like that with the tools we now have available.

Let's get to work!

Week 4: Nothing is Ever Lost

joaovdafloriano@usp.br (joaodellagli) — Mon, 26 Jun 2023 21:17:07 +0000

Welcome again to another weekly blogpost! Today, let's talk about the importance of guidance throughout a project.

Last Week's Effort

So, last week my project was struggling with some supposedly simple in concept, yet intricate in execution issues. If you recall from my last blogpost, I could not manage to make the Framebuffer Object setup work, as its method, SetContext(), wasn't being able to generate the FBO inside OpenGL. Well, after some (more) research about that as I also dived in my plan B, that involved studying numba as a way to accelerate a data structure I implemented on my PR #783, me and one of my mentors decided we needed a pair programming session, that finally happened on thursday. After that session, we could finally understand what was going on.

Where the Problem Was

Apparently, for the FBO generation to work, it is first needed to initialize the context interactor:

FBO = vtk.vtkOpenGLFramebufferObject()

manager.window.SetOffScreenRendering(True) # so the window doesn't show up, but important for later as well

manager.initialize() # missing part that made everything work


FBO.SetContext(manager.window) # Sets the context for the FBO. Finally, it works
FBO.PopulateFramebuffer(width, height, True, 1, vtk.VTK_UNSIGNED_CHAR, False, 24, 0) # And now I could populate the FBO with textures

This simple missing line of code was responsible for ending weeks of suffer, as after that, I called:

print("FBO of index:", FBO.GetFBOIndex())
print("Number of color attachments:", FBO.GetNumberOfColorAttachments())

That outputted:

FBO of index: 4
Number of color attachments: 1

That means the FBO generation was successful! One explanation that seems reasonable to me on why was that happening is that, as it was not initialized, the context was being passed null to the SetContext() method, that returned without any warning of what was happening.

Here, I would like to point out how my mentor was essential to this solution to come: I had struggled for some time with that, and could not find a way out, but a single session of synchronous pair programming where I could expose clearly my problem and talk to someone way more experienced than I, someone designated for that, was my way out of this torment, so value your mentors! Thanks Bruno!

This Week's Goals

Now, with the FBO working, I plan to finally render something to it. For this week, I plan to come back to my original plan and experiment with simple shaders just as a proof of concept that the FBO will be really useful for this project. I hope the road is less bumpier by now and I don't step on any other complicated problem.

Wish me luck!

Week 3: Watch Your Expectations

joaovdafloriano@usp.br (joaodellagli) — Wed, 21 Jun 2023 17:39:41 +0000

Hello everyone, it's time for another weekly blogpost! This week, I will talk about how you should watch your expectations when working with any project.

This Last Week's Effort

As I supposedly managed to make the texture allocation part working, this last week's goal was to render something to a FBO. Well, I could make textures work, but what I wasn't expecting and later realised, was that the FBO setup not working. Below I will describe where I got stuck.

Where the Problem Was

After getting the textures setup right, I was ready to render some color to the FBO. Well, I was, because I didn't expect I would have another problem, this time, with the FBO setup. As described in my week 1 blogpost, a FBO needs some requirements to work. My current problem relies on the FBO method FBO.SetContext(), that for some reason is not being able to generate the FBO. Below, how the method is currently operating:

Apparently, the method is stuck before the this->CreateFBO(), that can be checked when we call FBO.GetFBOIndex(), that returns a 0 value, meaning the FBO was not generated by the glGenFramebuffers() function, that is inside the GetContext() method.

This Week's Goals

As I got stuck again with this simple step, talking with my mentors we concluded that a plan B is needed for my GSoC participation as my current project is not having much progress. This plan B that I am gonna start working on this week involves working on FURY Speed, a FURY addon that aims to develop optimized functions and algorithms to help speed up graphical applications. The suggestion was to work on a PR I submitted months ago, #783, in a way to integrate that into FURY Speed. Also, I plan to keep working on my current project to find the solution I will need to make the FBO usage work.

Let's get to work!

The Importance of (good) Documentation - Week 2

joaovdafloriano@usp.br (joaodellagli) — Tue, 13 Jun 2023 13:40:46 +0000

Hello everybody, welcome to the week 2 of this project! I must admit I thought this would be simpler than it is currently being, but I forgot that when it comes to dealing with computer graphics' applications, things never are. Below, some updates on what I have been up to for this past week.

This Last Week's Effort

Last week, I was facing some issues with a VTK feature essential so I could move forward with my project: Framebuffer Objects. As described in my last blogpost, for some reason the 2D allocation methods for it weren't working. In a meeting with my mentors, while we were discussing and searching through VTK's FramebufferObject and TextureObject documentation, and the code itself for the problem, one TextureObject method caught my attention: vtkTextureObject.SetContext().

Where the Problems Were

My last week's code was:

color_texture = vtk.vtkTextureObject() # color texture declaration
color_texture.Bind() # binding of the texture for operations

color_texture.SetDataType(vtk.VTK_UNSIGNED_CHAR) # setting the datatype for unsigned char
color_texture.SetInternalFormat(vtk.VTK_RGBA) # setting the format as RGBA
color_texture.SetFormat(vtk.VTK_RGBA)
color_texture.SetMinificationFilter(0) # setting the minfilter as linear
color_texture.SetMagnificationFilter(0) # setting the magfilter as linear

color_texture.Allocate2D(width, height, 4, vtk.VTK_UNSIGNED_CHAR) # here is where the code stops

But it turns out that to allocate the FBO's textures, of type vtkTextureObject, you need to also set the context where the texture object will be present, so it lacked the line:

color_texture.SetContext(manager.window) # set the context where the texture object will be present

And with that line added after Bind():

color_texture = vtk.vtkTextureObject()
color_texture.Bind()

color_texture.SetContext(manager.window) # set the context where the texture object will be present

color_texture.SetDataType(vtk.VTK_UNSIGNED_CHAR)
color_texture.SetInternalFormat(vtk.VTK_RGB)
color_texture.SetFormat(vtk.VTK_RGB)
color_texture.SetMinificationFilter(0)
color_texture.SetMagnificationFilter(0)

The code worked fine. But as my last blogpost showed, Allocate3D() method worked just fine without a (visible) problem, why is that? Well, in fact, it didn't work. If we check the code for the Allocate2D and Allocate3D, one difference can be spotted.

While in Allocate2D there is an "assert(this->Context);", in Allocate3D the assertion is translated into:

if(this->Context==nullptr)
{
  vtkErrorMacro("No context specified. Cannot create texture.");
  return false;
}

This slight difference is significant: while in Allocate2D the program immediatly fails, in Allocate3D the function is simply returned false, with its error pushed to vtkErrorMacro. I could have realised that earlier if I were using vtkErrorMacro, but this contrastant implementation made it harder for me and my mentors to realise what was happening.

This Week's Goals

After making that work, this week's goal is to render something to the Framebuffer Object, now that is working. To do that, first I will need to do some offscreen rendering to it, and afterwards render what it was drawn to its color attachment, the Texture Object I was struggling to make work, into the screen, drawing its texture to a billboard. Also, I plan to start using vtkErrorMacro, as it seems like the main error interface when working with VTK, and that may make my life easier.

See you next week!

The FBO Saga - Week 1

joaovdafloriano@usp.br (joaodellagli) — Mon, 05 Jun 2023 19:54:45 +0000

This Past Week

As mentioned in the last week's blogpost, the goal for that week was to investigate VTK's Framebuffer Object framework. An update on that is that indeed, VTK has one more low-level working FBO class that can be used inside FURY, however, they come with some issues that I will explain further below.

My Current Problems

The problems I am having with these FBO implementations are first something related to how an FBO works, and second related to how VTK works. In OpenGL, a custom user's FBO needs some things to be complete (usable):

At least one buffer should be attached. This buffer can be the color, depth or stencil buffer.
If no color buffer will be attached then OpenGL needs to be warned no draw or read operations will be done to that buffer. Otherwise, there should be at least one color attachment.
All attachments should have their memory allocated.
Each buffer should have the same number of samples.

My first problem relies on the third requirement. VTK's implementation of FBO requires a vtkTextureObject as a texture attachment. I figured how to work with this class, however, I cannot allocate memory for it, as its methods for it, Allocate2D, Create2D and Create2DFromRaw does not seem to work. Every time I try to use them, my program stops with no error message nor nothing. For anyone interested in what is happening exactly, below is how I my tests are implemented:

color_texture = vtk.vtkTextureObject() # color texture declaration
color_texture.Bind() # binding of the texture for operations
color_texture.SetDataType(vtk.VTK_UNSIGNED_CHAR) # setting the datatype for unsigned char
color_texture.SetInternalFormat(vtk.VTK_RGBA) # setting the format as RGBA
color_texture.SetFormat(vtk.VTK_RGBA)
color_texture.SetMinificationFilter(0) # setting the minfilter as linear
color_texture.SetMagnificationFilter(0) # setting the magfilter as linear

color_texture.Allocate2D(width, height, 4, vtk.VTK_UNSIGNED_CHAR) # here is where the code stops

In contrast, for some reason, the methods for 3D textures, Allocate3D works just fine.I could use it as a workaround, allocating memory for a width x height x 1 texture (equivalent to a 2D texture) but I do not wish to, as this just does not make sense.

My second problem relies on VTK. As VTK is a library that encapsulates some OpenGL functions in more palatable forms, it comes with some costs. Working with FBOs is a more low-level work, that requires strict control of some OpenGL states and specific functions that would be simpler if it was the main API here. However, some of this states and functions are all spread and implicit through VTK's complex classes and methods, which doubles the time expended to make some otherwise simple instructions, as I first need to dig in lines and lines of VTK's documentation, and worse, the code itself.

What About Next Week?

For this next week, I plan to investigate further on why the first problem is happening. If that is accomplished, then things will be more simple, as I will be a lot easier for my project as I will finally be able to implement the more pythonic functions I will need to finally render some kernel distributions onto my screen.

Wish me luck!

The Beggining of Everything - Week 0

joaovdafloriano@usp.br (joaodellagli) — Mon, 29 May 2023 16:51:59 +0000

So It Begins...

Hello everyone, welcome to the beggining of my journey through GSoC 2023! I would like to thank everyone involved for the opportunity provided, it is an honour to be working side by side with professionals and so many experienced people from around the world.

The Community Bonding Period

During my community bonding period, I had the opportunity to meet my mentors ans some people from the FURY team. It was a great time to learn about community guidelines and everything I will need to work with them during this summer.

The Project's Goal

For some reason I am having trouble with uploading my proposal, so I will introduce my goals. Briefly explaining this project, I plan to implement a real-time Kernel Density Estimation shader inside FURY library, based on Filipi Nascimento's WebGL implementation. KDE, or Kernel Density Estimation, is a visualization technique that provides a good macro visualization of large and complex data sets, like point clouds, well summarizing their spatial distribution in smooth areas. I really think FURY will benefit from this as a scientific library, knowing it is a computer graphics library that originated in 2018 based on the Visual Toolkit API (VTK), and has been improving since then.

This Week's Goal

For all of this to work, the project needs one component working: the KDE framebuffer. As this Khronos wiki page well explains:

"A Framebuffer is a collection of buffers that can be used as the destination for rendering. OpenGL has two kinds of framebuffers: the Default Framebuffer, which is provided by the OpenGL Context; and user-created framebuffers called Framebuffer Objects (FBOs). The buffers for default framebuffers are part of the context and usually represent a window or display device. The buffers for FBOs reference images from either Textures or Renderbuffers; they are never directly visible."

Which means that a framebuffer is an object that stores data related to a frame. So the goal for this week is to investigate whether VTK, the API which FURY is written on, has a framebuffer object interface, and if it has, to understand how it works and how to use it for the project.

Let's get to work!