Google Summer of Code Final Work Product

• Name: João Victor Dell Agli Floriano
• Organisation: Python Software Foundation
• Sub-Organisation: FURY
• Project: FURY - Project 2. Fast 3D kernel-based density rendering using billboards.

Abstract

This project had the goal to implement a 3D Kernel Density Estimation rendering to Fury. Kernel Density Estimation, or KDE, is a statistical method that uses kernel smoothing for modeling and estimating the density distribution of a set of points defined inside a given region. For its graphical implementation, it was used post-processing techniques such as offscreen rendering to framebuffers and colormap post-processing as tools to achieve the desired results. This was completed with a functional basic KDE rendering result, that relies on a solid and easy-to-use API, as well as some additional features.

Proposed Objectives

• First Phase : Implement framebuffer usage in FURY

  • Investigate the usage of float framebuffers inside FURY's environment.
  • Implement a float framebuffer API.

• Second Phase : Shader-framebuffer integration

  • Implement a shader that uses a colormap to render framebuffers.
  • Escalate this rendering for composing multiple framebuffers.

• Third Phase : KDE Calculations

  • Investigate KDE calculation for point-cloud datasets.
  • Implement KDE calculation inside the framebuffer rendering shaders.
  • Test KDE for multiple datasets.

Objectives Completed

• Implement framebuffer usage in FURY

  The first phase, addressed from May/29 to July/07, started with the investigation of VTK's Frambuffer Object, a vital part of this project, to understand how to use it properly.

  Framebuffer Objects, abbreviated as FBOs, are the key to post-processing effects in OpenGL, as they are used to render things offscreen and save the resulting image to a texture that will be later used to apply the desired post-processing effects within the object's fragment shader rendered to screen, in this case, a billboard. In the case of the Kernel Density Estimation post-processing effect, we need a special kind of FBO, one that stores textures' values as floats, different from the standard 8-bit unsigned int storage. This is necessary because the KDE rendering involves rendering every KDE point calculation to separate billboards, rendered to the same scene, which will have their intensities, divided by the number of points rendered, blended with OpenGL Additive Blending, and if a relative big number of points are rendered at the same time, 32-bit float precision is needed to guarantee that small-intensity values will not be capped to zero, and disappear.

  After a month going through VTK's FBO documentation and weeks spent trying different approaches to this method, it would not work properly, as some details seemed to be missing from the documentation, and asking the community haven't solved the problem as well. Reporting that to my mentors, which unsuccessfully tried themselves to make it work, they decided it was better if another path was taken, as the 32-bit float FBO would result in a much more complicated pipeline, using VTK's WindowToImageFilter method as a workaround, described in this blogpost. This method helped the development of three new functions to Fury, window_to_texture(), texture_to_actor() and colormap_to_texture(), that allow the passing of different kinds of textures to Fury's actor's shaders, the first one to capture a window and pass it as a texture to an actor, the second one to pass an external texture to an actor, and the third one to specifically pass a colormap as a texture to an actor. It is important to say that WindowToImageFilter() is not the ideal way to make it work, as this method does not seem to support float textures. However, a workaround to that is currently being worked on, as I will describe later on.

  Pull Requests:

  • KDE Rendering Experimental Program (needs major revision): fury-gl/fury#804

    The result of this whole FBO and WindowToImageFilter experimentation is well documented in PR #804 that implements an experimental version of a KDE rendering program. The future of this PR, as discussed with my mentors, is to be better documented to be used as an example for developers on how to develop features in Fury with the tools used, and it shall be done soon.

• Shader-framebuffer integration

  The second phase, which initially was thought of as "Implement a shader that uses a colormap to render framebuffers" and "Escalate this rendering for composing multiple framebuffers" was actually a pretty simple phase that could be addressed in one week, July/10 to July/17, done at the same time as the third phase goal, documented in this blogpost. As Fury already had a tool for generating and using colormaps, they were simply connected to the shader part of the program as textures, with the functions explained above. Below, is the result of the matplotlib viridis colormap passed to a simple gaussian KDE render:

  

  That is also included in PR #804. Having the 2D plot ready, some time was taken to figure out how to enable a 3D render, that includes rotation and other movement around the set rendered, which was solved by learning about the callback properties that exist inside VTK. Callbacks are ways to enable code execution inside the VTK rendering loop, enclosed inside vtkRenderWindowInteractor.start(). If it is desired to add a piece of code that, for example, passes a time variable to the fragment shader over time, a callback function can be declared:

  from fury import window
  t = 0
  showm = window.ShowManager(...)
  
  def callback_function:
      t += 0.01
      pass_shader_uniforms_to_fs(t, "t")
  
  showm.add_iren_callback(callback_function, "RenderEvent")
  

  The piece of code above created a function that updates the time variable t in every "RenderEvent", and passes it to the fragment shader. With that property, the camera and some other parameters could be updated, which enabled 3D visualization, that then, outputted the following result, using matplotlib inferno colormap:

  

• KDE Calculations (ongoing)

  As said before, the second and third phases were done simultaneously, so after having a way to capture the window and use it as a texture ready, the colormap ready, and an initial KDE render ready, all it was needed to do was to improve the KDE calculations. As this Wikipedia page explains, a KDE calculation is to estimate an abstract density around a set of points defined inside a given region with a kernel, that is a function that models the density around a point based on its associated distribution sigma.

  A well-known kernel is, for example, the Gaussian Kernel, that says that the density around a point p with distribution sigma is defined as:

  

  Using that kernel, we can calculate the KDE of a set of points A with associated distributions S calculating their individual Gaussian distributions, summing them up and dividing them by the total number of points p:

  

  So I dove into implementing all of that into the offscreen rendering part, and that is when the lack of a float framebuffer would charge its cost. As it can be seen above, just calculating each point's density isn't the whole part, as I also need to divide everyone by the total number of points p, and then sum them all. The problem is that, if the number of points its big enough, the individual densities will be really low, and that would not be a problem for a 32-bit precision float framebuffer, but that is definitely a problem for a 8-bit integer framebuffer, as small enough values will simply underflow and disappear. That issue is currently under investigation, and some solutions have already been presented, as I will show in the Objectives in Progress section.

  Apart from that, after having the experimental program ready, I focused on modularizing it into a functional and simple API (without the n division for now), and I could get a good set of results from that. The API I first developed implemented the EffectManager class, responsible for managing all of the behind-the-scenes steps necessary for the kde render to work, encapsulated inside the ÈffectManager.kde() method. It had the following look:

  from fury.effect_manager import EffectManager
  from fury import window
  
  showm = window.ShowManager(...)
  
  # KDE rendering setup
  em = EffectManager(showm)
  kde_actor = em.kde(...)
  # End of KDE rendering setup
  
  showmn.scene.add(kde_actor)
  
  showm.start()
  

  Those straightforward instructions, that hid several lines of code and setup, could manage to output the following result:

  

  And this was not the only feature I had implemented for this API, as the use of WindowToImageFilter method opened doors for a whole new world for Fury: The world of post-processing effects. With this features setup, I managed to implement a Gaussian blur effect, a grayscale effect and a Laplacian effect for calculating "borders":

  

  

  

  As this wasn't the initial goal of the project and I still had several issues to deal with, I have decided to leave these features as a future addition.

  Talking with my mentors, we realized that the first KDE API, even though simple, could lead to bad usage from users, as the em.kde() method, that outputted a Fury actor, had dependencies different from any other object of its kind, making it a new class of actors, which could lead to confusion and bad handling. After some pair programming sessions, they instructed me to take a similar, but different road from what I was doing, turning the kde actor into a new class, the KDE class. This class would have almost the same set of instructions present in the prior method, but it would break them in a way it would only be completely set up after being passed to the EffectManager via its add function. Below, how the refactoring handles it:

  from fury.effects import EffectManager, KDE
  from fury import window
  
  showm = window.ShowManager(...)
  
  # KDE rendering setup
  em = EffectManager(showm)
  kde_effect = KDE(...)
  em.add(kde_effect)
  # End of KDE rendering setup
  
  showm.start()
  

  Which outputted the same results as shown above. It may have cost some simplicity as we are now one line farther from having it working, but it is more explicit in telling the user this is not just a normal actor.

  Another detail I worked on was the kernel variety. The Gaussian Kernel isn't the only one available to model density distributions, there are several others that can do that job, as it can be seen in this scikit-learn piece of documentation and this Wikipedia page on kernels. Based on the scikit-learn KDE implementation, I worked on implementing the following kernels inside our API, that can be chosen as a parameter when calling the KDE class:

  • Cosine
  • Epanechnikov
  • Exponential
  • Gaussian
  • Linear
  • Tophat

  Below, the comparison between them using the same set of points and bandwidths:

  

  Pull Requests:

  • First Stage of the KDE Rendering API (will be merged soon) fury-gl/fury#826

    All of this work culminated in PR #826, that proposes to add the first stage of this API (there are some details yet to be completed, like the n division) to Fury. This PR added the described API, and also proposed some minor changes to some already existing Fury functions related to callbacks, changes necessary for this and other future applications that would use it to work. It also added the six kernels described, and a simple documented example on how to use this feature.

Other Objectives

• Stretch Goals : SDE Implementation, Network/Graph visualization using SDE/KDE, Tutorials
  • Investigate SDE calculation for surface datasets.
  • Implement SDE calculation inside the framebuffer rendering shaders.
  • Test SDE for multiple datasets.
  • Develop comprehensive tutorials that explain SDE concepts and FURY API usage.
  • Create practical, scenario-based tutorials using real datasets and/or simulations.

Objectives in Progress

• KDE Calculations (ongoing)

  The KDE rendering, even though almost complete, have the n division, an important step, missing, as this normalization allows colormaps to cover the whole range o values rendered. The lack of a float FBO made a big difference in the project, as the search for a functional implementation of it not only delayed the project, but it is vital for the correct calculations to work.

  For the last part, a workaround thought was to try an approach I later figured out is an old one, as it can be check in GPU Gems 12.3.3 section: If I need 32-bit float precision and I got 4 8-bit integer precision available, why not trying to pack this float into this RGBA texture? I have first tried to do one myself, but it didn't work for some reason, so I tried Aras Pranckevičius implementation, that does the following:

  vec4 float_to_rgba(float value) {
     vec4 bitEnc = vec4(1.,256.,65536.0,16777216.0);
     vec4 enc = bitEnc * value;
     enc = fract(enc);
     enc -= enc.yzww * vec2(1./255., 0.).xxxy;
     return enc;
  }
  

  That initially worked, but for some reason I am still trying to understand, it is resulting in a really noisy texture:

  

  One way to try to mitigate that while is to pass this by a gaussian blur filter, to try to smooth out the result:

  

  But it is not an ideal solution as well, as it may lead to distortions in the actual density values, depending on the application of the KDE. Now, my goal is to first find the root of the noise problem, and then, if that does not work, try to make the gaussian filter work.

  Another detail that would be a good addition to the API is UI controls. Filipi, one of my mentors, told me it would be a good feature if the user could control the intensities of the bandwidths for a better structural visualization of the render, and knowing Fury already have a good set of UI elements, I just neeeded to integrate that into my program via callbacks. I tried implementing an intensity slider. However, for some reason, it is making the program crash randomly, for reasons I still don't know, so that is another issue under investigation. Below, we show a first version of that feature, which was working before the crashes:

  

  Pull Requests:

  • UI intensity slider for the KDE rendering API (draft): fury-gl/fury#849
  • Post-processing effects for FURY Effects API (draft): fury-gl/fury#850

GSoC Weekly Blogs

• My blog posts can be found at FURY website and Python GSoC blog.

Timeline

Hello everyone, it's time for another GSoC blogpost! Today, I am going to talk about some minor details I worked on my project.

Last Week's Effort

After the API refactoring done last week, I focused on addressing the reviews I would get from it. The first issues I addressed was related to style, as there were some minor details my GSoC contributors pointed out that needed change. Also, I have addressed an issue I was having with the `typed hint` of one of my functions. Filipi, my mentor, showed me there is a way to have more than one typed hint in the same parameter, all I needed to do was to use the Union class from the typing module, as shown below:

from typing import Union as tUnion
from numpy import ndarray
 
def function(variable : tUnion(float, np.ndarray)):
    pass

Using that, I could set the typedhint of the bandwidth variable to float and np.ndarray.

So how did it go?

All went fine with no difficult at all, thankfully.

The Next Steps

My next plans are, after having PR #826 merged, to work on the float encoding issue described in this blogpost. Also, I plan to tackle the UI idea once again, to see if I can finally give the user a way to control de intensities of the distributions.

Wish me luck!

Hello everyone, it's time for another weekly blogpost! Today I am going to share some updates on the API refactoring I was working on with my mentors.

Last Week's Effort

As I shared with you last week, the first draft of my API was finally ready for review, as I finished tweaking some remaining details missing. I was tasked with finding a goode example of usage of the tools we proposed, and I started to do that, however after testing it with some examples, I figured out some significant bugs were to be fixed. Also, after some reviews and hints from some of my mentors and other GSoC contributors, we realised that some refactoring should be done, mainly focused on avoiding bad API usage from the user.

So how did it go?

Initially, I thought only one bug was the source of the issues the rendering presented, but it turned out to be two, which I will explain further.

The first bug was related to scaling and misalignment of the KDE render. The render of the points being post-processed was not only with sizes different from the original set size, but it was also misaligned, making it appear in positions different from the points' original ones. After some time spent, I figured out the bug was related to the texture coordinates I was using. Before, this is how my fragment shader looked:

vec2 res_factor = vec2(res.y/res.x, 1.0);
vec2 tex_coords = res_factor*normalizedVertexMCVSOutput.xy*0.5 + 0.5;
float intensity = texture(screenTexture, tex_coords).r;

It turns out using this texture coordinates for *this case* was not the best choice, as even though it matches the fragment positions, the idea here was to render the offscreen window, which has the same size as the onscreen one, to the billboard actor. With that in mind, I realised the best choice was using texture coordinates that matched the whole screen positions, coordinates that were derived from the gl_FragCoord.xy, being the division of that by the resolution of the screen, for normalization. Below, the change made:

vec2 tex_coords = gl_FragCoord.xy/res;
float intensity = texture(screenTexture, tex_coords).r;

This change worked initially, although with some problems, that later revealed the resolution of the offscreen window needed to be updated inside the callback function as well. Fixing that, it was perfectly aligned and scaled!

The second bug was related with the handling of the bandwidth, former sigma parameter. I realised I wasn't dealing properly with the option of the user passing only one single bandiwidth value being passed, so when trying that, only the first point was being rendered. I also fixed that and it worked, so cheers!

As I previously said, the bugs were not the only details I spent my time on last week. Being reviewed, the API design, even though simple, showed itself vulnerable to bad usage from the user side, requiring some changes. The changes suggested by mentors was, to, basically, take the kde method out of the EffectManager class, and create a new class from it inside an effects module, like it was a special effects class. With this change, the KDE setup would go from:

em = EffectManager(show_manager)

kde_actor = em.kde(...)

show_manager.scene.add(kde_actor)

To:

em = EffectManager(show_manager)

kde_effect = KDE(...)

em.add(kde_effect)

Not a gain in line shortening, however, a gain in security, as preventing users from misusing the kde_actor. Something worth noting is that I learned how to use the functools.partial function, that allowed me to partially call the callback function with only some parameters passed.

This Week's Goals

Having that refactoring made, now I am awaiting for a second review so we could finally wrap it up and merge the first stage of this API. With that being done, I will write the final report and wrap this all up.

Let's get to work!

Hello everyone, it's time for another weekly blogpost!

Last Week's Effort

After talking with my mentors, I was tasked with getting my API PR #826 ready for review, as it still needed some polishing, and the most important of all, it needed its tests working, as this was something I haven't invested time since its creation. Having that in mind, I have spent the whole week cleaning whatever needed, writing the tests, and also writing a simple example of its usage. I also tried implementing a little piece of UI so the user could control the intensity of the bandwidth of the KDE render, but I had a little problem I will talk about below.

So how did it go?

Fortunately, for the cleaning part, I didn't have any trouble, and my PR is finally ready for review! The most complicated part was to write the tests, as this is something that requires attention to understand what needs to be tested, exactly. As for the UI part, I managed to have a slider working for the intensity, however, it was crashing the whole program for a reason, so I decided to leave this idea behind for now. Below, an example of how this should work:

This Week's Goals

After a meeting with my mentors, we decided that this week's focus should be on finding a good usage example of the KDE rendering feature, to have it as a showcase of the capability of this API. Also, they hinted me some changes that need to be done regarding the API, so I will also invest some time on refactoring it.

Wish me luck!

Hello everyone, it's time for another weekly blogpost! Today, I am going to update you on my project's latest changes.

Last Week's Effort

After having finished a first draft of the API that will be used for the KDE rendering, and showing how it could be used for other post-processing effects, my goal was to clean the code and try some details that would add to it so it could be better complete. Having that in mind, I invested in three work fronts:

1. Fixing some bugs related to the rendering more than one post-processing effect actor.
2. Experimenting with other rendering kernels (I was using the *gaussian* one only).
3. Completing the KDE render by renormalizing the values in relation to the number of points (one of the core KDE details).

Both three turned out more complicated than it initially seemed, as I will show below.

So how did it go?

The first one I did on monday-tuesday, and I had to deal with some issues regarding scaling and repositioning. Due to implementation choices, the final post-processed effects were rendered whether bigger than they were in reality, or out of their original place. After some time dedicated to finding the root of those problems, I could fix the scaling issue, however I realised I would need to,probably, rethink the way the api was implemented. As this general post-processing effects is a side-project that comes as a consequence of my main one, I decided to leave that investment to another time, as I would need to guarantee the quality of the second.

The second was an easy and rather interesting part of my week, as I just needed to setup new kernel shaders. Based on scikit-learn KDE documentation, I could successfully implement the following kernels:

• Gaussian
• Tophat
• Epanechnikov
• Exponential
• Linear
• Cosine

That outputted the following (beautiful) results for a set of 1000 random points with random sigmas:

The third one is still being a trickier challenge. If you recall from my first blogposts, I spent something around one month trying to setup float framebuffer objects to FURY with VTK so I could use them in my project. After spending all of that time with no results, me and Bruno, my mentor, found a way to do what we wanted to do, but using a different VTK class, vtkWindowToImageFilter. Well, it was a good workaround back then and it lead me all the way here, however now it is costing a price. The float framebuffers were an important part of the project because they would allow us to pass *2-bit float information from one shader to another, which would be important as they would allow the densities to have higher precision and more fidelity to the calculations. When rendering a KDE of a given set of points, we use the below function:

KDE(x, y) = n^-1sum( kernel(x, y) )

If the number of points n is big enough, some KDE results will be really low. This presents a real problem to our implementation because, without the float framebuffers, it is currently only being possible to pass 8-bit unsigned char information, that only allows 256 values. This is far from ideal, as low values would have alone densities low enough to disappear. This presented a problem as to renormalize the densities, I was retrieving the texture to the CPU, calculating its minimum and maximum values, and passing to the fragment shader as uniformsfor the renormalization, which didn't work if the maximum values calculated were zero.

One solution I thought to solve that was a really heavy workaround: if an unsigned float is 32-bit and I have exactly 4 8-bit unsigned chars, why not try to pack this float into these 4 chars? Well, this is an interesting approach which I figured out is already an old one, being reported in GPU Gems's chapter 12. Unfortunately I haven't tried yet this implementation yet, and went for one I thought myself, which haven't exactly worked. I also tried this implementation from Aras Pranckevičius' website, which seems to be working, even though not perfectly:

That looks way better, even though not ideal yet.

This Week's Goals

Talking with my mentors, we decided it was better if I focused on the version without the renormalization for now, as it was already done and running fine. So for this week, I plan to clean my PR to finally have it ready for a first review, and maybe add to it a little UI tool to control the intensity of the densities. That should take me some time and discussion, but I hope for it to be ready by the end of the week.

Let's get to work!