KDE Rendering Experimental Program #804

JoaoDell · 2023-06-15T23:07:47Z

[UPDATED]
Hello everyone, this PR contains an experimental implementation of Kernel Density Estimation calculations for a set of points. This uses some of fury's existing APIs and some straight vtk ones. This program may be a good example of shader usage and how to apply post-processing effects inside FURY. The result from running that should be:

Below, what is the idea here:

Render billboards with KDE calculations of points to an offscreen window.
Capture the offscreen window with a texture.
Render a second billboard, this time to the screen, with the texture of the prior rendered window, having a post processing effect applied to it (color map application).

In more details, what is done:

The offscreen and normal screen managers are created. The offscreen window is set to be for offscreen rendering with SetOffscreenRendering(True).
The parameters of the rendering are created: Random points are initialized, the sigma for the KDE is set, and the scale of the billboards is defined. The points are passed to the billboard actor as its centers.
shaders.shader_apply_effects() is used to tell OpenGL additive blending is desired for this code.
The billboards are rendered to the offscreen window.
After this render, the second billboard, the one that will receive the texture from the prior rendering, is created.
The colormap is chosen from matplotlib.colormaps and passed to the billboard as a texture named "colormapTexture" with the function colormap_to_texture()
The window is captured as a texture and passed to the second billboard with window_to_texture()
The billboard is then finally rendered to the screen.

Some minor details that may be useful for a better understanding of it:

The billboard actor got a set o shaders for the KDE calculation. In its declaration, a function that calculates the non-normalized (yet, I intend to implement a normalized version) KDE as: $KDE(x, \sigma) = exp(-\frac{1}{2}(\frac{x}{\sigma})^2)$.
For all of that to work, I have created some functions to ease the work, functions that may be even used later in other FURY applications, so I wonder it would be a good idea to make a separate small PR for them.

skoudoro · 2023-06-16T14:34:23Z

thank you for your feedback. we will look at all this links.

codecov · 2023-06-23T17:47:42Z

Codecov Report

Merging #804 (b611368) into master (3d53e48) will decrease coverage by 0.05%.
The diff coverage is n/a.

❗ Current head b611368 differs from pull request most recent head bf2b91a. Consider uploading reports for the commit bf2b91a to get more accurate results

@@            Coverage Diff             @@
##           master     #804      +/-   ##
==========================================
- Coverage   84.43%   84.38%   -0.05%     
==========================================
  Files          43       43              
  Lines       10274    10274              
  Branches     1393     1393              
==========================================
- Hits         8675     8670       -5     
- Misses       1239     1243       +4     
- Partials      360      361       +1

see 1 file with indirect coverage changes

JoaoDell · 2023-06-23T17:52:15Z

After a pair with @devmessias, we made some progress on the FBO setup issue. Now, the FBO is sucessfully generated, and it seems ready to be drawn to. Apparently, the issue was that the context needed to have the window interactor initialized with ShowManager.initialize() before the it was set to the FBO. I will prepare a better explaination and investigate further on why is that soon.

devmessias · 2023-06-26T22:26:03Z

docs/experimental/framebuffer_example.py

+
+
+frag_decl = """varying vec2 out_tex; 
+               uniform vec2 res0;  


@JoaoDell you should declare in vec3 normalizedVertexMCVSOutput; here

frag_decl = """varying vec2 out_tex; in vec3 normalizedVertexMCVSOutput; uniform vec2 res0; uniform vec3 point0;"""

It seems like the problem was somewhere else, but i have just updated the PR with some changes that don't conflict with the shaders

devmessias · 2023-06-26T22:50:51Z

docs/experimental/framebuffer_example.py

+
+# Render every point existing inside loop
+for i in range(n_points):
+    shader_custom_uniforms(billboard, "fragment").SetUniform3f("point0", points[i, :])


I don't believe this action is feasible as it would attempt to create new instances of 'point0' and 'res0' variables. This leads to an error. The error message I received :

2023-06-26 19:22:22.899 ( 0.328s) [ B8DAB000] vtkShaderProgram.cxx:453 ERR| vtkShaderProgram (0x558732a3cb20): 0:109(15): error: `res0' redeclared 0:110(15): error: `point0' redeclared

We have a old PR that deals with that
#424 maybe we need to look into this again @skoudoro and @filipinascimento

Interesting, I didn't know how VTK handled uniforms so I thought the SetUniform methods only passed uniform values to the shader. That seems like something worth checking out indeed 👍

JoaoDell · 2023-06-27T19:59:15Z

Hey @devmessias thanks for pointing those out, I have just updated this PR with my current progress:

I could manage to stabilize the shader issues and now they seem to be working.
I have cleaned better the code itself to make it simpler.
I am being able to render simple colors to the billboard screen, however, I cannot seem to be able to use the color attachment as a texture after drawing to it offscreen on the FBO. When I activate the color texture to render to the screen, it outputs a pitch black render. I think that is due to my lack of enough understandment on how vtkFBOs and vtkTextureObjects work and get activated, will look better into it.

Maybe @filipinascimento could have a better idea of what is happening as well.

devmessias · 2023-06-27T20:28:33Z

docs/experimental/framebuffer_example.py

+uniform sampler2D screenTexture;
+
+void main(){ 
+    vec4 texture = texture(screenTexture, out_tex);


texture should be 2D vec right?
try this

void main(){ //vec2 texture = texture(screenTexture, out_tex); // gl_FragColor = texture; vec3 normalColor = texture2D(screenTexture, out_tex).rgb; gl_FragColor = vec4(normalColor.r, normalColor.g, normalColor.b, 1); //gl_FragColor = vec4(0.0, 0.0, 1.0, 1.0); }

I was passing out_tex as texture coordinates, but I will try what you recommended

JoaoDell · 2023-07-14T17:15:44Z

Hey everyone, I just updated and completely refactored this PR as, as well as it is base program, as it went on a slight different direction in comparison to its original intetion. Everything seems to be working now, my only question is if it would be better if I closed this PR and opened a new one with a new branch containing only the recent commits, as the older ones have been completely replaced, or should I rebase it, I don't know. What should I do?

Anyway, this version is ready for review 👍

tvcastillod

Hi @JoaoDell,
I need to take a deeper look at your work, but I've left some comments below. Overall is looking good.

docs/experimental/viz_points_kde.py

ganimtron-10

Hey @JoaoDell ,
Giving a first look I found out these below points. Some of them would require more discussion from mentors to clarify few things.
If you think any of these comments doesn't align according to your approach while implementing it feel free to ping me or mentors at the comment.

Thanks in advance!

docs/experimental/viz_points_kde.py

ganimtron-10 · 2023-07-21T16:43:57Z

docs/experimental/viz_points_kde.py

+        window : window.RenderWindow,
+        texture_name : str,
+        target_actor : actor.Actor,
+        blending_mode : str = "None",
+        wrap_mode : str = "ClampToBorder",
+        border_color : tuple = (
+            0.0,
+            0.0,
+            0.0,
+            1.0),
+        interpolate : bool = True):


I am not sure whether we use typedhint for the function parameters as we basically mention them in our docs.
Maybe @skoudoro or @filipinascimento can clarify this.

Not yet, but we should go toward this direction.

Actually, I think we should make it a standard for FURY and start slowly the process for all module

Sounds good to me.

Okay, so should I leave it like that and keep using typedhint for my future commits or should I stay on today's standards and not use it for now until the make it standard decision for FURY is made @skoudoro?

docs/experimental/viz_points_kde.py

JoaoDell · 2023-07-21T17:42:23Z

@ganimtron-10 I have fixed and updated most of your recommendations. Indeed, typedhint isn't a FURY standard, but I will wait Filipi and Serge comments on that to change anything. Thanks for the comprehensive review!

JoaoDell · 2023-07-21T18:20:44Z

Also, I just realised there is a native colormap generator in fury.colormp named create_colormap() so I just updated it with that.

skoudoro

Hi @JoaoDell,

Thank you for this. This experimental is much more advanced than the other experimental.

I would recommend to start adding some text and explanation and then, we can move it to the examples folder. Maybe we can create a new section named "advanced Topic".

To discuss....

skoudoro · 2023-07-26T17:56:27Z

docs/experimental/viz_points_kde.py

+        window : window.RenderWindow,
+        texture_name : str,
+        target_actor : actor.Actor,
+        blending_mode : str = "None",
+        wrap_mode : str = "ClampToBorder",
+        border_color : tuple = (
+            0.0,
+            0.0,
+            0.0,
+            1.0),
+        interpolate : bool = True):


Not yet, but we should go toward this direction.

Actually, I think we should make it a standard for FURY and start slowly the process for all module

JoaoDell · 2023-07-27T14:08:35Z

Okay @skoudoro, I will be looking into the existing examples to understand how I should explain and detail this one so we can try using it as an example then, thanks for the opinion!

JoaoDell added 4 commits June 15, 2023 18:20

First experimental program for framebuffers

ad3fda1

Updating branch with master's latest changes

41bf2b1

Shader changes

1a0ae7c

Codespell fixes

3de5745

JoaoDell requested review from filipinascimento and devmessias June 19, 2023 14:58

JoaoDell added 2 commits June 22, 2023 14:25

Irrelevant changes

3c5e4c0

FBO setup working

d08f137

JoaoDell added 3 commits June 23, 2023 14:52

Updating branch with master latest changes.

0cc63f5

Render structuring and cleanup

ceb076f

Minor changes

c848f56

devmessias reviewed Jun 26, 2023

View reviewed changes

Code cleanup

620060a

devmessias reviewed Jun 27, 2023

View reviewed changes

JoaoDell added 4 commits June 29, 2023 16:55

minor changes

961dbd7

New approach

dc989b7

Offscreen rendering option

06ad2a8

KDE demonstration refactoring

1d937dc

JoaoDell changed the title ~~[WIP] Framebuffer Object Interface for FURY~~ KDE Rendering Experimental Program Jul 14, 2023

JoaoDell marked this pull request as ready for review July 14, 2023 17:04

redirected function calling

b611368

JoaoDell requested review from ganimtron-10 and tvcastillod July 14, 2023 17:09

JoaoDell requested a review from devmessias July 20, 2023 13:53

tvcastillod reviewed Jul 20, 2023

View reviewed changes

docs/experimental/viz_points_kde.py Outdated Show resolved Hide resolved

docs/experimental/viz_points_kde.py Outdated Show resolved Hide resolved

docs/experimental/viz_points_kde.py Outdated Show resolved Hide resolved

docs/experimental/viz_points_kde.py Outdated Show resolved Hide resolved

fix: General changes and cleanups

06dd4a1

ganimtron-10 suggested changes Jul 21, 2023

View reviewed changes

fix: Adapting to FURY's standards and some fixes

93ca6c2

fix: Now using native fury.colormap native generator

bf2b91a

skoudoro reviewed Jul 26, 2023

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

KDE Rendering Experimental Program #804

KDE Rendering Experimental Program #804

JoaoDell commented Jun 15, 2023 •

edited

Loading

skoudoro commented Jun 16, 2023

codecov bot commented Jun 23, 2023 •

edited

Loading

JoaoDell commented Jun 23, 2023

devmessias Jun 26, 2023

JoaoDell Jun 27, 2023

devmessias Jun 26, 2023

JoaoDell Jun 27, 2023

JoaoDell commented Jun 27, 2023 •

edited

Loading

devmessias Jun 27, 2023 •

edited

Loading

JoaoDell Jun 27, 2023

JoaoDell commented Jul 14, 2023

tvcastillod left a comment

ganimtron-10 left a comment

ganimtron-10 Jul 21, 2023

skoudoro Jul 26, 2023

ganimtron-10 Jul 26, 2023

JoaoDell Jul 27, 2023

JoaoDell commented Jul 21, 2023

JoaoDell commented Jul 21, 2023

skoudoro left a comment

skoudoro Jul 26, 2023

JoaoDell commented Jul 27, 2023

KDE Rendering Experimental Program #804

Are you sure you want to change the base?

KDE Rendering Experimental Program #804

Conversation

JoaoDell commented Jun 15, 2023 • edited Loading

skoudoro commented Jun 16, 2023

codecov bot commented Jun 23, 2023 • edited Loading

Codecov Report

JoaoDell commented Jun 23, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

JoaoDell commented Jun 27, 2023 • edited Loading

devmessias Jun 27, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

JoaoDell commented Jul 14, 2023

tvcastillod left a comment

Choose a reason for hiding this comment

ganimtron-10 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

JoaoDell commented Jul 21, 2023

JoaoDell commented Jul 21, 2023

skoudoro left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

JoaoDell commented Jul 27, 2023

JoaoDell commented Jun 15, 2023 •

edited

Loading

codecov bot commented Jun 23, 2023 •

edited

Loading

JoaoDell commented Jun 27, 2023 •

edited

Loading

devmessias Jun 27, 2023 •

edited

Loading