Articles on xtanion's Blog

Week 14 - Morphing is here!

anandshivam54321@gmail.com (xtanion) — Mon, 10 Oct 2022 12:28:26 +0000

What did you do this week?

This week, I started with Multiple actor support in skinning and managed to do it successfully. Here's the preview using the BrainStem model.

Implementing multiple animation channels support (as seen in the Fox model). The get_skin_timelines() method now returns a dictionary of all animation channels with Timeline as their value.
We merged two PRs, #689 (colors from Material) and #643 (simple animations).
Added ability to load morphing information and create timelines from it. Here's a preview of the AnimatedMorphCube and AnimatedMorphSphere models:

What is coming up next week?

Cleaning and Rebasing Skinning animation PR #685.
Creating a PR for morphing code.
Multi primitive (actor) support in morphing.

Did you get stuck anywhere?

No, I didn't get stuck this week.

Week 13 - Multi-bone skeletal animation support

anandshivam54321@gmail.com (xtanion) — Tue, 20 Sep 2022 20:33:59 +0000

What did you do this week?

This week I fixed all the issues with skeletal animations, and we got to see our first render of skeletal animation :).

Implemented a hierarchical timeline system (i.e., one timeline for each bone, and the timeline will contain its parent timeline in a hierarchy).

- I figured out that we don't need to apply the parent transform as we're applying it to the vertex data while forming the actor. So the skin matrix becomes SkinMatrix = InverseBindPose * BoneDeform where BoneDeform = CurrentBoneTransform * ParentBonetransform.

Here's a preview using the CesiumMan model:

What is coming up next week?

Add tests for simple animation PR #643.
Multiple actor support for skeletal animation.
Take a look at Morphing.

Did you get stuck anywhere?

No, I didn't get stuck this week.

Week 12 - Adding skeleton as actors and fix global transformation

anandshivam54321@gmail.com (xtanion) — Tue, 20 Sep 2022 20:32:00 +0000

What did you do this week?

I started this week by fixing the segmentation fault for the skinning test. I could not fix this as of now.
I imported the RiggedFigure model into the Blender and I was quite impressed with how it visualizes each bone. So, I created a function that creates an arrow actor in place of a bone, and by applying the correct transformation, we get the bones of a model in place. Here's a quick preview of bones in Blender vs bones in FURY:

After having the bones actor, I noticed that some bones are not aligned correctly. It was happening due to multiplication of the same transformation matrix twice.
I also created a function that calculates the total transformation of a node, it eliminates the need to use timeline.get_value in get_skin_timeline function.

What is coming up next week?

It seems like I fixed everything, but we are not getting the correct model at timestamp 0. We need to find the cause and fix it!
Cleaning the Simple Animation PR #643, and merging it.

Did you get stuck anywhere?

While applying the timeline, we were getting the identity matrix for timestamp 0.0s, it was set to a new value before. We figured this in our meeting that it's happening due to some model's animation not starting from 0.0s.

Week 11 - Multiple transformations support and adding tests

anandshivam54321@gmail.com (xtanion) — Tue, 20 Sep 2022 20:30:14 +0000

What did you do this week?

As decided last week, I added support for multiple animations at the same timestamp. But this didn't solve the animation problems of RiggedSimple model.

I figured out that I was not applying the global transform of joints in the skinning matrix causing the joints to move to a new position. I referred to this answer by irradicator.
We had to invert the multiplication order (ie, np.dot(skin_mat, vertex) instead of np.dot(vertex, skin_mat)) for this model. We still need to figure out from where we get the multiplication order of the two.

<iframe height="390" src="https://user-images.githubusercontent.com/74976752/187798821-7306cb13-dfc8-477c-8f86-1f5a6181f08a.mp4" width="600"></iframe>
I also tried to create a custom Interpolator that does all calculations required for vertex skinning. However, we couldn't get this working. I got a few suggestions from Mohamed in our meeting, I'll try to implement those.
Added tests for animations.

What is coming up next week?

Adding more tests.
Fixing the Interpolator.
Order of multiplication in the skinning matrix.

Did you get stuck anywhere?

No, I didn't get stuck this week.

Week 10 - Multi-node skinning support

anandshivam54321@gmail.com (xtanion) — Thu, 01 Sep 2022 00:45:55 +0000

What did you do this week?

As we figured out that the SimpleSkin model is not animated as intended. This week I started working with that model; I figured out that we need to apply the InverseBindMatrix to each timer callback. I had a hard time figuring out what inverse bind matrix actually does. Here's a good blog that answers that.

In short: The InverseBindMatrix "undoes" any transformation that has already been applied to your model in its bind pose. Note: Inverse bind matrix shouldn't be applied with weights.

I got the SimpleSkin model to work perfectly. Here's a preview:
Support for multiple transformations is still missing. However RiggedSimple is also working fine, except that It doesn't interpolate translation and scaling matrices.

What is coming up next week?

Adding tests and adding support for combining transformation matrices for the same timestamp.
Write an Interpolator that applies skinning to the mesh internally.

Did you get stuck anywhere?

No, I didn't get stuck this week.

Week 9 - First working skeletal animation prototype

anandshivam54321@gmail.com (xtanion) — Wed, 24 Aug 2022 14:47:44 +0000

What did you do this week?

This week I had the first working example of skeletal animation ready. I was able to render the SimpleSkin model. Here's a quick preview:

<iframe height="390" src="https://user-images.githubusercontent.com/74976752/184981715-a83a8a36-f2bf-4ff1-9366-0c302dcf3f9b.mp4" width="600"></iframe>
I wrote a custom interpolator (just like the tangent cubic spline interpolator) for interpolating two transformation matrices.

What is coming up next week?

Generalising the skinning code to work with other models as well (it supports only the SimpleSkin model as of now).
Creating a custom interpolator to interpolate more than 4 matrices at once.

Did you get stuck anywhere?

No, I didn't get stuck this week.

Week 8 - Fixing animation bugs

anandshivam54321@gmail.com (xtanion) — Wed, 24 Aug 2022 14:32:31 +0000

What did you do this week?

This week I had to fix the transformation issue in glTF animation. As discussed in the last meeting, I disabled vertex transformation and applied the transformation after the actor was formed.

Creating transformation matrix using the TRS data from nodes.
Extract Translation, Rotation, and Scale matrices from the transformation matrix.
<iframe height="360" src="https://user-images.githubusercontent.com/74976752/184015060-48d79f0d-1377-4f69-b147-cd53448ccf02.mp4" width="640"></iframe>

I also started investigating more on the skinning animation.

I created functions to extract and apply the InvertBindMatrices to the vertices.
Also, applying the globalTransformMatrices to the mesh.

What is coming up next week?

Skeletal animations support

Did you get stuck anywhere?

Even after applying the transformation to the actor, after it's created. Some models did not behave as intended. We still need to figure this issue out and fix it.
<iframe height="360" src="https://user-images.githubusercontent.com/74976752/184015110-035f1ce7-5a5c-4480-b49c-ae575d4b7ccd.mp4" width="640"></iframe>

Week 7 - Fixing bugs in animations

anandshivam54321@gmail.com (xtanion) — Tue, 09 Aug 2022 21:43:20 +0000

What did you do this week?

This week I started with implementing scaling to the animation example.
I had a meeting with Mohamed in which we discussed the rotation issue, and we found out that glTF uses Slerp interpolator instead of the LinearInterpolator. Using Slerp interpolator for rotation fixed the rotation issue.
Another issue we faced was with the multi-actor system. Some actors weren't rotating or translating as intended. This was caused because the transformations were applied to the polydata before creating an actor; Mohamed suggested applying transformation after the actor is created from polydata (keeping the center to origin).

Expected animation: Interpolation Test Sample

Created functions to return a list of animation timelines and apply them to the main timeline for keyframe animations.
CubicSpline has not been implemented to glTF animation yet since it works differently than other Interpolators (takes tangent input to smoothen the curve) but it'll be done before our next meeting.

What is coming up next week?

Adding skinning animations support

Did you get stuck anywhere?

I still need to figure out how to apply the transformation matrix to the actor and not to the polydata.

Week 6 - Extracting the animation data

anandshivam54321@gmail.com (xtanion) — Tue, 02 Aug 2022 18:05:40 +0000

What did you do this week?

This week, it was all about reading docs and extracting the animation data from buffers.
Currently, the glTF class can extract simple node transformation and morphing data, as they are stored in the Animations of the glTF file.
Skinning (Skeletal Animation) data is stored in Nodes inside the skin parameter. We shall be able to load that before our next meeting on Wednesday.
Created a tutorial using keyframe animations (#626) and adding multiple timelines into a main timeline as suggested by Mohamed.
<iframe height="400" src="https://user-images.githubusercontent.com/74976752/180841262-7356204e-2097-4ea4-ab71-39f04a1e7a07.mp4" width="550"></iframe>

What is coming up next week?

As of now, we've decided the following:

Create a custom Interpolator (for simple node transformations, morph and Skeletal animations).

Did you get stuck anywhere?

No, I didn't get stuck this week.

Week 5 - Creating PR for glTF exporter and fixing the loader

anandshivam54321@gmail.com (xtanion) — Mon, 25 Jul 2022 16:43:32 +0000

What did you do this week?

Finalised the glTF export PR #630., adding tutorial, docs, and tests for all functions.
Added support for exporting lines and dot actors.
primitive.modes is now set if the rendering mode is a line, dot, or triangle.
Working on importing different primitive modes, the current glTF importer can render only triangles.

What is coming up next week?

This week I'll be working on the following:

Get the PR #630. merged by this week.
Loading the animations (simple, morph, and skeletal) data as dictionaries from the glTF model so that it can be sent to the timeline.
Try different examples on Mohamed's PR (#626.) and try running glTF animations if time permits.

Did you get stuck anywhere?

There wasn't any model in the Khronos glTF samples repository that uses the LINE or POINT modes. So I had to rely on the models that I exported using the glTF exporter.

Week 4 - Finalizing glTF loader

anandshivam54321@gmail.com (xtanion) — Tue, 19 Jul 2022 14:36:49 +0000

What did you do this week?

This week I had to travel back to my college since the summer vacations have ended. I had a coding session with Serge this week, we modified the exporter function and looked at some bugs I faced.

We managed to use the io.load_io function. There was a strange issue with loading png files. I had to convert the PIL Image to RGB format, and it fixed the issue, turns out png images are stored as P (pallet) mode.

I also added the glb format support to the importer. While loading a .glb model, I noticed that the image data is also stored in the buffer and there can be a bufferview index to get the buffer.

During this time, I also tested the glTF loader with all models in the KhronoosGroup/glTF-samples repository. Here's the table of models working well: see file.

Model Name	Status	Error Type
BoxInterleaved	No	"BoxInterleaved.gltf: shapes (4,4) and (7,12) not aligned: 4 (dim 1) != 7 (dim 0)"
Box	Yes
SpecularTest	Yes
AnimatedCube	Yes
NormalTangentMirrorTest	Yes
AnimatedTriangle	Yes
SpecGlossVsMetalRough	No	SpecGlossVsMetalRough.gltf: 'NoneType' object has no attribute 'baseColorTexture'
CesiumMilkTruck	Yes
VC	Yes
WaterBottle	Yes
AnimatedMorphCube	Yes
Sponza	Yes
SciFiHelmet	No	SciFiHelmet.gltf: Python int too large to convert to C long
Triangle	No	None: file not downloaded
IridescenceMetallicSpheres	No	None: file not downloaded
Corset	Yes
Cube	Yes
TextureLinearInterpolationTest	Yes
SimpleMeshes	Yes
Lantern	Yes
TextureTransformMultiTest	Yes
TextureSettingsTest	Yes
LightsPunctualLamp	Yes
DamagedHelmet	Yes
CesiumMan	Yes
Cameras	Yes
BarramundiFish	Yes
MetalRoughSpheresNoTextures	Yes
EnvironmentTest	No	None: file not downloaded
MosquitoInAmber	No	MosquitoInAmber.gltf: buffer is smaller than requested size
BoxTexturedNonPowerOfTwo	Yes
BrainStem	Yes
SimpleMorph	Yes
OrientationTest	Yes
BoxAnimated	Yes
StainedGlassLamp	Yes
TextureTransformTest	No	None: file not downloaded
ClearCoatTest	Yes
IridescenceLamp	No	None: file not downloaded
DragonAttenuation	No	DragonAttenuation.gltf: buffer is smaller than requested size
RecursiveSkeletons	No	RecursiveSkeletons.gltf: cannot reshape array of size 120 into shape (9)
Suzanne	Yes
RiggedSimple	Yes
TextureEncodingTest	Yes
2CylinderEngine	Yes
NormalTangentTest	Yes
MorphStressTest	Yes
IridescenceDielectricSpheres	No	None: file not downloaded
TextureCoordinateTest	Yes
Duck	Yes
AlphaBlendModeTest	Yes
TriangleWithoutIndices	Yes
MultiUVTest	Yes
BoomBoxWithAxes	Yes
Box With Spaces	No	Box%20With%20Spaces.gltf: 'NoneType' object has no attribute 'shape'
SheenCloth	Yes
ToyCar	No	None: file not downloaded
MaterialsVariantsShoe	No	MaterialsVariantsShoe.gltf: buffer is smaller than requested size
IridescentDishWithOlives	Yes
VertexColorTest	Yes
SheenChair	Yes
Fox	Yes
AntiqueCamera	Yes
TransmissionTest	No	None: file not downloaded
TransmissionRoughnessTest	Yes
BoxVertexColors	Yes
ReciprocatingSaw	Yes
MorphPrimitivesTest	Yes
MetalRoughSpheres	Yes
GearboxAssy	Yes
TwoSidedPlane	Yes
Buggy	Yes
SimpleSparseAccessor	Yes
BoxTextured	Yes
UnlitTest	Yes
SimpleSkin	Yes
FlightHelmet	Yes
Unicodeâ¤â™»Test	No	Unicode%E2%9D%A4%E2%99%BBTest.gltf: 'NoneType' object has no attribute 'shape'
Avocado	Yes
InterpolationTest	Yes
GlamVelvetSofa	Yes
RiggedFigure	Yes
BoomBox	Yes
EmissiveStrengthTest	No	None: file not downloaded
AttenuationTest	Yes
AnimatedMorphSphere	Yes
IridescenceSuzanne	Yes

What is coming up next week?

Adding tests and merging export function PR.
Start working on Simple Animations.

Did you get stuck anywhere?

To create a texture, we needed the RGB values, However .png images were returning a 2D array when read using PIL. It is fixed by
pygltflib's load method doesnot handle glb files very well. It does not contain the buffer uri. I used glb2gltf method as of now.

Week 3 - Fixing fetcher, adding tests and docs

anandshivam54321@gmail.com (xtanion) — Fri, 08 Jul 2022 16:10:58 +0000

What did you do this week?

The first task for this week was to fix the glTF fetcher. We noticed that while running tests it reaches the API limit. So we decided to use a JSON file that contains the download URLs for all models.
Created a function to generate JSON files with download URLs for all models, which can be found here.
Modified the tests and fetcher.py to download using the JSON URLs and merged the PR #616.
Added docstring for all functions in #600. Wrote tests for transformation functions.
Multiple actor support in glTF export. Added tutorial and created a new branch for the same. Here's an example: (The glTF model below is created using FURY and rendered back using the glTF class)

What is coming up next week?

I will be doing the following:

We still need to figure out how to get indices from vtkSource actors
I'll be finishing the exporting functions and adding tests for the same and I should be able to create a mergeable PR for the exporting function.

Other tasks will be decided after the meeting.

Did you get stuck anywhere?

Week 2 - Improving Fetcher and Exporting glTF

anandshivam54321@gmail.com (xtanion) — Mon, 04 Jul 2022 10:23:21 +0000

What did you do this week?

This week I worked primarily on the glTF fetcher and exporting scenes to a glTF file. I added tests and docstrings for all functions. I modified the fetch_gltf function to return the downloaded files. As we planned, the PR for the glTF fetcher was merged this week.

I fixed the color issue of the glTF exporter by manually appending the color data from the actor to the polydata. But there's another issue raised while using actors from vtkSource. The utils.get_polydata_triangles(polydata) method only works with primitives and it raises an error when used with vtkSource actors.

Textures and Cameras can now be added to the glTF file. However, it supports baseTexture only. I'll be working on materials support later on.

What is coming up next week?

Saving all models download link (from the Khronos glTF-Samples-Models repository) to a JSON file, create a separate branch and add the download script.
Add tests and docstring for PR #600.
Create a PR for glTF exporter.

Did you get stuck anywhere?

I managed to fix colors in polydata by adding them manually, but it raised another issue with indices (triangles) of the actor weren't of the correct shape. We decided to take a look at it later.
Due to Github's API limit, it raised an error due to limits exceeding. We decided to save a JSON file with all model names and their download links. Then use that to download the model.

Week 1 - A Basic glTF Importer

anandshivam54321@gmail.com (xtanion) — Wed, 29 Jun 2022 15:02:07 +0000

What did you do during the Community Bonding Period?

In the community bonding period I met with my mentor Serge Koudoro, along with other mentors in FURY and gsoc contributors. We discussed about my project and set up project timeline on github projects section. As my project (glTF Integration) and Keyframe animations were related so we discussed on that too.

I read documentation of various glTF loading libraries (pygltflib, panda3D-gltf and pyrender) in python and tried to understand how do they work. I started creating a basic glTF loader, it was using dataclasses to store json data. I created polydata for each primitive in a mesh, each polydata will contain the vertices, triangles, normals. To apply textures properly, I had to apply texture coordinates to polydata and flip the texture along X axis by 90 degrees.

What did you do this week?

After discussing on pros and cons of various glTF libraries we decided to use pygltflib to handle json to python dataclass conversion. This week I reshaped PR #600 to use pygltflib. I also modified the code to handle multiple base textures. While experimenting with textures, I accidentally applied normals to the polydata and discovered that it makes the surface look much smoother, however in few models it results in a darker model which reflects almost no light. So I made it optional for now to apply normals in a model.

I also created a basic fetcher (PR #602) to get glTF models from Khronos group's glTF sample repository. I also made this function asynchronous, as Serge suggested me to use asyncio and aiohttp to make the API callbacks asynchronous and it decreased the downloading time of multiple models.

I am also working on exporting scene to a glTF file. Using pygltflib I am converting the python dataclass to a json like structure.

What is coming up next week?

Create a PR for the fetcher function, add tests, fix bugs and merge it by the end of the week. Fix the colors issue in the glTF exporter. Add texture and camera in glTF exporter and create a PR for it.

Did you get stuck anywhere?

var: list[int] was causing the error TypeError: 'type' object is not subscriptable since the ability to use the [] operator on types like list was added in 3.9+. I had to modify the dataclasses and use Typing.List instead.
Texture in actors weren't applied correctly. I tried flipping the texture image by 90 degrees using gimp and the texture worked nicely. The reason for this issue was the coordinate system of glTF texture format which has origin (0, 0) at top-left and (1, 1) in the bottom-right corner, Where as in vtkTexture we want coordinate (0, 0) and (1, 1) at bottom-left and top-right corner respectively. Here's an example of the issue:

Praneeth told me that some models with multiple nodes and multiple meshes weren't loading correctly. The reason for it was the way SCALAR data was extracted from the binary blob, I had to change some variables in get_accessor_data method and everything started to work just fine.

My Journey to GSoC 2022

anandshivam54321@gmail.com (xtanion) — Wed, 29 Jun 2022 14:44:55 +0000

About Me

Hi! I'm Shivam, currently pursuing my bachelor's (expected 2024) in Production and Industrial engineering from the Indian Institute of Technology (IIT) Roorkee.

I was introduced to the C programming language through the Introduction to programming course in my first year of college. I always liked computers, and I was overwhelmed by printing "hello world" in the terminal. The course continued to teach us data structures and algorithms. It was a lot of fun competing with friends over a programming question. I learned python on my own and did a lot of projects. After learning basic programming, I participated in many hackathons and won some, but lost a lot. Coding was enjoyable, and I knew this was what I wanted to do for the rest of my life. In my second semester, I learned about open-source, git, and GitHub. I came across some computer graphics enthusiasts in my college; they told me that they created a 2D game engine on their own using OpenGL. This gave me enough motivation to learn OpenGL and basic computer graphics.

Intro to Open-Source and GSoC

In October 2021, I participated in Hactoberfest and completed it successfully. I learned a lot about Free and Open Source Software during this time. I heard about GSoC around this time from one of my seniors and asked him about the program.

I went through the previous year's accepted organizations list and shortlisted 2-3 organizations. I started contributing to a few projects based on android and Kotlin, but I lost interest after some time. It was about December, and I hadn't chosen any organization yet.

I heard about FURY from one of my seniors. I started looking at its docs. Picked up some books from the library to brush up on my concepts of computer graphics. The documentation of FURY is remarkable. I created my first pull request by improving one of the tutorials, and It got merged in a few days.

I started looking at the source code of FURY and tried to understand it again. I read the API reference part of FURY docs this time along with the documentation of VTK. I also tried to solve some of the open issues. I created a few pull requests for the same. My first contribution was in the fury primitive class, I created a sphere primitive (inspired by the sphere in Blender).

I started looking at bugs and improvements that can be made to the source code and created PRs for the same. During this time I was also learning how computer graphics work from the UCSC lectures & OpenGL by Victor Gordon.

After the accepted organizations were announced, I was so happy that FURY got selected this year for GSoC and went straight to the Ideas list. The first project idea was glTF Integration. I heard about the glTF file format before but didn't know how it works. So I went straight through the reference materials provided on the wiki. I read a lot of documentation by the Khronos group. I also tried to implement a basic glTF loader during this time using VTK's built-in glTFReader.

I also liked the Improve UI drawing project idea (I had a basic understanding of line drawing algorithms and rasterizations at that time) and thought I'll make a proposal for that too. After completing my proposal for glTF integration I started my research on this one too.

I started writing the proposal early so that I could get my proposal reviewed at least once with the project mentors. I almost forgot the fact that I had my end-term examinations at hand (Somehow I managed to pass all of my courses), I kept contributing to the project till the end of April 2022.

Code contributions:

The Day

May 18: I was a bit anxious since on May 18th the GSoC website was showing my proposal for glTF integration had been rejected. (p.s. maybe it was a bug of some sort that they fixed later on).

May 20: I woke up very early in the morning and started checking the contributor's profile. I kept checking for new emails but didn't receive any emails that day. "The result will be announced at 11:30 PM, isn't it? " My dad said, It was 8:00 AM IST. I called my friends in the night to join me on discord, I was talking to them and refreshing the GSoC site at the same time. One of my friends shouted that he got selected in NumFocus. I refreshed the page again, my proposal for glTF Integration was accepted. I can't express what I felt at that moment. I told my friends and my parents, they were happy for me and I got a lot of blessings :). I received an official email the next day.

My Journey to GSoC 2022

anandshivam54321@gmail.com (xtanion) — Wed, 29 Jun 2022 14:14:31 +0000