Hi!

What did you do this week?

I found out that for the manual sync feature to work correctly, the partitions need to span the missing frames introduced by setting a negative offset too, along with spanning any missing data at the end. That means only the tiling part should be changed, not the partitions. So I wrote a simple prototype for setting a positive sync offset for the HDF5 format’s get_tiles function. If the approach is good, a similar one can be used for setting an offset for rest of the formats, the difference being the way data is read. I also found out that this way, the K2IS format’s sync flag can be handled too. I also implemented reshaping in the GUI form this week.

What is coming up next?

I’ll work on handling the offset with a ROI, make changes to the API to handle additional info for the features and improve the prototypes. I’ll start working on generalizing them for n-D datasets after I get my mentors’ approval.

Did you get stuck anywhere?

No.

Hi!

What did you do this week?

Last week’s manual sync feature wasn’t correct as the partitions are supposed to span the missing data and ignore the tiles, not just skip the missing frames. So I fixed that issue this week, wrote some tests accordingly, added some UX improvements and finished implementing the manual sync feature for the MIB dataset. I wrote a prototype for reshaping a 3D dataset with flattened scan dimensions into 4D. It doesn’t actually reshape the data; instead, it reshapes the 4D slice objects into 3D and then reads the data using the slices.

What is coming up next?

I’ll implement the manual sync feature with tests for K2IS, HDF5 and the memory format this week. The K2IS format would need some more looking into as it is quite different from the other formats and allows specifying a sync flag. The prototype for reshaping might also need to be improved or completely changed, so I’ll work on that too.

Did you get stuck anywhere?

No.

Hello again!

What did you do this week?

I added tests and made some improvements according to the feedback I got on the manual sync feature for SEQ files. I also added some tests for the JS based GUI, and that’s how I was introduced to Jest. Rest of my time was spent on looking into ways to implement the dataset reshaping feature.

What is coming up next?

LiberTEM supports a variety of data formats and each of them have their own quirks. So my next challenge is to find the best approach to reshaping which is flexible enough to support all the formats.

Did you get stuck anywhere?

Yes, while working on the manual sync feature’s offset validation. I couldn’t figure out how to dynamically change the json schema on the server side. Also, the form implementation includes a nested Formik form field component and the error handling felt a bit off as for some reason, validation error messages weren’t showing at all. So either I’m doing it wrong or I’m not supposed to be doing it. But I learned quite a lot about Formik and I’ll be taking a look into it again after my prototype for reshaping is ready.

Hello!

I’m Anand, a third-year CSE undergraduate at GEC Palakkad and I’m going to be contributing to LiberTEM throughout the summer (and beyond). My project basically involves enhancing LiberTEM’s data pipeline to handle certain use-cases in processing 4D STEM datasets. I began contributing in February and it has been a great experience already as I was exposed to different parts of LiberTEM under the guidance of my mentors.

What did you do this week?

I worked on implementing a feature for manual synchronisation of Norpix SEQ files which would allow its users to specify an offset to skip frames or insert blank ones in case of acquisition/synchronisation problems. Apart from my really helpful mentors, I was fortunate to receive feedback from a LiberTEM user, who helped me find bugs and gave suggestions on the feature.

What is coming up next?

I’ll be working on handling missing/unfinished data to complete the manual sync feature. Once the feature is bug-free and approved for the SEQ format, I’ll start implementing it for the other data formats. On the side, I’m going to discuss and try writing prototypes to allow reshaping of nD-datasets-which-are-actually-4D.

Did you get stuck anywhere?

Yes, a few times. I made a silly mistake in my implementation and in addition to that, I initially misunderstood what ‘skipping’ frames actually meant. Then I came up with a working solution for the feature but as it was specific to just the SEQ format, I went ahead with a better solution my mentor suggested as it allowed the functionality to be shared between other formats too. While working towards the better solution, I got stuck on an error for about a day because I had not gone into detail about the data tiling concept. I finally decided to turn to my mentor who quickly helped me out with the concept with a code example. That’s also when I realized that using LiberTEM’s Python API in a notebook is a better way to try out prototypes than monkey-patching within the GUI.

I’m really excited to learn more and keep contributing to LiberTEM! Thanks to my mentors, the LiberTEM community, Python Software Foundation and Google for this opportunity!