Articles on jaladh-singhal's Blog

Week-12: Developing Analytical Web Apps

jaladhsinghal@gmail.com (jaladh-singhal) — Mon, 19 Aug 2019 20:16:25 +0000

Hello folks,

This week I finally created a web app from an interactive Jupyter Notebook. Sounds unreal but no that's the magic of Voila! ✨

What did I do this week?

Concluding my research on converting Jupyter Notebook into Web App, I found out about Appmode and Voila which can hide input cells & make notebook appear as an App. Then it can be deployed on web by hosting it on a cloud service capable of running Notebook Server like mybinder or Heroku. I analyzed pros & cons each of these options as per our requirements, and decided to use voila + mybinder for the prototypical web app I needed to develop.
I read the relevant docs and created necessary configuration files in my forked repository to render the interactive spectrum notebook (that I created last week) as Voila app, when it is served to users on mybinder. You can also try out interactive plotting of the spectrum by checking this web app.

I took a off for 2nd half of this week so I did a limited amount of work, but a lot of fun 🙃 since I was hiking in the Aravalli mountain range! ⛰️

What is coming up next?

Yet this was last week of Coding period but there's a lot more ahead to be done. I've planned to continue my project of developing interactive photometry interface post-GSoC, as we've finally started to move in that direction.

Did I get stuck anywhere?

Yep! when I tried to deploy the Voila app also on Heroku (besides mybinder), it became really difficult to configure it for running our package since there is no default conda support on Heroku. Also I couldn't find a way to define post build steps in it like I did in mybinder. Since the created web app is already working on mybinder, I left this problem to be solved later (post-GSoC).

What was something new I learned?

📄 Purpose of Configuration files when deploying stuff on Cloud: To build our repo on a cloud service (for deployment purposes), configuration files let us define how to build it. They are as powerful as allowing us to run a bash script to perform any set of actions we want, when building the repo. While building my repo on mybinder for hosting the notebook as web app, I came to know about various types of configuration files it offer. I really liked the option of postBuild file i.e. basically a bash file which I used to define the commands I want to execute, once my repo is built.

🔮 Jupyter Extensions that can convert Notebook into Web apps: Appmode and Voila are the two extensions that I found for this purpose. Both of them work really neat by hiding the input cells & displaying the output of notebook as a web app. I preferred to use Voila over Appmode since it is more than a extension, it can also be used independently as a standalone tornado application. Voila also provides additional feature of templates that can be used to customise the look of web app.

📤 Deploying Python Apps on Heroku: Heroku makes deploying python apps really a breeze. We just need a Heroku account, couple of files (runtime.txt, requirements.txt, ProcFile) in our repo and then just create a heroku instance & push the code to see our app alive on web. Though there are several other things we have to setup at Heroku, if our repo requires some additional configuration (this is the part where I found mybinder super easy since it provides various inbuilt options to do so).

Thank you for reading. Stay tuned to know more about my journey!

Week-11: From Interactive Notebooks to Web Apps

jaladhsinghal@gmail.com (jaladh-singhal) — Sun, 11 Aug 2019 20:25:02 +0000

Hello folks,

This week we finally came to the part for which I was eagerly waiting, i.e. developing analytical web interfaces (web app stuff) so that users can easily interact with the data to get the visualizations & results they need. 🙃 I played around with visualization stuff and dived into researching the technologies present around Jupyter to make web apps from interactive notebooks. This way the developed interfaces can be accessed both locally as well as on web. 😍

What did I do this week?

I apparently fixed the problem that I shared in last week's blog (PR builds failing since I made pipeline to access filter data from artifacts for executing notebooks). I experimented & found that when I manually build the PR commits (being a member of my Org), the build passes which was failing when getting triggered as a PR build (from me as a contributor). So this made it clear that we can merge the PR in our codebase. And PR build failure was due to a problem of authorization when calling REST API for which, I also opened an issue (called as feedback there) in Azure DevOps Developer Community. I followed up with their team, need to try some things but in future since it's not so prior when more important things are ahead!
Since all major work to make wsynphot stable to use is done, I finally got chance to do the work I wanted to do: creating analytical web interfaces for our packages (major part of my proposed project)! Our main aim is to build a web app for calculating photometry of stars, which lies at the intersection of both of our packages: starkit & wsyphot. So I started using starkit to plot star's spectrum from grids (HDF5 files that contain relevant data of star) and played around with it.
- I created a table in starkit docs, listing all test grids created by mentors that are available at Zenodo for open access. I also needed to find out the bound values of each grid's characteristics in the table, which improved my understanding how starkit works with grids.
- Then I followed up documentation of bqplot & ipywidgets to create an interactive plot of the spectrum by using sliders for changing the values of grid characteristics.
Now before advancing in plotting, my mentor told me to try converting this simple interactive plot (a prototypical interface) into a web app. So I researched about possible ways to convert an interactive Jupyter notebook into a web app, and found out about several interesting things to implement. 😮

What is coming up next?

Now I'll start implementing the several ways I found to create web app from Jupyter notebook. Then we will compare & decide which one should we use, before I continue developing more interactive interfaces.

Did I get stuck anywhere?

Not any such, this week was more about exploration & experimentation.

What was something new I learned?

📊 Bqplot: It is a 2-D plotting for Jupyter, based on Grammar of Graphics. Thus it is a perfect choice for integrating visualizations with other Jupyter interactive widgets to create integrated GUIs - which is what we want to develop at StarKit.

🚀 Variety of services Project Jupyter provides: From interactive UI controls to a tool for running notebooks in the cloud, it is really powerful & amazing!

IPyWidgets are used to interactively visualize and control changes in the data to instantly see how it affects the results.
Binder allows us to create custom computing environments on cloud that can be used to run & share notebooks with remote users on web. Popularly used is mybinder.org which is a pre-existing deployment of BinderHub that is built on JupyterHub.
JupyterLab is the next generation of notebooks interface providing an IDE for data science workflow!

Thank you for reading. Stay tuned to learn about my upcoming experiences!

Week-10: Executing notebooks from the Pipeline

jaladhsinghal@gmail.com (jaladh-singhal) — Mon, 05 Aug 2019 12:00:45 +0000

Hello folks,

This week was pretty tough. A seemingly simple task of setting up docs CD pipeline to execute notebooks ate up entire week, thanks to the complexities of DevOps! 🙄

What did I do this week?

To execute our notebooks from docs CD pipeline, we need to make cached filter data available at the VM as a pre-build step. So I decided to store it at Azure as artifact (Universal Package) so that each time pipeline runs, it can instantly download artifact into VM and then also update it before using. Since it's updating the artifact, I decided to also publish it back to artifact feed thereat so that next time we get a more up-to-date data.
- For 1st build of such a pipeline, I needed to make sure that artifact is already present in feed - which means publish the artifact without pipeline but from Azure CLI, locally. The authentication in Azure CLI was really cumbersome but I figured out how to use a PAT for it & then published the filter data as artifact.
- Then I wrote script steps in pipeline to download artifact & bring it in right directory. The challenge here was to make pipeline download the latest versioned artifact from feed. After some mind-boggling research, I found how to achieve it with Azure REST API, but problem was again authentication while calling the API. On trying to solve it next day with a clear head, I found the solution that was right in front of my eyes in an already opened SO thread which I was overlooking!
- I wrote script to conditionally publish the filter data as a newer versioned artifact if it got updated. Here I needed to write a python command that calls update function & returns update status back to bash script. But due to logger enabled in function, the returned value was entirely messed up - I fixed it up by disabling logger.
- I also improved the versioning of artifacts by using date as version but there were conflicts with SemVer accepted by Azure, which again took time to manipulate date into acceptable version.
Next I needed to make sure that executed notebooks give right outputs.
- The matplotlib plots didn't appear in rendered noteboook. After some searching & digging, I found it was because we were interactively plotting graphs using %pylab notebook. By using %matplotlib inline magic which works non-interactively I made the plots appear.
- I also cleaned some unnecessary data from quickstart notebook & made documentation more clear.

What is coming up next?

There's still a problem with pipeline, it is failing for PR builds although it works fine for my fork. I will try to fix that and then we will possibly move to starkit, where we can integrate it with wsynphot (on which I am currently working) to produce an interface for calculating photometery.

Did I get stuck anywhere?

Yes, it was these unexpected problems that took me finishing off this task of making pipeline execute the notebooks, an entire week! But I eventually solved all of them except that PR build failing problem.

What was something new I learned?

⚙️ This week made me learn many new things about Azure DevOps, like:

Azure CLI & how to use & authorize it to manage Azure resources from another (local) system
Azure REST API i.e. a really powerful API which lets you create/retrieve/update/delete the Azure resources
System.AccessToken which is a special predefined variable that used as OAuth token to access the REST API
Unlike powershell task which runs Windows PowerShell, we can use pwsh task on LINUX VMs since it runs PowerShell Core
Conditionally run a step in Azure by using conditions option.

🔡 While passing variable from a child process (python script) to parent shell (bash), make sure that you only write the value which you want to be passed on stdout. This means keep a check that there are no such function calls with print or logging statements in your script other than value you want to pass by printing it.

🧐 Openness for the strange options while researching: When we search for the solution of a problem on internet, lot of new & weird information comes before us which we just skim enough to decide that it is not for our case. But even then if we try to understand it, we may get how to make it work for our case by experimenting. Same happened when I was looking for how to authorize my build for Azure API call. On a SO thread I found a powershell script for it but I didn't bother to understand it thinking that powershell script can't be of any use to a LINUX VM. But when I eventually found the solution (SystemAcessToken & pwsh for LINUX), I was like: After all this time answer was right in front of my eyes and I was searching it here & there, by not caring to give it some minutes to understand!

Thank you for reading. Stay tuned to learn about more interesting things with me!

Week-9 (Evaluation Period-2): Finally Cached'em all

jaladhsinghal@gmail.com (jaladh-singhal) — Mon, 29 Jul 2019 19:26:40 +0000

Hello folks,

Evaluation week knocked again to remind us that it has been 2 months since we started! This week I finally finished up developing & testing the update cache mechanism to keep our cached filter data up-to-date. The challenges I encountered in writing IO tests for it, unlocked an unknown realm of unit-testing for me. 😮

What did I do this week?

I finished up my work on cache_filters module that handles cached filter data, by completing the implementation of update cache mechanism:
- Created configuration functions that reminds user to update their cache if >1 month since last update, by reading/writing cache updation date in configuration file.
- Improved design of codebase by decoupling the concerning modules (this was because I got stuck in circular import problem as soon as I added my code to the codebase)
- Documented the use of update function
I began to write tests for the update function but again this was like download function which downloads a lot of files (~1K) from SVO (our data source) so I could not test it directly but only its helper function that it calls. But I was not satisfied with this approach as my function was not getting entirely covered so I searched extensively to find is there a way to change the code of my function only when testing it. And fortunately I found about test mocking & monkeypatching. 🧐
- Hence I drastically reduced the huge data dependency of my update & download functions by creating a mock function to download only a very less no. of files (<10). And then I injected it each time these functions are tested by using pytest monkeypatch fixture.
- Also I realized that for preventing the manipulation of data in unexpected ways, I need separate directories while testing my I/O functions - a temporary directory for the functions that write data to disk, and tests/data directory for the functions that read data from disk. So I used pytest tempdir_factory fixture for the tests of my update & download functions which write the data.

What is coming up next?

My mentor told me to make nbsphinx execute the notebook each time it build the docs so as to check that code in our package works fine. So I need to configure our docs building pipeline to consume cached filter data when executing the notebooks which I can possibly do by storing the cache at Azure as an artifact.

Did I get stuck anywhere?

Yes I got stuck in several problems this week like circular imports, test a function with huge dependencies, etc. But by patiently searching solutions & trying them, I resolved all of those problems, ultimately improvising our package.

What about Evaluations?
I can't believe it has been almost 2 months in this GSoC journey! When I look back, I see lot of work has been done and a lot more is waiting to be done. And obviously I passed my 2nd evaluation as I'm writing this blog post. My mentor is on vacations so unlike 1st evaluation, he couldn't write back a feedback to me which he said to provide later. 😊

What was something new I learned?

📑 Purpose of Configuration file in a package: It gives users control of various parameters used in the package without delving into the source code to make the required changes. It is handled by a configuration system in the package e.g. a config.py file to use & treat the data defined by user.

🔁 Circular dependencies: When you import some entity in a module from another module which inturn import some other entity from the former module - you get stuck in a circular import leading to ImportErrors & AttributeErrros! It is due to a bad design such that modules of your package become tightly coupled. It can be resolved by carefully tracing the import logic of your package to root out the problematic entity (a variable, function or object). Then it can be usually fixed by placing that entity in right module (or even in a new module) where it should logically belong, and then import it.

📥 Unit testing a download function: I was confused about what to assert in the test of download function so I searched & read about it. It may appear that we can test whether connection has made to data source and data file is fetched without errors or not. But it is not required as exception handling mechanism in our function will do that. The unit testing means we assert what function should has done instead of how it has done, so we should assert that downloaded file is right directory and its contents are as expected (by reading & comparing it with expected data).

🐒 Test mocking & monkeypatching: Mocking means creating such objects that imitates the behavior of real objects, and dynamically changing a piece of software (module, object or function) at runtime is referred as monkeypatching. So while testing, we can inject (or substitute) the complex dependencies with the mock objects so that our test only focuses on code of the unit we're testing, instead on its dependencies. This also helps in making tests run faster as it minimizes the real amount & interaction with dependency. This can be implemented by using pytest monkeypatch fixture as explained wonderfully in this blog.

Thank you for reading. Stay tuned to learn about more interesting things with me!

Week-8: Remodeling the data access Mechanism

jaladhsinghal@gmail.com (jaladh-singhal) — Sun, 21 Jul 2019 20:41:45 +0000

Hello folks,

This week I worked as a package mechanic, developing two mechanisms for our package wsynphot. 😜 Firstly I migrated the mechanism used by our package to access filter data from a HDF storage to the cache on disk. Then I started setting up an update mechanism for the cached filter data to keep it up-to-date. Let me share how!

What did I do this week?

I changed the data access methods in base module of our package to use filter data cache (which is handled by the module I created last week), instead of using HDF data file. This seemingly simple task, was successfully completed after doing a couple of unanticipated tasks:
- Fixed a problem in base module to access a calibration file. I dug rabbit holes to find the cause of this annoying problem - thanks to the search history option of git and the discussions I had with my mentor, that I finally solved it.
- Fixed the all-of-sudden failing IO tests due to a recent update in filter data at SVO.
- Integrated the functions from cache_filters module into the base module and made sure it works fine.
- Removed the older code that was meant for handling HDF storage of filter data, along with some other necessary cleanup in the package.
- Re-documented quickstart notebook due to these changes.
Next I started working on cache updating mechanism to ensure that filter data on user's disk is up-to-date with that at SVO (our data source).
- For this I came up with a logic to compare filter index in cache with that at SVO online and then calculate the filters to add/remove in cache by using set difference operations. I felt great about it realizing how set theory helps in solving practical problems! 😌
- I created a function for implementing the same and solved several problems I encountered in the process, like data type conflicts (byte vs unicode), etc.
- I tested the function against a recent update at SVO, but I noticed discrepancies in the result which I traced back to find that they have updated their web interface but not their API which we're using. So I mailed their team for informing them about it & they instantly fixed it.

What is coming up next?

Now I need to finish up my work on developing this update cache mechanism. Also we have planned to mend some documentation of our package to make it accessible for more users.

Did I get stuck anywhere?

Not as such, all the seemingly difficult problems that I encountered, eventually got solved.

What was something new I learned?

🗃️ Accessing package data: The data we store in our package doesn't get included while building it unless we specify it in package_data which is often configurable from setup.py file. This needs us to specify path of files we want to include in built package by using file globbing patterns. So I also learned that file globbing doesn't work recursively if we use /*, so we need to make sure to specify the paths of all depths we want to be included.

🔎 Tracing how a piece of code changed throughout the history of a project: Thanks to git that it provides numerous options with log command, using which you can find all the commits that affected a string (let it be the function name, or any other uniquely-identifying part of the code you want to search). It came into my use when I needed to know why the function listed in a error message of our package, doesn't exist in the codebase.

📝 How to edit and install a package simultaneously: Earlier I used to edit code files of the installed package in /../site-packages/ (which is even referred as bad practice) but I couldn't track the changes I made because it is not in a git repo. So I looked for a better approach on internet and discussed with my mentor to find that I can use setup.py develop instead of installing the package. This creates a .egg-link in the site-packages back to the project source code directory so that you can see the changes directly without having to reinstall every time you made an edit in package.

Thank you for reading. Stay tuned to learn more things with me!

Week-7: Coding the Caching

jaladhsinghal@gmail.com (jaladh-singhal) — Mon, 15 Jul 2019 20:10:34 +0000

Hello folks,

This week I finished writing a module for caching filter data that our package wsynphot needs. So I wrote quite a lot of code which obviously required tests, thereby deepened my understanding of unit-testing. 🤓

What did I do this week?

I mainly worked on creating cache_filters module that handles cached filter data. I did various tasks under this:
- Decorated download data function with progress bar using tqdm
- Created functions to load filter data from cached VOTables to dataframes
- Documented all function in a notebook
- Created tests for entire module - used various new things like pytest fixtures, pandas testing framework and figured out how to make sure tests/data is available in built package
- Improvised error reporting mechanism of the cache loader functions & created test for fail cases
I also setup RTD redirects to docs on Github pages (same last week's work) on a repository of our sister project TARDIS. And amazingly this time, I figured out how to create exact redirects to Github pages from RTD index page - now I know 2 methods of creating redirects: implicitly & explicitly.

What is coming up next?

Now I need to integrate these cache_filters module in our base module by dropping the functions using HDF storage (old way of filter data access). Then I'll create an updating mechanism for updating the filter data cache (as our data source SVO FPS keeps on updating).

Did I get stuck anywhere?

This week made me realize creating tests for I/O functions, is a really tricky task. I was confused on how to write unit test for download function that iteratively fetched over 4500 files. I extensively researched for such situations and found only solution was code refactoring for the sake of making function testable - which also have diverse views on internet. My mentor resolved this dilemma by telling me that I can skip lines from tests if they are either trivial or tested somewhere else, so I created test only for the functions being called by that massive download function.

What was something new I learned?

📇 Writing tests in Pytest that access data: I learned several things while creating unit tests for my module, like:

By using pytest fixture objects, we can setup & teardown data resources easily in our tests.
To make data stored in tests/data/ available to tests, we also need to make sure that data files are listed in package data of setup file - for this astropy setup helpers even provide a function get_package_data() to define the data we need to access from our built package.
For failing cases, we can create tests that check whether an expected exception is raised by using pytest.raises.

✏️ Logger instead of print statements: In a package we often need to show user some message, may be an error or information about the action they are performing by using our package. For these cases, print is an obvious choice but we have a much better option: Logger (object from python's logging module). Logger is preferred over print because logs are highly configurable - you can save them to files, can locate a logging call by line number they display, can hide/show them, etc.

Thank you for reading. Stay tuned to know about my upcoming experiences!

Week-6: Gotta Cache 'em All!

jaladhsinghal@gmail.com (jaladh-singhal) — Mon, 08 Jul 2019 22:44:15 +0000

Hello folks,

So I am still implementing several methods of accessing filter data from SVO FPS - redesigning our old way of HDF storage. After implementing a module to access filter data using API, I've entered 1st time into the realm of caching! Now I'll create a module to cache the data on disk (for fast access obviously), so let me tell you how I'm going to cache 'em all 🙃 and what I completed until now...

What did I do this week?

To contribute our query functions to access filter data from SVO FPS, following my previous work, I created a clean PR at Astroquery as per their API specification. They added it to their milestone so I think they will include this in their package.
There was a bunch of several tasks hanging around, that I finished this week:
- As we have switched from readthedocs (RTD) to Github pages, my mentor told me to create redirect at our homepage of RTD docs to github pages. He had already setup it at our sister project TARDIS, so I understood his brilliant idea and how to use readthedocs, and successfully setup RTD redirects for our packages.
- I removed python 2.7 support from our packages because it was becoming time-consuming to maintain both versions when we have a lot of important things to implement.
- I also did some tweaks in the module I created last week (that fetch filter data using SVO API in real-time) to make it more package specific like removing the class structure (unlike what I did for Astroquery) & got it merged - my 1st entirely self-written module in our codebase, oh yeah! 😊
Besides having a module to access data in real-time, we decided to have another module that caches filter data (locally on disk) for fast access next time. So I opened a PR for it, & created function that can download entire filter data and cache it on disk systematically (by storing data in a hierarchy of category-named folders).

What is coming up next?

Now I'll work further on this PR to finish up with cache_filters module by adding more functions to read & update the cache, tests & documentation, etc. Also I'll try to make sure that both of our packages work robustly, before we can move on to interface developing part.

Did I get stuck anywhere?

While creating RTD redirects, it was using cached environment instead of updated one, in each build. I even didn't find anything relevant to this problem on internet! After a lot of trials, I came up with a workaround of changing the default branch that finally fixed it. Also I got quite confused on how will I make use of functions I created in previous module in this new cache_filters module. But after having a thorough discussion with my mentor, he resolved my dilemma by clearing out what this module needs to do.

What was something new I did and what I learned?

📃 Documentation setup on readthedocs.io: Though I worked on redirecting our docs from RTD to Github pages, but this task required me to build docs successfully on RTD using a dedicated branch for redirect (hollowed up docs). Hence I learned the entire process of docs setup on RTD from how to change default branch from Admin settings (in RTD project account) to creating a readthedocs.yml file for configuring the build.

🚧 Right way to open a work-in-progress PR: Earlier I used to believe that unless there isn't something presentable in your code, you shouldn't create a PR. But my mentor told me that I can create a PR even with a rough prototype, by using the checklists for indicating work progress in the PR. It's a really great feature, each time you finish off a task, check it off the list and bonus - get the feel! 😌

Thank you for reading. Stay tuned to know about my upcoming experiences!

Week-5: Evaluation Period-1

jaladhsinghal@gmail.com (jaladh-singhal) — Sun, 30 Jun 2019 14:41:07 +0000

Hello folks,

So the evaluation week passed marking the completion of 1 month of GSoC! 😮 It has been an incredible journey so far with lot of learnings not only about coding but also about life. Thanks to the evaluation period that we're able to get wonderful feedback from our awesome mentors! 😊

What did I do this week?

After creating functions for fetching filter data using API, I created a PR and worked on it, doing the changes as left on review by my mentor. I did a lot new things in this PR to make my written code better, like storing downloaded data temporarily (as a buffer), adding docstrings for functions, using Astropy units, etc. I further improvised my code by reducing the redundant code by creating another function for it. This was all about code, my mentor suggested there are other 2 things which should also be present in a good PR besides code:
- Documentation: I created a notebook illustrating the use of all functions.
- Tests: I used pytest, to create tests for the functions. I also learned about Astropy testing suite which is used by our package for running pytests. And I also came to know about remote-data plugin used for running tests that access data from internet.
My mentor suggested it will be great if we could contribute this (SVO FPS) to another popular Open Source package: Astroquery, a popular python package to access data from astronomical data resources. So I created an issue for it at astroquery to share our work at wsynphot, they welcomely accepted it and told me to do changes in my code as per their standard API. So I created a class for all those functions and I still need to do some work, before I create a PR in their repository.

What is coming up next?

Now I will create a PR in astroquery and see how things turn out. Also we will finish up these filter data access methods for our package wsynphot. Possibly we will create some offline support (to cache the fetched data on disk) - since data is huge, so it takes time in fetching data based on your internet connection speed (a limitation of the real-time fetching approach).

Did I get stuck anywhere?

While wrapping my created functions in a class, it took me time to figure out how to import it by creating a public object. Also I got stuck in running tests, 1st because it was using Astropy test suite and then these tests were accessing data over internet, so it failed to run them. But by taking a break and trying again with a calm head, I figured it out that remote-data option needs to be specified for running such tests.

What about Evaluations?
As this was Evaluation Week, both students and mentors evaluate each other by filling their responses in an evaluation form provided by GSoC. So yesterday, I recieved a mail from GSoC that I passed the evaluation and can continue the GSoC program. And in it my mentor's feedback (evaluation) was also mentioned, which was so wonderful that I read it 4 times in a row! 😍 He mentioned lots & lots of appreciation of what I do well and also the suggestions of where could I improve. I am really grateful that he let me know about my capabilities as well as how could I improve myself. This week really made me feel the importance of this program and a mentor, that there is someone to guide you and help you becoming better not only as a developer but also as a person.

What was something new and exciting I learned?

📥 Saving files temporarily: While handling files there are cases when we need to pass file (or path) as an argument to a library function. For parsing the downloaded filter data, I needed to store it in file before parsing. But I wanted it to be deleted after it is used. That's why I came to know about python tempfile package which lets you store your data in a temporary file which gets deleted when you close it. My mentor suggested a better approach to use file-like objects instead. So I used BytesIO object to store data as a memory buffer, temporarily.

📂 Handling Python package & modules: My mentor told me to put my created python file (containing functions) in a subpackage io within our repo. When I needed to import it while using, so I learned how to achieve it by creating a __init.py__ file for it. Also when I put my functions in class, I learned how to import it by instantiating an object and then importing it publically in the init.py file. Also when I wrote tests for my module, I felt kind of satisfied that now I almost know how to create a package on my own!

⛱️ Recognize when you need a break: We know that a break is necessary if we're stuck while solving a problem. But we overlook this fact, while we keep pushing ourselves to solve the problem which takes us close not to the solution but to frustration. Same my mentor told me in feedback that he likes my enthusiasm but to keep that alive, I need to take breaks when my mind/body needs it. And it really works, like it did when I was stuck in figuring out the problem in running pytests. All you need is to step back from problem & relax, solution will come itself with time (as a new possibility) because our subconscious mind keeps on working on it, even when our conscious mind is relaxing!

Thank you for reading. Stay tuned to learn the cool stuff I am learning each week!

Week-4: Turning point in the Pipeline - jumped onto API

jaladhsinghal@gmail.com (jaladh-singhal) — Tue, 25 Jun 2019 10:33:52 +0000

Hello folks,

So the 4 weeks flew by pretty fast and 1st phase of GSoC Coding Period has came to an end! This week brought a lot of twist & turns in our plan but that made me learn about lot of things. So this blog post will be quite longer than usual, but more interesting!

What did I do this week?

After setting up auto-ingest pipeline last week, my mentor told me to try using Azure Artifacts for deploying our files (executed ingestion notebook & generated filter dataset). I read about it and successfully enabled the pipeline to deploy files as versioned universal packages (artifacts). But we realized that they can't be made publically available and can only be downloaded by using Azure CLI. So we decided to drop it & continue our previous idea of deploying the files to a web server.
- I created a PR for this pipeline so that my mentor can review my last week's work. We also made some discussions & decided to use Dokuwiki to create a simplistic web interface for displaying deployed files in a presentable way on web server.
- I worked further on this PR to evolve the auto-ingest pipeline from a prototype to finished state. I created couple of scripts: (i) to execute notebook until 1st error & return error status; (ii) to deploy dataset conditionally (only when notebook executed successfully and when dataset has changed from previous deployed version).
Now since I needed to compare datasets (HDF files) which is not possible without a HDF comparison utility (external deb package), so I discussed with my mentor whether I can write a script to check if there were any changes at SVO (by scraping their changelog). My mentor told me why not to use a programmatic interface to access data from SVO (as he had heard about this). And it turned out to be a turning point for our ongoing plan. This would mean we won't be needing any pipeline (on which I was working since a week) as we can directly fetch data from SVO. It was quite surprising for me to grasp, but since it is a better way so I understood & accepted this change.
- I searched & found that besides a web interface, SVO Filter Profile Service (FPS) also provides a VO interface (sort of API) so that applications can send HTTP queries to the service and retrieve data as VOTable. My mentor told me to use a VOTable parser provided by Astropy to obtain tables as dataframes and see if data makes sense.
- I tried it and fortunately SVO VOTables get parsed accurately! Hence I created functions for fetching filter data using requests & VOTable parser.

What is coming up next?

As my mentor told me during discussions, we will possibly contribute these functions to astroquery (an Open Source python package to access data from astronomical data resources). Also we will decide & work on redesigning the way our package, wsynphot fetches filter data from SVO.

Did I get stuck anywhere?

Yes while reading about programmatic interface of SVO FPS, the HTTP queries I found in their document, were not working. But my mentor suggested me to take a time off and it proved fruitful. Next day when I read document with clear mind, I realized my mistake (I was earlier referring the wrong section) and found the right queries.

What was something new I did and what I learned?

✅ Code Review: I reviewed the PR of another GSoC student in our sister Org - TARDIS, on docs deployment pipeline. Since I had already setup that successfuly in our Org so my mentor told me to review his code. I suggested him several changes in review, and explained a mistake he was doing in deploying docs to gh-pages - this helped him to understand the concept of docs deployment better. I realized that code review is inturn helpful for reviewer too as it allowed me to assess how much I already know.

💡 The work I did in this week made me learn about a variety of tools & techniques:

Azure Artifacts allows us to create a package feed, where all versions of our deployed packages (collection of files) are listed. We can even connect artifacts with package hosts like PyPI to make users download it.
Dokuwiki is a Open Source wiki software that allows us to create versatile wikis that don't need database. Using this we can even create a simple presentable web interface (like one we wanted to display deployed files). There's also a python module for Dokuwiki to create & manage wikis, so within pipeline we can source such a python script to update our wiki for files deployed.
Passing variables between shell & scripts - To access variables of calling shell in python script we can use os.environ dictionary. To use the variables defined on Azure VM environment in bash script we need to do some changes in naming style of variable. Also in pipleine yml we can declare the variable within the tasks instead globally. To return a variable from script, we can simply print it and access it from shell by storing the output of command sourcing the script.
VOTable (Virtual Observatory Table) is a way to represent & store data in XML syntax. A VOTable consist of Resources elements which contain Table elements. They can be parsed into Astropy tables simply by using astropy.io.votable.parse which can be further converted to a Pandas dataframe.

🗃️ Simplicity an API brings: Earlier we were using a ingestion notebook that scrapes the data from SVO web interface, download the data files and ingest them into a HDF file. As data at SVO updates with time, so we began to create a pipeline that can execute this notebook and generate HDF and then deploy them at server to access them. But by using VO interface of SVO (API), we can fetch data directly from SVO instead of that from HDF. So there's no need to update HDF and hence no need of pipeline. All we need is to call a function that can fetch VOTable from SVO using HTTP queries and return it after parsing it as a dataframe - simple!

Thank you for reading. Stay tuned to know more exciting things with me!

Week-3: Travelling through Pipelines

jaladhsinghal@gmail.com (jaladh-singhal) — Sun, 23 Jun 2019 14:59:40 +0000

Hello folks,

So pretty much happened this week - I got some stability in my schedule and got success in setting up 3 different kind of pipelines for our package.😊 Last 2 weeks were quite hectic for me that I am writing this blog post (for Week-3) much later. Let me share why ...

What did I do this week?

After setting up CI & docs CD Azure pipelines in last week, I began setting up them for both of our main repos (starkit & wsynphot).
- Since in this case, it was the Azure project & repo of my mentor, so I didn't have many of the access rights required in the process of setting up. We discussed about the process & access rights I need, my mentors gave me the permissions.
- Still there were couple of more authorization problems at Azure project, it took time to figure them out. And I finally I successfully setup both of the pipelines (CI and docs CD) for both of our repos. Now we have officialy switched from Travis & doctr to Azure completely.
The next task was to create another schedule triggered pipeline for executing the notebook (that ingests the filter data fetched from SVO in a HDF file) and deploying it on a server from where we can see if there's any error happened in generating HDF (filter data). This way we want to automate the generation of filter data time to time by using a pipeline, since the filter data fetched from SVO keeps on changing!
- I figured out the basic workflow of how can I implement such a pipeline. Also set up the ssh connection to the server my mentor provided me for deploying the files.
- Since there was lot to do in this - my mentor suggested me to create 1st a very simple pipeline for a test notebook (instead of actual) which is capable of deploying stuff on the server.
There were several other tasks I also did in this week:
- I helped the guy (Youssef) who helped me in setting up CD pipeline (last week), in documenting the process of setting up Azure pipeline.
- I improved my PR of integrating filter list in docs, as per review given by mentor. And I also discussed extensively with him to understand what is the problem in my approach of generating RST files (since he suggested to ind a elegant way to do it).
- I also reported the errors in the SVO FPS web interface by mailing the SVO team, they were so quick to fix it.

What is coming up next?

Now I've setup this auto-ingestion pipeline for test notebook, next plan will be to evolve it from a prototype to all the stuff we want it to do (like stop on 1st error in notebook, conditionally deploy HDF, etc.). Ultimately making it work for our actual ingestion notebook.

Did I get stuck anywhere?

As I shared above, there were various authorization problems while setting up pipeline on a repo in which I don't have admin rights - so I did felt terribly stuck as I could not proceed without my mentors aid. But after some patience & discussions (when they got time) - we fixed it.

What was something new and exciting I learned?

🤖 Automation using Pipelines: Earlier I was doing just CI & docs CD using Azure pipelines but when I started working on auto-ingest pipeline, I found out we can automate any process using a pipeline - their real power! It's just what we're doing locally, we tell the Virtual Machine (VM) to do same by means of scripts, and set desired triggers for initiating it. This saves not only our time but also our resources, as entire work is being carried out on a VM without any need to put efforts in it.

🖥️ The server whose access my mentor provided me is actually a Linux machine which can serve as web host using Apache. I learned how to communicate with a remote machine over SSH by setting up SSH key pair. I also got to know about basics of Apache for web hosting - we just put files in a public_html folder on machine and they get available on web automatically by root URL.

🗒️ Importance of creating a Prototype 1st: After planning, when we start to develop a complex project, we should not forget to break it down & identify the core functionality in it - because other tasks won't yield any good result if that main task doesn't work. Same thing my mentors told me to clear up the confusion I was having in overwhelming choices I had to develop. Also a prototype means that we use a smaller test input instead of big actual input, so that output obtained is smaller thereby debugging problems become lot easier!

Thank you for reading. Stay tuned to know about my upcoming experiences!

Week-2: Open Source Rocks!

jaladhsinghal@gmail.com (jaladh-singhal) — Mon, 10 Jun 2019 13:53:43 +0000

Hello folks,

I am pleased to share that today my no. of contributions crossed 100! I know it's not a very big achievement, but atleast it gives happiness to a person who stepped in Open Source world, just 4 months ago. Thanks to Google Summer of Code 2019 ❤️

Reached 3-digit contributions!

What did I do this week?

I resumed my work on an old PR that integrates filter curves & information to our Sphinx docs by auto-generating them, each time docs are build. It is meant for providing a simple static interface to users in the docs itself. For this I'm using Jinja templating to generate Sphinx docs from the dataset.
- I devised & implemented a better approach to integrate lots of data, by generating the required ~80 RST files at time of build, instead of storing them by defaut in docs/ directory.
- By using Jinja, I converted Pandas dataframes (part of our dataset) into RST tables and made sure that filter tables appear in docs.
Then my mentors told me to put it on a hold, and start working on setting up Azure pipelines for our repo. They are required for the Continuous Integration & Deployment (CI/CD), which are more prior than development for any project. Earlier we were using Travis for CI/CD but due to some of its shortcomings we decided to move to Azure DevOps. I was absolutely new to this, so my mentor told me to take help of another GSoC student - Youssef, who is setting up Azure pipelines for Tardis (another PSF SubOrg that my mentor is mentoring).
- I created my Azure DevOps account, communicated with Youssef extensively for the Azure docs I need to read to understand this. He helped me a lot by providing his code & steps that are not listed in docs.
- So after figuring out the process I needed to follow, and customizing his code as per our repository, I successfully setup both pipelines for our repo's fork. One pipeline is for testing (CI) and another is for building & deploying docs to GitHub pages (CD).

What is coming up next?

My mentor has already added me to our Org's Azure account as a member, so I will start setting up these pipelines for our main repositories - starkit & wsynphot. And if time allows, I'll resume my work on that PR to integrate filter curves in docs.

Did I get stuck anywhere?

While setting up azure pipeline for docs deployment, I felt quite stuck because there was so much new for me to grasp. The process required generation of SSH keys and setting up them with Azure by configuring variables, secure files, etc. of the pipeline. But by patient reading of Azure docs & clarifying my doubts with Youssef, I finished it up successfully.

What was something new and exciting I learned?

📝 PEP8 (Python Styling Guide): While reviewing my PR, my mentor pointed out that I should adopt PEP8 style in my added code to maintain code consistency. I never knew that there exists some standard styling guide for Python. I read it, it explains really well about almost every formatting & styling choices we need to make in our code and which of them is preferable!

🔐 SSH RSA keys are used as authentication credential for single sign-on without needing username & password. I came to know about this while setting up docs deployment pipeline where we need to use SSH to push commits made by VM to our GitHub repo. These keys always exist in pair as a private key & a public key - I read about this past semester in my coursebook, but I understood it better when I implemented it.

❤️ Open Source = Fast Completion: This setup of Azure pipelines that I completed in only 2 days, could not have been possible if I was not working in an Open Source Community. As Youssef had already given enough time to figure things out and writing code for the pipeline, I need not to start from scratch. I used his code & quickly customized it as per our repo. I need to understand the process, I reached him & quickly got the relevant sources. A big thanks to him & the community. More power to Open Source!

Thank you for reading. Stay tuned to know about my upcoming experiences!

Week-1: Mending the roads for smooth travel in subsequent weeks

jaladhsinghal@gmail.com (jaladh-singhal) — Mon, 03 Jun 2019 13:08:52 +0000

Hello folks,

So the 1st week ended pretty quick and here I am to share my 7-day journey. The most shockingly amazing thing happened this week was that my mentors made me the member of our Org - StarKit. Now I'm not only a contributor but officially a collaborator! I am thankful to my mentors for believing in me. 😊

What did I do this week?

- I began with fixing issue in some of the filters we're using (problem in loading them) by writing code to clean the filter data. After several discussions with mentors, I finally ended up creating a new updated HDF file of filters - that works perfectly fine!

- Our Travis test builds unexpectedly began to fail - I identified it was due to a recent update in pluggy (used by pytest). So I created an issue on pluggy repository, their developers were amazingly fast as they fixed the problem within 6 hours.

- Updated codebase for the setup of doctr (to deploy our docs on GitHub pages directly from travis build).

What is coming up next?

This week, work was mostly inclined on fixing & setting things up. In following week, I'll most probably work on creating a static interface on our Sphinx docs that will let users to access required filter curves & information. Or if my mentors suggest to undertake something more prior, I'll work on that (since our package WSynPhot is in its initial phase - so I can't strictly go according to my proposed timeline).

Did I get stuck anywhere?

While setting up doctr - I got confused and mistakenly took the OAUTH key sent by my mentor as public key to put in Travis. Due to my PR, it got publicized! My mentor forwarded me the mail he got from GitHub about privacy leak and I realized what a blunder I committed! *facepalm*

But he was cool about it, said that it was a learning experience for me. So I created another PR for doctr setup, once he himself added newly generated public keys. We had some discussions about the uncertainities he had about my PR but I convinced him and it worked as expected. And viola! we got our docs deployed on github pages. Later my mentor realized OAUTH key was actually not required in the process and cleared the confusion for all of us. 😅

What was something new and exciting I learned?

- Taking initiative to communicate: I reported issue 1st time to another org whose tool/service we're using (pluggy, and SVO from which we obtained unclean filter data) - my mentors even admired me for this!

- HDF5 files (a way to store & organize large amount of data hierarchically in groups): Understood about how they organize data while investigating erroneous data in them and reading/writing them using pandas and h5.py

- And the biggest thing - Keeping Patience: Sounds pretty obvious to already have. But no! when you work in a team, you become very eager to hear from other members, so that you can proceed and quickly finish your work. As my mentors are astronomers by profession - they remain quite busy in their work & conferences. So I've learned that it is OK to wait (get involved in something other than this project) before you hear back from them, even if it takes a day. But that is for sure that they do follow up when they become free. So keep patience & stay calm!

Thank you for reading. I'll get back to you next week with my new experience & exciting learnings. Stay tuned!

Summer Begins!

jaladhsinghal@gmail.com (jaladh-singhal) — Mon, 27 May 2019 21:47:28 +0000

So after 3 weeks when GSoC'19 results were announced, Coding Period has finally arrived - the Summer of Google Summer of Code has began!
It had been an incredible journey so far, with lots of learnings and lot of struggles. What was a dream for me in Jan'19, became an unbelievable reality at May 6 (the D-day when results were declared). It was a thrilling experience for me, a Sophomore, to get selected in GSoC'19 at PSF, who just stepped into Python last year. I've shared this exciting journey from my dream till reality in a dedicated article: https://medium.com/@jaladh_singhal/gsoc19-from-dream-to-reality-3bf07d6f17bb.

Besides being Pythonista, I'm an Astronomy enthusiast, that's why I chose the PSF SubOrg - StarKit: A Python package for Stellar spectroscopy. This Summer, I'll be developing interactive analytical interfaces to visualize the Filter Curves of our package wsynphot. These interfaces will not only be able to auto-generate the Filter Curves but will also calculate the Photometry of Stars. The purpose is to make it easier for Astronomers to access the info they want within couple of clicks. Sounds cool! Right?

You can find details of my project idea at GSoC 2019 Projects page and package I'll be working upon i.e. wsynphot on GitHub page of StarKit. Within the Community Bonding Period (last 3 weeks), I was busy in my exams but I still managed to attend a Video Telecon with my mentors to discuss about a multiple things, from working in Community to the interesting work my mentors are doing in the field of Astronomy. They are really cool people - one of them is researching on Supernovae and another on nuclei of Galaxies & Black Holes!

As the Coding Period has began from today, I'm pumped up to start working on my project; to learn lot of things by implementing them - the real essence of Open Source. And Astronomy is already fueling my excitement! Stay tuned with me for this exciting journey, more posts to come. :D