Articles on programmer290399's Blog

Weekly Blog Post #12 [Aug. 23, 2021]

programmer290399@gmail.com (programmer290399) — Mon, 23 Aug 2021 05:55:33 +0000

What did you do this week?

This week I streamlined the SimpleModel class which derives from the Model base class in which I recently added archive support code.
I also made multiple minor changes [1,2] as requested by my mentor.
Other than that I updated all the models to reduce code duplication and support archive storage.
While I was updating all the models I noticed that I was having problems in saving and loading trained models for the spaCy model, I discovered that there was an issue in the archive creation code.
I quickly fixed the problem in a subsequent PR.
I also added a tutorial on how to save/load models as archives.
Last but not the least I added test cases to cover dffml.df.archive.

What is coming up next?

This week marks the end of my GSoC'21 journey, I would continue contributing to DFFML in the future.

Did you get stuck anywhere?

Not really, I was stuck for some time figuring out why the spaCy model was not working, but I was able to spot out the issue and fix it pretty quickly.

Weekly Check-In #11 [Aug. 16, 2021]

programmer290399@gmail.com (programmer290399) — Mon, 16 Aug 2021 05:33:31 +0000

What did you do this week?

At the beginning of this week, I received some feedback from my mentor regarding the flow and some other stuff[1,2].
I worked on the changes requested by my mentor and fixed various bugs [1,2,3,4,5].
Other than that I also fixed some test cases that were failing after I made the changes last week [1,2].

What is coming up next?

Testing and implementing/fixing archive storage support in all 11 models listed in this PR.
Other than I have noticed that some models have repetitive code blocks, I would try to eliminate any such duplication.

Did you get stuck anywhere?

Nope, this week I spent most of my time reading the model code and analyzing where and what changes would be needed.

Weekly Blog Post #10 [Aug. 9, 2021]

programmer290399@gmail.com (programmer290399) — Wed, 11 Aug 2021 07:04:13 +0000

What did you do this week?

This week I worked upon suggestions [1,2,3] from my mentor related to archive support PR.
I also came up with this flow to make sure that I and my mentor(s) are on the same page.
There was some confusion around how configs could be restored which I resolved with my mentor in this week's meet.
Other than that I removed the dataflow creation code for saving and loading from archives from the Model base class.
And added a much less verbose helper function to create those dataflows in dffml.df.archive.
Also added config loading code which I am planning to update in the coming week, alongside adding standard definitions for loading/saving flows for models.

What is coming up next?

I would be working on the flow linked above and would be streamlining various other parts of code to work properly with archive code.
I would be perusing through the code for various models to understand and plan where & what all changes would be needed to properly support archive storage.
Other than that I'd be fixing any bugs that might have been introduced in my recent commits.

Did you get stuck anywhere?

I got stuck when I was trying to condense the code for archive dataflow creation, but with some time, thought and basic math, I was able to significantly reduce the code's verbosity while improving its readability.

Weekly Check-In #9 [Aug. 2, 2021]

programmer290399@gmail.com (programmer290399) — Mon, 02 Aug 2021 13:09:05 +0000

What did you do this week?

This week I got two of my PRs merged after a long time:
- ci : lint : commits : Adding ci job to validate commit message format #1076
- util:log: Added log_time decorator #1101
However merging PR #1076 introduced an issue in master in which the lint commit command kept running forever and was killed by Github Actions for exceeding run time limits.
I fixed this issue quickly in another PR which simply skipped the linting test on master branch.
Other than that I continued working on my possibly final PR for my GSoC project based on the inputs [1,2,3] received from my mentor.

What is coming up next?

As per the feedback received from my mentor on archive support part I have to make some changes which I'd be picking up in coming week.
Also, currently the archive dataflow creation code is very verbose and I will try to improve upon it.

Did you get stuck anywhere?

Not really, I have been thinking a lot about how the verbosity of dataflow creation code can be reduced, how various blocks of code which are pretty similar can be removed and how the overall mechanism can be made to look more pythonic and clean.

Weekly Blog Post #8 [July 26, 2021]

programmer290399@gmail.com (programmer290399) — Tue, 27 Jul 2021 10:53:00 +0000

What did you do this week?

This week I worked on two PRs:
1. high_level: Move code into a directory and splitting out into files #1172
2. WIP: model : Add Support for Archive Storage of Models #1174
The first one was related to an issue I faced earlier and the second PR was dependent on the first one.
So before making the first PR I opened up an issue to get feedback from my mentor before I proceeded with the implementation.
After the first PR was merged I continued working on the second one.
There were some issues I faced in both of these PRs which I'd be discussing later in this post.
Other than that I rebased my other two PRs which are ready to merge with all relevant CI tests passing.
1. util:log: Added log_time decorator #1101
2. WIP : ci : lint : commits : Adding ci job to validate commit message format #1076

What is coming up next?

I would be working on to complete all the tasks in my possibly the final PR to fix issue #662
This would take a long time as I have to make changes in all the models and update their respective tests and docs.
Other than that I would be fixing a logging issue, I still have to think about how it can be fixed.

Did you get stuck anywhere?

Yes, starting with the high_level splitting PR, I was stuck on a failing docstring test, I don't know how it was working before I made this change, after a ton of debugging I just gave up and solved it with a pretty trivial solution.
Other than that I was not really stuck but spent quite a lot of my time in thinking how the tar support should be implemented in the Model class, I have pushed a rough implementation to get feedback from my mentor(s).

Weekly Check-In #7 [July 19, 2021]

programmer290399@gmail.com (programmer290399) — Sun, 18 Jul 2021 15:26:39 +0000

What did you do this week?

Finally got my directory property to location renaming PR merged.
Other than that I closed two similar issues [1,2] with a PR, this change now returns an output for archive and compression operations merged earlier.
I also made a small fix for a docstring test that was failing in this PR.
Also fixed some typos and made some housekeeping related changes here and there [1,2,3].
Other than that I worked on rebasing my open PRs [1,2] with master because Accuracy scorers have been merged into master branch.

What is coming up next?

I hope I'd now work on implementing archive support in Model base class finally.
But before that I would be fixing a related issue which is sort of blocking the implementation of DataFlows required to perform archiving ops.
This in itself is going to be a big change which would take a considerable amount of time.

Did you get stuck anywhere?

Nope, this week I mostly fixed simple stuff so there was nothing that bothered me.

Weekly Blog Post #6 [July 12, 2021]

programmer290399@gmail.com (programmer290399) — Mon, 12 Jul 2021 02:17:17 +0000

What did you do this week?

I worked on implementation of archive support in the Model base class using Operations I implemented earlier and created a PR.
Then I requested feedback from my mentors on the changes I made in the Model base class to make sure that I was working in the right direction.
As per the feedback from my mentor I rolled back the changes related to archive support.[1,2]
Fixed some typos here and there as well.
After the receiving the feedback, this week I worked hard on renaming the directory property to location and made changes to 144 files in the codebase to make sure that all the tests pass locally as well as in CI.
However, still one test is failing and perhaps it is not an issue.
I have requested for another review of my PR related to renaming directory property to location after making all the requested changes to code, docs and tests for all models.
Other than that my other two open PRs:
- util:log: Added log_time decorator #1101
- ci : lint : commits : Adding ci job to validate commit message format #1076
  
  are also ready to merge with all the CI tests passing 🎉.

What is coming up next?

After receiving feedback on my current changes I will work on any other changes that my mentor(s) might request.
Other than that I will also discuss the archive support implementation details with my mentor in the upcoming weekly sync and work on a new PR for the same.
I will also work on the Commit Linting Issue's PR to cover more enhancement points depending on the bandwidth I have this week.

Did you get stuck anywhere?

Yes and no as well, I took a considerable amount of time in reading the code base in couple of past weeks, to get a good idea of where the renaming is to be done, it is not like find and replace, there are a lot of places where directory property is not even related to a model and thus it was not as simple as it might seem.
Making the changes also broke a lot of stuff, almost all model tests and other various seemingly unrelated tests as well. So I was stuck at certain errors in the CI for sometime but eventually I was able to sort all of them out.
However, I cannot really say I was stuck all the time in the above listed things, as it was more about understanding and spotting out sources of problems which were causing various tests to fail.

Weekly Check-In #5 [July 5, 2021]

programmer290399@gmail.com (programmer290399) — Thu, 08 Jul 2021 02:28:32 +0000

What did you do this week?

I continued working on the update for Model base class to support archive.
I also made all the requested changes for Archive and Compression related Operations and got my PR merged.
Other than that I completed the implementation of a couple of enhancement points for Commit Linting Issue and updated the tests as well. This also increased coverage of master commits by ≈ 2%.
I also finally Fixed the MacOS error in the CI by refactoring the test case for the Commit Linting Issue.

What is coming up next?

My main focus this week would be on Updating the Model base class to support archive storage, as I have also mentioned in my previous blog post that it is a bit time consuming and thus I might not be able to push working changes very soon.
Other than I'd be looking into the code for other models as well to foresee where changes would be required to adapt to the changes I've made in the Model Base class.

Did you get stuck anywhere?

Yes, for sometime I was stuck while implementing the body mutation for Commit Linting Issue , as conditional mutations would fail in catching cases if no_muation was added to the list, but not adding no_mutation would lead to failure in other common cases.
I solved the issue by implementing a composition function generation method and making the conditional body mutation act as no_mutation if the condition was not met.

Weekly Blog Post #4 [June 28, 2021]

programmer290399@gmail.com (programmer290399) — Tue, 29 Jun 2021 00:30:06 +0000

What did you do this week?

I worked on two issues this week:
1. Adding Operations for Archives and Compression
2. Commit Linting Issue
I removed options parameter from and updated definitions for existing zip operations as per the feedback from my mentor. [1,2]
Other than that I cleaned up and also optimized some code here and there. [1,2,3,4,5,6 & 7]
Starting with Point 1, I implemented operations for creating and inflation of tar archives, also implemented their respective test cases.
The test for extract_tar_archive was especially a bit too tricky, writing it correctly took a considerable amount of time (more on that later).
Also added compression operations to support .gz, .bz2 & .xz formats with their respective test cases.
Moving on to Point 2, I implemented one of the five parts of enhancement issue related to it, i.e. added support for commits related to tests by implementing a body mutation.
I also optimized the code to avoid multiple calls to _get_all_exts(), by making a class attribute for it's output.
Last but not the least, I updated test cases to make sure that the linting support for tests related body mutation is working correctly.

What is coming up next?

Updating the Model base class to support archive storage using these operations would be my top priority for the coming week.
Obviously this would break a considerable number of models and would also call for implementation of new test cases and a number of small tweaks.
Other than that I would continue to work on the commit linting issue as per the enhancement checklist.

Did you get stuck anywhere?

Yes, while implementing the test case for extract_tar_archive, I ran into a lot of issues, mostly consisting of problems of faulty patching and problems regarding getting the right mock calls in place. However, I was able to solve the problem successfully after investing almost a complete day in it.
Deciding on how to implement the compression algorithms, also took some time, I wasn't really stuck on it, but it took some tome and discussion with my mentors to come to a conclusion.
Initially I thought I would give a compression option with the archive operations itself but as per the feedback from my mentor I wasn't supposed to implement options for now.
Also, binding the compression with archiving operations would not have been the best thing to do, implementing compression as an operation itself was the best way which I and both of my mentors agreed as it is comparatively more modular and flexible approach at solving this issue.
I still have not been able to resolve the MacOS related CI error in the commit linting issue, I would be discussing that with my mentor in the upcoming weekly sync.

Weekly Check-In #3 [June 21, 2021]

programmer290399@gmail.com (programmer290399) — Tue, 22 Jun 2021 06:38:59 +0000

What did you do this week?

I worked on two things this week:
- Commit Linting Issue
- Archive manager Implementation
The commit linting issue was open from a long time and I had to debug some CI errors and I finally was able to get it up and running on the CI.
I also worked on the Archive Manager implementation, which is the first part of my project after taking input from my mentor.

What is coming up next?

A couple of CI errors are still left in the commit linting issue, would be fixing them.
Would be brushing up and finalising the archive manager stuff.
Would make necessary changes to model base class.

Did you get stuck anywhere?

Yes, I was stuck with one thing for a long time,that was getting the current branch name in the CI, and it was not that easy for me because I wasn't aware of the fact that some env variables hold it and the actual PR branch name is different, anyways it didn't solve the issue anyways and I had to change the code to not require the current branch name. More on that whole thing here.
Also some CI errors bothered me this week, I haven't been able to fix them yet, I'd be discussing them with my mentor soon.

Weekly Blog Post #2 [June 14, 2021]

programmer290399@gmail.com (programmer290399) — Mon, 14 Jun 2021 02:38:10 +0000

What did you do this week?

This week I spent most of my time reading the docs, stack-overflow answers and blog-posts of various archive handling methods and all which is built-in the python standard library.
The goal of reading all this was to come up with a design of archive manipulation module which can be format agnostic.
First I thought I should use shutil.unpack_archive as it handles most of the common formats, and it would've been easier and simpler to implement in tandem with shutil.make_archive, essentially completing the functionality of the archiving module I want to implement.
But I didn't use the shutil methods because they are limited to a small number of formats and people may need to use other formats, in that case if I would've written this module using these methods, it would've been very difficult to extend it to other formats, as there are other popular formats which these methods don't support.
This was my first iteration on this and it may be a good idea but I felt that we need something more maintainable and easy to extend, thus I came up with the idea of using a simple dictionary which would map format names to their respective methods from python standard library. something like:
```
  SUPPORTED_ARCHIVE_FORMATS = {
  'format_name' : class_that_handles_it
   .
   .
   .
   }
```
This looked a promising solution to me initially as new formats can be added directly to this dictionary and then used in the relevant methods.
But this was not the case, it turns out that the interface of various format specific archive handling methods are not very consistent, for example : For writing to a zip file the method used is zip.write(file) but the same function in tarfile is performed by tarfile.add(file) method.
All in all these little inconsistencies in the interface lead to the current design, which uses a helper class for each format and that class is registered in that common dictionary and each helper class inherits from an abstract base class which basically defines how the class should be implemented, and also provides some helper functions.
This way all the archive handling methods could be brought down to a consistent interface and can be used in related methods, also extending to new methods would be easy and they all should ideally work with the existing code like a charm.

What is coming up next?

I have made a rough implementation to get inputs from my mentor and to improve upon this.
Would need to implement test cases for this and more helper classes the current state of the PR can be seen here.
And then when I will add this to the model's base class and proceed to update tests cases for other models and make sure that they work with archives.
Also would need to update the file source as it has some archive handling code which should be removed and updated to use this module.

Did you get stuck anywhere?

Not really, this week was more about thinking and trying out various implementations,and evaluating them based on extensibility and maintainability, as I have discussed above.
I was a bit confused on how shall I implement the tests for this, but I think it would be better to take input from the mentors first and then put in effort in covering it in the tests.

Weekly Check-in #1 [June 7, 2021]

programmer290399@gmail.com (programmer290399) — Wed, 09 Jun 2021 01:36:05 +0000

Hello everyone !!

My name is Saahil Ali and I am a 3rd year student at IIPS,DAVV pursuing my integrated M.Tech (I.T.) currently. I have been working on DFFML from a few months now and I would be working on Adding archive support in DFFML this summer

What did you do this week?

I mostly focused on working towards finishing any open issues and requesting reviews for them so that I can completely focus on my project in this coming week. I also had a 1:1 meeting with my mentor, and yes also wrote this blog beforehand and waited for last date to publish only to find out that this site was down :P

What is coming up next?

I am planning to come up with a rough design of the archiving module, and would discuss and refine it with the help of my mentors and then would implement it with the test cases and documentation.

Did you get stuck anywhere?

Aah yes !! I was working on a commit linting issue from a long time and when it is almost ready for getting merged , I was getting some errors in the CI [1, 2] and I haven't been able to debug them yet, I have also asked my mentor about those errors and would continue to debug them and would eventually fix em all.