Articles on sakshamarora's Blog

Final Work Report - Last Weekly Check-in

sakshamarora1001@gmail.com (sakshamarora) — Mon, 31 Aug 2020 16:27:58 +0000

End of my GSoC journey - 31/08/2020

My GSoC'20 journey end today. This summer was a truly an amazing journey and surely an unforgettable one!

I would like to thank my mentors John Andersen for guiding me and being a very patient & supportive mentor, Yash Lamba for being supportive & understanding and Sudharsana for being helpful throughout my journey. Their guidance and support is the reason why this project was successful. I have learnt a ton of important things from them.

I'd also like to thank my fellow GSoC students Himanshu Tripathi and Aghin Shah Alin for helping me during the summer.

Thank you to Google and Python Software Foundation for providing this opportunity!

What did you do this week?

I finalized the Custom Neural Networks PR, so to summarize what this PR adds are the following stuff:

1. Create custom neural networks using config files for performing image processing tasks.
2. Replace last layer of torchvision pre-trained models to perform classification using powerful architectures.
3. Entrypoint style loading of Loss functions to modify their parameters.
4. Accommodate image processing tasks other than image classification.
5. Tests and example tutorial for creating networks using DFFML to classify Rock Paper Scissors hand poses.

Final Report and Future Work

Aim

Add ways of training and testing machine learning models in DFFML on image datasets and perform image processing and computer vision tasks via DFFML.

The project is divided into 2 parts:

Wrapping the image processing libraries namely OpenCV and Scikit-Image.
High Level Operation Workflow, i.e., Custom Operations which will act as high level operations implementing a predefined flow of OpenCV and Scikit-Image functions.

Modifications to the proposal and scope of the project

During the summer, the proposed work was modified to achieve better results. The finalized work done in the project:

Wrapping important image pre-processing functions as DFFML operations.
PyTorch based pre-trained Convolutional Networks for image classification.
Custom Neural Networks for image pre-processing and classification tasks via DFFML.

Project Tasks Completed

Sources for reading and pre-processing of image datasets

To enable working with image datasets and pre-process these datasets, the Directory source and DataFlow pre-processing source were added. The Directory source is used to read images stored in a directory format and DataFlow pre-processing source is used to modify the data using DFFML operations and creating a flow of operations to run the data through. The edit command uses DataFlow pre-processing Source to modify records using operations and provides an option to overwrite modified records on old records.

Related Links:
- https://github.com/intel/dffml/pull/604
  - DataFlow pre-processing Source
- https://github.com/intel/dffml/pull/824
  - Doctestable Example for DataFlow pre-processing Source
- https://github.com/intel/dffml/pull/644
  - Edit command to edit records present in a source
  - https://intel.github.io/dffml/master/cli.html#all
- https://github.com/intel/dffml/pull/718
  - Directory Source
Image Processing Operations

Added DFFML operations that wrap functions from OpenCV for pre-processing images after reading the image dataset from the source provided. The operations are put in a flow through which the image data runs by and is pre-processed before they are feeded to a machine learning model in DFFML for performing various tasks such as image classification!

Related Links:
- https://github.com/intel/dffml/pull/709
  - Added OpenCV functions as DFFML operations
  - PyPi: https://pypi.org/project/dffml-operations-image/
- https://github.com/intel/dffml/pull/731
  - Added more pre-processing functions and support for default values in operations
Convolutional Neural Networks

Visual data being very complex and containing high dimensions of features of same type in different orientation can be very hard to classify using classification models such as RandomForestClassifier, KNearestNeighbours, etc. This is where powerful deep learning models come into the picture. Deep learning networks like Convolutional Neural Networks offer great accuracy and versatility for tasks such as image classification, object detections, etc.

By adding Convolutional Neural Networks to DFFML, it becomes very easy to perform image processing techniques and get great results by using Transfer Learning or creating custom networks.

Related Links:
- https://github.com/intel/dffml/pull/784
  - Transfer learning PyTorch based models with dynamic loading for image classification & add documentations and tests for the same
  - PyPi: https://pypi.org/project/dffml-model-pytorch/
- https://github.com/intel/dffml/pull/777
  - Added example usages for classifying flower species using OpenCV and Transfer Learning approach and added tests for the examples
- https://github.com/intel/dffml/pull/839
  - Custom Neural Networks, custom layer addition support, loss entrypoint classes along with their example usage and tests
Many other important features that aided the addition and proper functioning of image processing tasks in DFFML are listed below:

Related Links:
- https://github.com/intel/dffml/pull/708
  - Improve String Representation for better viewing of image records
- https://github.com/intel/dffml/pull/768
  - Added support for loading images from fle formats namely JPEG, PNG and TIFF
  - PyPi: https://pypi.org/project/dffml-config-image/
- https://github.com/intel/dffml/pull/771
  - Add ability to add operations while creating a dataflow with different names so as to use same operations for different tasks in a single dataflow
- https://github.com/intel/dffml/pull/838
  - Ability to load YAML/JSON file formats as dictionaries via DFFML CLI

Future Work

DFFML is a machine learning based project which aligns with my interest in the field, so I will be more than happy to stay a part of the community to keep contributing and learning!

Goals for future contributions:

Adding more image processing techniques.
Adding image processing examples via DFFML's PyTorch model plugin such as image colorization.
Connecting Transfer Learning models with custom Neural Networks to use powerful architectures for different Computer Vision tasks.
Contribute to DFFML's Web UI.

Link to read the complete final report: GSoC2020FinalReport.md

Thank you for reading!

The Final Week - Weekly Check-in 13

sakshamarora1001@gmail.com (sakshamarora) — Mon, 24 Aug 2020 16:35:52 +0000

End of Week 12 - 24/08/2020

What did you do this week?

I added Entrypoints for loss functions in PyTorch to expose their parameters for tweaking. The optimizer functions in the official PyTorch library don't have parameter annotations as they are still working on adding annotations to everything in the PyTorch library, so I will be adding PyTorch optimizer entrypoints when a new torch release comes out! Now onto the final report! :D

What is coming up next?

As the coding period has officially ended, I will be writing my Final Submission Report this week!

Did you get stuck anywhere?

I got stuck with the entrypoint loading in DFFML while adding options to choose loss and optimizer functions and tweak their parameters. After discussion with the mentor we figured out and solved the problem!

Thank you for reading!

Custom Neural Networks - Weekly Check-in 12

sakshamarora1001@gmail.com (sakshamarora) — Mon, 17 Aug 2020 17:32:41 +0000

End of Week 11 - 17/08/2020

What did you do this week?

This week I worked on polishing the way one can create a use a custom neural network using DFFML. I will continue working on it and discussing with mentors on this front this week and then move onto the various applications of neural networks!

What is coming up next?

Continuing from last week's plans, I will work on making the code flexible to extend applications of CNNs in DFFML beyond classification tasks and support other neural network applications.

Did you get stuck anywhere?

I am currently working on making the PyTorch model plugin code flexible. Getting stuck at a few places but tackling all the issues one by one. :D

Thank you for reading!

Extending CNNs beyond classification - Weekly Check-in 11

sakshamarora1001@gmail.com (sakshamarora) — Mon, 10 Aug 2020 18:05:57 +0000

End of Week 10 - 10/08/2020

What did you do this week?

Iterating on last week's trial and error work on making it easier to add and connect Neural Networks (using the PyTorch library), this week, I made progress on that part! After discussions with the mentors, I devised a way to create and connect Neural networks that will make playing with neural networks easy and accessible to everyone! I am currently working on extending the applications of CNNs beyond classification and have support for various tasks that use CNNs.

What is coming up next?

I will continue working on Computer Vision related Custom Neural Networks and work on making the code flexible so as to not be confined to only classification applications of CNNs which is currently the case in the DFFML PyTorch model plugin.

Did you get stuck anywhere?

Yes, I got stuck at finding a way to make adding the neural networks using JSON and YAML files as flexible as possible so as to make adding neural networks more user friendly. I will discuss my doubts and ideas to tackle this in the next Weekly Sync meeting with the mentors.

Thank you for reading!

Phase 3 - Weekly Check-in 10

sakshamarora1001@gmail.com (sakshamarora) — Mon, 03 Aug 2020 18:27:45 +0000

End of Week 9 - 03/08/2020

What did you do this week?

This week, I planned on how DFFML will be able to provide a way that will make it very easy to work with Deep Neural Networks. I will be having further discussions with the mentors regarding how we can add support for connecting neural networks and make advanced and intricate architectures.

What is coming up next?

Next, I will working be on Custom Neural Networks mainly focused on Computer Vision based experiments and examples!

Did you get stuck anywhere?

No, this week was mostly planning for the project ahead.

Thank you for reading!

Dynamic Loading - Weekly Check-in 9

sakshamarora1001@gmail.com (sakshamarora) — Mon, 27 Jul 2020 15:46:45 +0000

End of Week 8 - 27/07/2020

What did you do this week?

Picking up from last week, I implemented dynamic loading of all the pre-trained torchvision models for image classification. This method creates the ``Model`` classes on the fly using type(), sys.modules and setattr() and loads the weights only for the model that is being used! I discussed with my mentor on more computer vision operations and examples that will be a great addition to DFFML.

What is coming up next?

After having a discussion with my mentor, we decided that we will work on exposing the optimization and loss function options available in PyTorch and layer support for making custom neural networks in a separate pull request next!

Did you get stuck anywhere?

No, this week went by smoothly!

Thank you for reading!

Convolutional Neural Networks - Weekly Check-in 8

sakshamarora1001@gmail.com (sakshamarora) — Mon, 20 Jul 2020 15:38:08 +0000

End of Week 7 - 20/07/2020

What did you do this week?

This week I started working on adding pre-trained Convolutional Neural Networks using the PyTorch Library! I have currently added VGG16 and ResNet18 pre-trained Models and will add more after further discussion with my mentor.

What is coming up next?

I will work on exposing the optimizing options available in the PyTorch library through DFFML to get the best results using the added models .

Did you get stuck anywhere?

I got a little confused on whether to split the train data into training and validation sets or not. Eventually I went with providing a splitting argument to split train data into validation and training data sets.

Thank you for reading!

Image Classification - Weekly Check-in 7

sakshamarora1001@gmail.com (sakshamarora) — Mon, 13 Jul 2020 16:52:25 +0000

End of Week 6 - 13/07/2020

What did you do this week?

I added example usage for using the OpenCV image processing operations to pre-process data before feeding into a Scikit-Learn classification model using DFFML's Command Line Interface. The dataset I used is the FLOWER17 dataset, to classify 17 flower species using their color, shape and texture as their feature maps to feed to the model! Although the accuracy isn't quiet good right now but it can get better if more pre-processing is done on the dataset especially if it's a multi-label classification problem. I've also started working to add pre-trained Convolutional Neural Networks.

What is coming up next?

Next, I will be adding pre-trained CNNs for the purpose of Image classification using Transfer Learning to DFFML.

Did you get stuck anywhere?

I had some issues with the performance of my laptop while training and testing on a large dataset of images. I have since shifted to using the Google Cloud Platform as suggested by my mentor and things have been going a lot smoother and faster now! :D

Thank you for reading!

Phase 2 - Weekly Check-in 6

sakshamarora1001@gmail.com (sakshamarora) — Mon, 06 Jul 2020 18:29:46 +0000

End of Week 5 - 06/07/2020

The first phase of GSoC ended and I'm very happy that I passed my first evaluation! At the beginning I was scared and unsure about a lot of things but I was able to make it through thanks to the very supportive mentors and fellow students who helped me! I always tried to give my best in any kind of task and I will continue giving my best in the upcoming weeks!

What did you do this week?

This week was a little slow but I did a lot of research on how we can get better accuracy with traditional Computer Vision techniques and what all processing operations are important to achieve this.

What is coming up next?

Next task is to discuss with my mentors and suggest on how the project will be going forward in this phase! I will be adding image processing operations if necessary and will discuss on the possibility of adding deep learning or OpenCV based models to get the best results while doing different computer vision tasks.

Did you get stuck anywhere?

As I was busy making my road-map ahead for the second phase, so I didn't have any major places I got stuck except some errors related to OpenCV functions but I eventually fixed them or went into the depths of internet to find the solution. :P

Thank you for reading!

Image Processing Operations Continued - Weekly Check-in 5

sakshamarora1001@gmail.com (sakshamarora) — Mon, 29 Jun 2020 10:07:10 +0000

End of Week 4 - 29/06/2020

What did you do this week?

I worked on adding the ability to use default parameters in operations to deal with optional parameters in OpenCV functions and wrote new image processing operations for performing various tasks such as extracting features like shape, color, texture, etc. With the directory source added last week, DFFML can now perform image classification on image datasets present in a directory format using the image processing operations to extract feature vectors to train and test on!

What is coming up next?

I'll continue to add operations and work on documenting how image classification can be done in DFFML using different image processing operations and their combinations to extract suitable feature vectors. For now, the next step is to add local feature extraction techniques such as ORB, KAZE, HOG, and more!

Did you get stuck anywhere?

Just some minor errors here and there. Nothing in particular. :)

Thank you for reading!

Image Processing Operations - Weekly Check-in 4

sakshamarora1001@gmail.com (sakshamarora) — Mon, 22 Jun 2020 15:30:43 +0000

End of Week 3 - 22/06/2020

What did you do this week?

I worked on finishing up the directory source for reading images and its test cases. The directory source will be able to read images for performing unsupervised classification too! Next, I have started writing new image processing operations for performing various tasks such as extracting features like shape, color, edges and more!

What is coming up next?

I'll continue to add image operations and start working on documenting how image classification will be performed using these operations in DFFML in the coming weeks!

Did you get stuck anywhere?

Got stuck at how the image operations may deal with default and optional parameters, but after discussions with my mentor things got clear.

Thank you for reading!

Images Images Images - Weekly Check-in 3

sakshamarora1001@gmail.com (sakshamarora) — Mon, 15 Jun 2020 09:56:45 +0000

End of Week 2 - 15/06/2020

What did you do this week?

I worked on getting the images stored in a directory to enter the network. For training data, the directory source reads the images and labels them with the same name as the sub-folder they are present in inside the train folder. The directory structure looks like this for training (supervised classification) :-

| -- dataset
| -- | -- train
| -- | -- | -- label_1
| -- | -- | -- | -- image_1.png
| -- | -- | -- | -- image_2.png
| -- | -- | -- | -- .....
| -- | -- | -- label_2
| -- | -- | -- | -- image_1.png
| -- | -- | -- | -- image_2.png
| -- | -- | -- | -- .....

I recently added a resize operation which would resize every image to be resized to the given provided new dimensions using the OpenCV resize function which will be reviewed by my mentor soon. I added a new flag to the CLI commands to display record data in a tabular format instead of JSON dumping them so as to make it easy to with long image arrays in a better way!

What is coming up next?

I will be working on finalising the directory source to accommodate for unsupervised classification too and on adding more functions from the OpenCV library to manipulate the image arrays and extract their features.

Did you get stuck anywhere?

Found out that the existing scikit models in DFFML are not able to train on features that are not single valued. So, I am trying to find a way to fix this bug for now.

Thank you for reading!

Brainstorming ideas - Weekly Check-in 2

sakshamarora1001@gmail.com (sakshamarora) — Mon, 08 Jun 2020 07:13:33 +0000

End of Week 1 - 08/06/2020

What did you do this week?

I had a lot of brainstorming sessions to decide on a well thought out layout for the project. Read lots of research papers, stack overflow answers, documentations and articles on image classification, pre-processing and feature extraction techniques. Tried out various image processing algorithms and feature extraction techniques and compiled a list of techniques and datasets I will be dealing with in the project. I had an individual meeting with the mentor where we discussed and decided upon the various ways we can go about the project. Also discussed the ways datasets would be handled by DFFML and what other possibilities there might be.

What is coming up next?

I will be starting on implementing ways to get the images from a dataset into the network and start wrapping various functions from the OpenCV library.

Did you get stuck anywhere?

Got stuck at lots of places like how will the image datasets get into the network and how will they be manipulated before finally extracting their features and feeding into a classification model, but got a clearer view on how things may work out after having discussions with my mentor.

Thank you for reading!

Beginning of Coding Period - Weekly Check-in 1

sakshamarora1001@gmail.com (sakshamarora) — Mon, 01 Jun 2020 01:29:01 +0000

>>> import Introduction
>>> Introduction.display()
Hello World! My name is Saksham Arora. I'm a 2nd year undergraduate student from India pursuing B. Tech in Information Technology. This is my blog for GSoC 2020 @ PSF!

Over the summer, I'll be working with DFFML under the umbrella of Python Software Foundation. My project for the summer is to Integrate Image Processing into DFFML!

GSoC 2020 Weekly Check-in 1 (End of Community Bonding Period - 01/06/2020)

What did you do this week?

Since it was the last week of Community Bonding period, I worked on finishing a few pending issues assigned to me, researched and brushed up on important topics related to my GSoC project. In the community bonding period, we had virtual meetings (also called Weekly Syncs) twice a week where I interacted with the mentors to discuss about new features and enhancements for the project. Also, I went through a few videos and documentation on asynchronous functions in python which was recommended by one of the mentors as a part of understanding the codebase better!

What is coming up next?

I will be adding the capability to normalize the MNIST dataset and pre-process any image provided for prediction on the dataset using the CLI before feeding it to a machine learning model. I will be discussing with the mentors on the best approach to get started on my project and start working on wrapping the OpenCV library with DFFML this week!

Did you get stuck anywhere?

I briefly got stuck at a unittesting error where it was trying to create a test class out of a decorator function which it shouldn't be doing, I was eventually able to figure it out after thoroughly going through the importing section in the Python documentation.

I'm very excited to get started on this journey. I hope everyone does great this summer!
Thank you for reading!