End of my GSoC journey - 31/08/2020

My GSoC'20 journey end today. This summer was a truly an amazing journey and surely an unforgettable one!

I would like to thank my mentors John Andersen for guiding me and being a very patient & supportive mentor, Yash Lamba for being supportive & understanding and Sudharsana for being helpful throughout my journey. Their guidance and support is the reason why this project was successful. I have learnt a ton of important things from them.

I'd also like to thank my fellow GSoC students Himanshu Tripathi and Aghin Shah Alin for helping me during the summer.

Thank you to Google and Python Software Foundation for providing this opportunity!

What did you do this week?

I finalized the Custom Neural Networks PR, so to summarize what this PR adds are the following stuff:

1. Create custom neural networks using config files for performing image processing tasks.
2. Replace last layer of torchvision pre-trained models to perform classification using powerful architectures.
3. Entrypoint style loading of Loss functions to modify their parameters.
4. Accommodate image processing tasks other than image classification.
5. Tests and example tutorial for creating networks using DFFML to classify Rock Paper Scissors hand poses.

Final Report and Future Work

Aim

Add ways of training and testing machine learning models in DFFML on image datasets and perform image processing and computer vision tasks via DFFML.

The project is divided into 2 parts:

1. Wrapping the image processing libraries namely OpenCV and Scikit-Image.
2. High Level Operation Workflow, i.e., Custom Operations which will act as high level operations implementing a predefined flow of OpenCV and Scikit-Image functions.

Modifications to the proposal and scope of the project

During the summer, the proposed work was modified to achieve better results. The finalized work done in the project:

1. Wrapping important image pre-processing functions as DFFML operations.
2. PyTorch based pre-trained Convolutional Networks for image classification.
3. Custom Neural Networks for image pre-processing and classification tasks via DFFML.

Project Tasks Completed

• Sources for reading and pre-processing of image datasets

  To enable working with image datasets and pre-process these datasets, the Directory source and DataFlow pre-processing source were added. The Directory source is used to read images stored in a directory format and DataFlow pre-processing source is used to modify the data using DFFML operations and creating a flow of operations to run the data through. The edit command uses DataFlow pre-processing Source to modify records using operations and provides an option to overwrite modified records on old records.

  Related Links:

  • https://github.com/intel/dffml/pull/604
    • DataFlow pre-processing Source
  • https://github.com/intel/dffml/pull/824
    • Doctestable Example for DataFlow pre-processing Source
  • https://github.com/intel/dffml/pull/644
    • Edit command to edit records present in a source
    • https://intel.github.io/dffml/master/cli.html#all
  • https://github.com/intel/dffml/pull/718
    • Directory Source

• Image Processing Operations

  Added DFFML operations that wrap functions from OpenCV for pre-processing images after reading the image dataset from the source provided. The operations are put in a flow through which the image data runs by and is pre-processed before they are feeded to a machine learning model in DFFML for performing various tasks such as image classification!

  Related Links:

  • https://github.com/intel/dffml/pull/709
    • Added OpenCV functions as DFFML operations
    • PyPi: https://pypi.org/project/dffml-operations-image/
  • https://github.com/intel/dffml/pull/731
    • Added more pre-processing functions and support for default values in operations

• Convolutional Neural Networks

  Visual data being very complex and containing high dimensions of features of same type in different orientation can be very hard to classify using classification models such as RandomForestClassifier, KNearestNeighbours, etc. This is where powerful deep learning models come into the picture. Deep learning networks like Convolutional Neural Networks offer great accuracy and versatility for tasks such as image classification, object detections, etc.

  By adding Convolutional Neural Networks to DFFML, it becomes very easy to perform image processing techniques and get great results by using Transfer Learning or creating custom networks.

  Related Links:

  • https://github.com/intel/dffml/pull/784
    • Transfer learning PyTorch based models with dynamic loading for image classification & add documentations and tests for the same
    • PyPi: https://pypi.org/project/dffml-model-pytorch/
  • https://github.com/intel/dffml/pull/777
    • Added example usages for classifying flower species using OpenCV and Transfer Learning approach and added tests for the examples
  • https://github.com/intel/dffml/pull/839
    • Custom Neural Networks, custom layer addition support, loss entrypoint classes along with their example usage and tests

• Many other important features that aided the addition and proper functioning of image processing tasks in DFFML are listed below:

  Related Links:

  • https://github.com/intel/dffml/pull/708
    • Improve String Representation for better viewing of image records
  • https://github.com/intel/dffml/pull/768
    • Added support for loading images from fle formats namely JPEG, PNG and TIFF
    • PyPi: https://pypi.org/project/dffml-config-image/
  • https://github.com/intel/dffml/pull/771
    • Add ability to add operations while creating a dataflow with different names so as to use same operations for different tasks in a single dataflow
  • https://github.com/intel/dffml/pull/838
    • Ability to load YAML/JSON file formats as dictionaries via DFFML CLI

Future Work

DFFML is a machine learning based project which aligns with my interest in the field, so I will be more than happy to stay a part of the community to keep contributing and learning!

Goals for future contributions:

1. Adding more image processing techniques.
2. Adding image processing examples via DFFML's PyTorch model plugin such as image colorization.
3. Connecting Transfer Learning models with custom Neural Networks to use powerful architectures for different Computer Vision tasks.
4. Contribute to DFFML's Web UI.

Link to read the complete final report: GSoC2020FinalReport.md

Thank you for reading!

End of Week 12  - 24/08/2020

What did you do this week?

I added Entrypoints for loss functions in PyTorch to expose their parameters for tweaking. The optimizer functions in the official PyTorch library don't have parameter annotations as they are still working on adding annotations to everything in the PyTorch library, so I will be adding PyTorch optimizer entrypoints when a new torch release comes out! Now onto the final report! :D 

What is coming up next?

As the coding period has officially ended, I will be writing my Final Submission Report this week!

Did you get stuck anywhere?

I got stuck with the entrypoint loading in DFFML while adding options to choose loss and optimizer functions and tweak their parameters. After discussion with the mentor we figured out and solved the problem!

Thank you for reading!

End of Week 11  - 17/08/2020

What did you do this week?

This week I worked on polishing the way one can create a use a custom neural network using DFFML. I will continue working on it and discussing with mentors on this front this week and then move onto the various applications of neural networks! 

What is coming up next?

Continuing from last week's plans, I will work on making the code flexible to extend applications of CNNs in DFFML beyond classification tasks and support other neural network applications.

Did you get stuck anywhere?

I am currently working on making the PyTorch model plugin code flexible. Getting stuck at a few places but tackling all the issues one by one. :D

Thank you for reading!

End of Week 10  - 10/08/2020

What did you do this week?

Iterating on last week's trial and error work on making it easier to add and connect Neural Networks (using the PyTorch library), this week, I made progress on that part! After discussions with the mentors, I devised a way to create and connect Neural networks that will make playing with neural networks easy and accessible to everyone! I am currently working on extending the applications of CNNs beyond classification and have support for various tasks that use CNNs. 

What is coming up next?

I will continue working on Computer Vision related Custom Neural Networks and work on making the code flexible so as to not be confined to only classification applications of CNNs which is currently the case in the DFFML PyTorch model plugin.

Did you get stuck anywhere?

Yes, I got stuck at finding a way to make adding the neural networks using JSON and YAML files as flexible as possible so as to make adding neural networks more user friendly. I will discuss my doubts and ideas to tackle this in the next Weekly Sync meeting with the mentors.

Thank you for reading!

End of Week 9  - 03/08/2020

What did you do this week?

This week, I planned on how DFFML will be able to provide a way that will make it very easy to work with Deep Neural Networks. I will be having further discussions with the mentors regarding how we can add support for connecting neural networks and make advanced and intricate architectures. 

What is coming up next?

Next, I will working be on Custom Neural Networks mainly focused on Computer Vision based experiments and examples!

Did you get stuck anywhere?

No, this week was mostly planning for the project ahead.

Thank you for reading!