GSoC 2020 PSF Blogs

Test post

ArcRiley@gmail.com (AR) — Wed, 09 Sep 2020 18:34:07 +0000

Testing for permissions

Week 14 blog!

cse170001012@iiti.ac.in (imaj_ashwini) — Thu, 03 Sep 2020 20:29:09 +0000

Hi everyone

This is the last blog for GSoC 2020. It was an amazing journey and an experience to cherish for lifetime.
I would like to thank Google for giving us, students this platform and Python Software Foundation for leading so many sub-organisations and the students towards the world of open source.
The mentors I got were amazing, probably I could not have asked for better mentors.

My work can be found here: https://github.com/panda3d/panda3d/pull/950

Thank you

Everyone, stay safe and happy :)

Final Check-in

abijithbahuleyan2@gmail.com (abijithbahuleyan) — Tue, 01 Sep 2020 03:35:16 +0000

Hey,

What did I do this week?

Cluster analysis template got merged. Mostly worked on the cluster status info. Also added support for the python script download support.

What's next?

Complete the current PR opened, and continue contributing to LiberTEM.

Did I get stuck somewhere?

No.

Final Words

dvijaywargiya@gmail.com (dvijaywargiya) — Tue, 01 Sep 2020 03:27:40 +0000

Link to final report

Link to repository

Link to production

Thank you Python Software Foundation for giving me this incredible opportunity. I got to learn a lot of new things and made several good contacts.
I hope to continue contributing to my organisation. Looking forward to what life throws at me next.

Super signing off!

Weekly Blog Post | GSoc | #14

shashankjarial10@gmail.com (ShashankJarial) — Tue, 01 Sep 2020 02:08:56 +0000

Greetings, People of the world!

Here's my last blog post for GSoC'2020. It has been an amazing journey. Full of learning and excitement every day. Here's a bit about the last week.

1. What did I do this week?

Wrapped things up, made a few animated icons and made the final GSoC report. Here's the link if you want to check it out: https://gitlab.com/ShashankJarial/gsoc-2020-project-report

2. What is coming next week?

Now that the program has almost ended, these blog posts will not be coming anymore but of course things will continue to come up right? I will continue to contribute to my project and to my organization's other projects. You can find me on linkedin: https://www.linkedin.com/in/shashank-jarial-354485152/ to know more about whats going on

3. Did you get stuck anywhere?

No I did not! I was a fun last week.

GSoC has been an amazing experience. I am really grateful to be a part of this. To anyone reading this if you are a GSoC aspirant, I'd say that just start started and you will reach great heights. And if you are someone who has been a part of GSoC or just here to read this blog, Thank you for reading this.

From console.log to GSoC 2020

sharmaaditya570191@gmail.com (sharmaaditya570191) — Mon, 31 Aug 2020 20:51:29 +0000

It is time to share my wonderful journey to Google Summer of Code, tips and tricks to get selected and what all you can do before GSoC to take a step closer to your dream internship or job.

I have written in detail about my journey here - https://medium.com/@sharmaaditya570191/from-console-log-to-gsoc-2020-1a6e9dc2334e

Thank you EOS Design System and Python Software Foundation for this awesome opportunity.

Signing off!

GSoC Weekly Blog #7

tanish1908@gmail.com (Tanish19) — Mon, 31 Aug 2020 16:55:17 +0000

After 3 months filled with coding, debugging and testing, GSoC'20 has finally come to an end. I spent this last week wrapping up the documentation and finishing up my final submission report for GSoC. I also discussed with my mentors the future steps we'll take with Mscolab.

My code was just recently merged with the develop branch of MSS and all tests passed!. I am really excited to have users use a piece of software that I wrote. I am already planning on what I want to work on in MSS after GSoC is over as there is a lot I can learn while working on this project.

These past 3 months have improved my coding skills significantly and introduced me to the amazing mentors at MSS. My time during GSoC has been absolutely wonderful and I can't wait to contribute to more interesting open source project. I hope next year, I would be able to mentor some bright students who are working on MSS .

Final Work Report - Last Weekly Check-in

sakshamarora1001@gmail.com (sakshamarora) — Mon, 31 Aug 2020 16:27:58 +0000

End of my GSoC journey - 31/08/2020

My GSoC'20 journey end today. This summer was a truly an amazing journey and surely an unforgettable one!

I would like to thank my mentors John Andersen for guiding me and being a very patient & supportive mentor, Yash Lamba for being supportive & understanding and Sudharsana for being helpful throughout my journey. Their guidance and support is the reason why this project was successful. I have learnt a ton of important things from them.

I'd also like to thank my fellow GSoC students Himanshu Tripathi and Aghin Shah Alin for helping me during the summer.

Thank you to Google and Python Software Foundation for providing this opportunity!

What did you do this week?

I finalized the Custom Neural Networks PR, so to summarize what this PR adds are the following stuff:

1. Create custom neural networks using config files for performing image processing tasks.
2. Replace last layer of torchvision pre-trained models to perform classification using powerful architectures.
3. Entrypoint style loading of Loss functions to modify their parameters.
4. Accommodate image processing tasks other than image classification.
5. Tests and example tutorial for creating networks using DFFML to classify Rock Paper Scissors hand poses.

Final Report and Future Work

Aim

Add ways of training and testing machine learning models in DFFML on image datasets and perform image processing and computer vision tasks via DFFML.

The project is divided into 2 parts:

Wrapping the image processing libraries namely OpenCV and Scikit-Image.
High Level Operation Workflow, i.e., Custom Operations which will act as high level operations implementing a predefined flow of OpenCV and Scikit-Image functions.

Modifications to the proposal and scope of the project

During the summer, the proposed work was modified to achieve better results. The finalized work done in the project:

Wrapping important image pre-processing functions as DFFML operations.
PyTorch based pre-trained Convolutional Networks for image classification.
Custom Neural Networks for image pre-processing and classification tasks via DFFML.

Project Tasks Completed

Sources for reading and pre-processing of image datasets

To enable working with image datasets and pre-process these datasets, the Directory source and DataFlow pre-processing source were added. The Directory source is used to read images stored in a directory format and DataFlow pre-processing source is used to modify the data using DFFML operations and creating a flow of operations to run the data through. The edit command uses DataFlow pre-processing Source to modify records using operations and provides an option to overwrite modified records on old records.

Related Links:
- https://github.com/intel/dffml/pull/604
  - DataFlow pre-processing Source
- https://github.com/intel/dffml/pull/824
  - Doctestable Example for DataFlow pre-processing Source
- https://github.com/intel/dffml/pull/644
  - Edit command to edit records present in a source
  - https://intel.github.io/dffml/master/cli.html#all
- https://github.com/intel/dffml/pull/718
  - Directory Source
Image Processing Operations

Added DFFML operations that wrap functions from OpenCV for pre-processing images after reading the image dataset from the source provided. The operations are put in a flow through which the image data runs by and is pre-processed before they are feeded to a machine learning model in DFFML for performing various tasks such as image classification!

Related Links:
- https://github.com/intel/dffml/pull/709
  - Added OpenCV functions as DFFML operations
  - PyPi: https://pypi.org/project/dffml-operations-image/
- https://github.com/intel/dffml/pull/731
  - Added more pre-processing functions and support for default values in operations
Convolutional Neural Networks

Visual data being very complex and containing high dimensions of features of same type in different orientation can be very hard to classify using classification models such as RandomForestClassifier, KNearestNeighbours, etc. This is where powerful deep learning models come into the picture. Deep learning networks like Convolutional Neural Networks offer great accuracy and versatility for tasks such as image classification, object detections, etc.

By adding Convolutional Neural Networks to DFFML, it becomes very easy to perform image processing techniques and get great results by using Transfer Learning or creating custom networks.

Related Links:
- https://github.com/intel/dffml/pull/784
  - Transfer learning PyTorch based models with dynamic loading for image classification & add documentations and tests for the same
  - PyPi: https://pypi.org/project/dffml-model-pytorch/
- https://github.com/intel/dffml/pull/777
  - Added example usages for classifying flower species using OpenCV and Transfer Learning approach and added tests for the examples
- https://github.com/intel/dffml/pull/839
  - Custom Neural Networks, custom layer addition support, loss entrypoint classes along with their example usage and tests
Many other important features that aided the addition and proper functioning of image processing tasks in DFFML are listed below:

Related Links:
- https://github.com/intel/dffml/pull/708
  - Improve String Representation for better viewing of image records
- https://github.com/intel/dffml/pull/768
  - Added support for loading images from fle formats namely JPEG, PNG and TIFF
  - PyPi: https://pypi.org/project/dffml-config-image/
- https://github.com/intel/dffml/pull/771
  - Add ability to add operations while creating a dataflow with different names so as to use same operations for different tasks in a single dataflow
- https://github.com/intel/dffml/pull/838
  - Ability to load YAML/JSON file formats as dictionaries via DFFML CLI

Future Work

DFFML is a machine learning based project which aligns with my interest in the field, so I will be more than happy to stay a part of the community to keep contributing and learning!

Goals for future contributions:

Adding more image processing techniques.
Adding image processing examples via DFFML's PyTorch model plugin such as image colorization.
Connecting Transfer Learning models with custom Neural Networks to use powerful architectures for different Computer Vision tasks.
Contribute to DFFML's Web UI.

Link to read the complete final report: GSoC2020FinalReport.md

Thank you for reading!

Outro

vn.mcsinyx@gmail.com (McSinyx) — Mon, 31 Aug 2020 14:48:23 +0000

Note: This article's HTML source is exported from reST. Without necessary CSS, some part might look hideous. Please consider viewing on my personal blog.

The Look

At the time of writing, implementation-wise parallel download is ready:

Does this mean I’ve finished everything just-in-time? This sounds to good to be true! And how does it perform? Welp…

The Benchmark

Here comes the bad news: under a decent connection to the package index, using fast-deps does not make pip faster. For best comparison, I will time pip download on the following cases:

Average Distribution

For convenience purposes, let’s refer to the commands to be used as follows

legacy-resolver: pip --no-cache-dir download {requirement}
2020-resolver: pip --use-feature=2020-resolver --no-cache-dir download {requirement}
fast-deps: pip --use-feature=2020-resolver --use-feature=fast-deps --no-cache-dir download {requirement}

In the first test, I used axuy and obtained the following results

legacy-resolver	2020-resolver	fast-deps
7.709s	7.888s	10.993s
7.068s	7.127s	11.103s
8.556s	6.972s	10.496s

Funny enough, running pip download with fast-deps in a directory with downloaded files already took around 7-8 seconds. This is because to lazily download a wheel, pip has to make many requests which are apparently more expensive than actual data transmission on my network.

Note

With unstable connection to PyPI (for some reason I am not confident enough to state), this is what I got

2020-resolver	fast-deps
1m16.134s	0m54.894s
1m0.384s	0m40.753s
0m50.102s	0m41.988s

As the connection was unstable and that the majority of pip networking is performed as CI/CD with large and stable bandwidth, I am unsure what this result is supposed to tell (-;

Large Distribution

In this test, I used TensorFlow as the requirement and obtained the following figures:

legacy-resolver	2020-resolver	fast-deps
0m52.135s	0m58.809s	1m5.649s
0m50.641s	1m14.896s	1m28.168s
0m49.691s	1m5.633s	1m22.131s

Distribution with Conflicting Dependencies

Some requirement that will trigger a decent amount of backtracking by the current implementation of the new resolver oslo-utils==1.4.0:

2020-resolver	fast-deps
14.497s	24.010s
17.680s	28.884s
16.541s	26.333s

What Now?

I don’t know, to be honest. At this point I’m feeling I’ve failed my own (and that of other stakeholders of pip) expectation and wasted the time and effort of pip’s maintainers reviewing dozens of PRs I’ve made in the last three months.

On the bright side, this has been an opportunity for me to explore the codebase of package manager and discovered various edge cases where the new resolver has yet to cover (e.g. I’ve just noticed that pip download would save to-be-discarded distributions, I’ll file an issue on that soon). Plus I got to know many new and cool people and idea, which make me a more helpful individual to work on Python packaging in the future, I hope.

Weekly Check-in #13

himanshutripathi366@gmail.com (0dust) — Mon, 31 Aug 2020 14:12:47 +0000

What did I do this week?

Finalised everything! My last PR on scikit operations tutorial got merged and this marks the completion of GSoC project.

What's next?

I will keep contributing to DFFML and help others to come and contribute to open source. :)

Did I get stuck somewhere?

No, this week didn't involve any new work, I only went through the old work that I did.

Final Blog Post

aghinsa@gmail.com (aghinsa) — Mon, 31 Aug 2020 12:46:42 +0000

This has been a fantastic journey. I got to meet some amazing people in the community. There is still work to be done, and I'll be continuing to contribute after GSoC. You can check out my final report here https://gist.github.com/aghinsa/4f251b20cd1b6ecf34a13152d4ac3a2d

Final Week Check-in

livingfromtheoutsidein@gmail.com (joaosferreira) — Mon, 31 Aug 2020 12:19:35 +0000

What did you do this week?

This week I concluded the PR for the random module by adding multimethods for the Generator class. This PR was the last one of my GSoC project. I also wrote the project's final report for my third and final evaluation where I included a short description of the project, links to the the code, a summary of the current state of the project, discussed future work, and enumerated challenges and learnings.

What will you do after GSoC has ended?

They say all good things come to an end and this blog post marks the end of my GSoC journey. Soon I will start working on my master's thesis and look for job opportunities in the industry. I also intend to keep working on unumpy and other open-source software projects.

All Done!

aryangupta973@gmail.com (ARYAN_GUPTA) — Mon, 31 Aug 2020 08:53:17 +0000

Hello there!

Today is the final day of my GSoC Journey. I am truly grateful for all that I have learnt the past 3 months. Here's my final overview of my work in the last week.

What did you do this week?

My mentors tested out my product and found many bugs, and advised changes for better use. As a developer, using the same features everyday, you get the feeling that users will be able to use it as easily. But its always better to have a third perspective. The best perspective is that of the users, and my mentors are exactly that. I implemented the changes (2- 3 PRs) .

The changes implemented messed up some of the tests. Not All. (Finally understood what Reimar had meant when he said that all tests should be independent from each other. Thanks for the tip!)

I worked on the Documentation, gave a small demo to the mentors, and made a demo video as well! So, a very productive final week :))

I am very happy to say that my code has been merged into the main development branch. And I'll continue to better the software from that branch now :))

What will I do next week?

I will continue to work on my skills, apart from GSoC.

Did you get stuck anywhere?

I was reluctant with some of the changes.. simply because I am a bit attached to my work. But mentors pointed out changes logically, and I agreed with them too . Other than this, there was no getting stuck.

You can find my Work Submission report here. You can check out my demo video right here!

Acknowledgements

I'm thankful to my mentors Jörn, Reimar and Christian who provided immense support and motivation to persevere through all my difficulties. They sympathised with my problems and gave me ideas on how to face the challenges head on. Without their guidance, I would have been lost. I would also like to thank my GSoC mate Tanish who helped me out with my small doubts from time to time :))

I'm also grateful to MSS, PSF & the GSoC Community. This program has made me more confident about my abilities, and has strengthened my belief in the fact that passion, perseverance and patience can help you achieve wonders!

Finally, to the Open Source Community : Thank you for being so welcoming. I started from scratch and three months later, its still hard for me to believe that I developed a feature of a software :)) Thank you for giving me this opportunity, and I'll personally continue forward the Open Source spirit with my future contributions.

GSoC: Week #14

anandbaburajan@gmail.com (anandbaburajan) — Mon, 31 Aug 2020 06:48:25 +0000

Hello!

What did you do this week?

I wrote docs, looked into K2IS’ sync_flag and completed implementing positive sync_offset for HDF5. I also extracted the recursive slice splitting code in HDF5 into its own function and wrote tests for it. While working on HDF5, I faced a strange bug in which the unit tests passed when ran on real datasets but failed when ran on randomly generated temporary data.

What is coming up next?

Handle combinations of sync_offset and reshaping, fix sig_shape reshaping and the bug mentioned above for HDF5. I’ve almost figured out how the sectors are synced in K2IS, so hope to finish it soon. I’ve a few more things on my todo list before my PR can be merged and I think this would be my last GSoC blog, so thanks a lot for reading!

Did you get stuck anywhere?

No, but I must say that the best and most challenging part of this journey was deciding between different algorithmic/design solutions to solve problems in my project.

Thanks again to my mentors, PSF and Google for this opportunity!

:-)

Weekly Check In - 12

k.aditya00@gmail.com (adityaa30) — Sun, 30 Aug 2020 12:59:23 +0000

What did I do till now?

Last week I was working on finishing up the HTTPNegotiateDownloadHandler. Presently the download handler uses ALPN or NPN (whichever is available) to negotiate a protocol (presently one of HTTP/1.1 or HTTP/2) from the remote server and issues the requests on the respective download handler. Presently, all requests made via proxy are directly issued using the HTTP11DownloadHandler.

What's coming up next?

I plan on continue working on implementing the CONNECT method for HTTP/2.

Did I get stuck anywhere?

Yep. I was stuck for almost a week on the CONNECT protocol. Now, I have managed to fix the bug where the raw TCP connection instance could not be switched to HTTP/2. However, there are some issues during the TLS handshake with the final target resource 😥.