Articles on aghinsa's Blog

Final Blog Post

aghinsa@gmail.com (aghinsa) — Mon, 31 Aug 2020 12:46:42 +0000

This has been a fantastic journey. I got to meet some amazing people in the community. There is still work to be done, and I'll be continuing to contribute after GSoC. You can check out my final report here https://gist.github.com/aghinsa/4f251b20cd1b6ecf34a13152d4ac3a2d

Weekly Check-in #13

aghinsa@gmail.com (aghinsa) — Thu, 27 Aug 2020 02:56:04 +0000

What did I do this week?

I added test cases to check whether the orchestrator can run operations in parallel. I also started working on the input network. Currently, I have added all the input network and an orchestrator instance to the main node.

What's next?

Finish testing
Finish implementing the input network
Documentation

Did I get stuck somewhere?

Yes. I got stuck in a deadlock. It was resolved with the help from mentors.

Weekly Check-in #12

aghinsa@gmail.com (aghinsa) — Mon, 17 Aug 2020 16:39:19 +0000

What did I do this week?

After discussing with my mentor, I decided to add more tests for the node connection stage before implementing the input network. In the new tests, each worker node will spin up its context, and all of the would-be alive at the same time working concurrently.

What's next?

Finish testing
Start implementing input network

Did I get stuck somewhere?

No.

Weekly Check-in #11

aghinsa@gmail.com (aghinsa) — Mon, 10 Aug 2020 19:49:47 +0000

What did I do this week?

Added test with multiple worker nodes. I started working on the input network.

What's next?

I'll be continuing to work on distributed orchestrator, specifically adding input network to orchestrator node.

Did I get stuck somewhere?

No.

Weekly Check-in #10

aghinsa@gmail.com (aghinsa) — Mon, 03 Aug 2020 18:28:17 +0000

What did I do this week?

Added a new command-line option, so that we can run dataflow without

the need for Sources and tests for the same. Updated docs to use new options.

What's next?

I'll be continuing working on distributed orchestrator, specifically adding input network.

Did I get stuck somewhere?

No.

Weekly Check-in #9

aghinsa@gmail.com (aghinsa) — Thu, 30 Jul 2020 01:30:10 +0000

What did I do this week?

I added support for Immediate response in the HTTP server. I also

added a new command-line option, so that we can run dataflow without

the need for Sources.

What's next?

I'll be adding tests for the same and updating the documentation to use the new features.

Did I get stuck somewhere?

No.

Weekly Check-in #8

aghinsa@gmail.com (aghinsa) — Mon, 20 Jul 2020 18:14:56 +0000

What did I do this week?

I finished milestone 1 of Distributed orchestrator. I also added tests and documentation for the Gitter chatbot.

What's next?

I'll be adding support for immediate responses in the HTTP server and Modifying the dataflow run command so that it can run without records.

Did I get stuck somewhere?

No. I was mostly writing tests and documentation this week.

Weekly Check-in #7

aghinsa@gmail.com (aghinsa) — Sat, 18 Jul 2020 04:20:38 +0000

What did I do this week?

I started working on Distributed Orchestrator. Currently, we decided the subnodes will act as worker nodes that run a specific set of operations. A primary node with a dataflow, request subnodes connected to the nats server for operations which it requires. Currently, I have coded till the part where subnodes spin up context upon receiving confirmation from the primary node.

What's next?

I'll be implementing input networks for both the primary and sub nodes.

Did I get stuck somewhere?

I have some doubts regarding the exceptions getting suppressed, I'll be discussing this with my mentor on today's meeting.

Weekly Check-in #7

aghinsa@gmail.com (aghinsa) — Mon, 13 Jul 2020 18:33:36 +0000

What did I do this week?

What's next?

I'll be implementing input networks for both the primary and sub nodes.

Did I get stuck somewhere?

I have some doubts regarding the exceptions getting suppressed, I'll be discussing this with my mentor on today's meeting.

Weekly Check-in #6

aghinsa@gmail.com (aghinsa) — Tue, 07 Jul 2020 16:40:27 +0000

<meta charset="utf-8">

What did I do this week?

Completed the chatbot example and added documentation for the same.

<meta charset="utf-8">

What's next?

I'll start working on the second phase of the project, implementing a distributed orchestrator.

Did I get stuck somewhere?

Yes, I had some doubts regarding the implementation of the distributed orchestrator. I had a meeting with my mentor today, and we have decided on an initial path.

Weekly Check-In: Week 5

aghinsa@gmail.com (aghinsa) — Mon, 29 Jun 2020 16:28:00 +0000

What did I do this week?

Completed the 'Locking' example with tests and documentation. I'm also working on the chatbot, which will be integrated with Gitter to run predictions from query messages.

What's next?

After finishing the code, I'll be writing the documentation for it. Once shared configs are implemented and merge, that'll be used in operations that make connections.

Did I get stuck somewhere?

No. This week's work was pretty simple, and most of the path was already discussed in previous meetings.

Weekly Check-In: Week 4

aghinsa@gmail.com (aghinsa) — Mon, 22 Jun 2020 16:10:06 +0000

Hey,

What did I do this week

We decided to skip the NLU part of the chatbot as the tutorial is about configs and dataflow. The chatbot will take structured data and make predictions on it. As a prerequisite to implementing the chatbot, I added a way to support shared configurations. As of now, each operation would need to open separate connections; with shared configs, this can be avoided. I also finished the locking example.

What's coming up next

I'll be finishing the documentation for the locking example and also complete the chatbot tutorial.

Did I get stuck anywhere

Implementing the shared config lead to some confusion but was resolved in the weekly meeting.

Weekly Check-in #3

aghinsa@gmail.com (aghinsa) — Mon, 15 Jun 2020 20:26:35 +0000

What did I do this week?

I read on implementing a chatbot, regarding a tutorial and realized that it would be too long, and something better would be suitable for the tutorial. I also worked on a PR that combines collecting data, prediction, and adding it to a database to a single DataFlow.

What's next?

I'll be adding the locking example this week. I'll also be discussing with the mentors as to what would be a suitable tutorial to demonstrate the use of configs.

Did I get stuck somewhere?

I'm currently stuck at adding the predictions to the database without using additional operations.I'll be discussing this with my mentor in a day and hopefully find a quick fix.

Week #2

aghinsa@gmail.com (aghinsa) — Mon, 08 Jun 2020 10:03:50 +0000

What did you do this week?

DFFML can now make dataflows out of arbitrary python functions, this week I changed the current ffmpeg example to use these new changes. I also added flow attribute to the dataflow create CLI command so that the input flow can now be modified. Support to receive and verify secrets from GitHub webhook was added. I also updated theses changes in the documentation of ffmpeg example usage.

What is coming up next?

I'll be adding new dataflow tutorial's covering specific areas and use cases.

Did you get stuck anywhere?

There were certain places where I had to discuss with the community to make suitable changes, the weekly meetings helped a lot in this regard.

Thanks for reading!

Weekly Check-in #01

aghinsa@gmail.com (aghinsa) — Mon, 01 Jun 2020 04:04:30 +0000

Hey all!! I'm Aghin Shah, a 3rd Year CS undergrad from IIT-Madras. I'll be working with DFFML, a sub-org under Python Software Foundation during GSoC on Implementing Distributed Orchestrator and Adding DataFlow tutorials.

What did I do this week?

I worked on setting up a DataFlow for coImplementingntinuous deployment of Docker containers. With the new flow, you can push your changes to GitHub, and it'll automatically pull and redeploy the defined containers. I've also been working on adding additional features to the CLI command for creating the DataFlow. We had meetings in the community (on Tuesdays and Fridays). Everyone was updated with the changes in the codebase. I also had an individual meeting with the mentor, where we discussed the possible ways to go about the project.

What is coming up next?

I'll be finishing patches for a couple of issues which I've been working on. I'll also start working on adding basic tutorials for DataFlow.

Did you get stuck anywhere?

There were a few places where I was confused, but my mentor is very active, so most of it was cleared by the day.