Articles on siddharthadr's Bloghttps://blogs.python-gsoc.orgUpdates on different articles published on siddharthadr's BlogenWed, 18 Aug 2021 07:23:14 +0000GSoC Final Reporthttps://blogs.python-gsoc.org/en/siddharthadrs-blog/gsoc-final-report/<h3> Summary </h3> <p> GSoC comes to an end this week. Here I have presented my contributions to the Scrapy project that I made this summer. All the features that I have proposed were implemented. The last feature had to be divided into 2 parts: soft limits and hard limits. The soft limits part has been implemented but hard limit is yet to be completed. </p> <h3> Contributions </h3> <p> 1. Issues </p><ul> <li><a href="https://github.com/scrapy/scrapy/issues/5161">Feeds Enhancement: Item Filters</a></li> <li><a href="https://github.com/scrapy/scrapy/issues/5168">Feeds Enhancement: Post-Processing</a></li> <li><a href="https://github.com/scrapy/scrapy/issues/5169">Feeds Enhancement: Batch Delivery Triggers</a></li> </ul> 2. Pull Requests <ul> <li><a href="https://github.com/scrapy/scrapy/pull/5178">Item Filters</a></li> <li><a href="https://github.com/scrapy/scrapy/pull/5190">Post-Processing</a></li> <li><a href="https://github.com/scrapy/scrapy/pull/5205">Batch Triggers (Soft Limits)</a></li> </ul> 3. Blog Posts: All my blog posts can be found <a href="https://blogs.python-gsoc.org/en/siddharthadrs-blog/">here.</a> <p></p> <h3> Future Work </h3> <p> Though the features have been implemented, there are scope of improvements: </p><ul> <li>Make hard limits for batch delivery triggers</li> <li>Switching batches based on item content</li> <li>batch scheduling based on cron expression</li> </ul> <p></p>siddharthadr11@gmail.com (siddharthadr)Wed, 18 Aug 2021 07:23:14 +0000https://blogs.python-gsoc.org/en/siddharthadrs-blog/gsoc-final-report/Blog Post #5https://blogs.python-gsoc.org/en/siddharthadrs-blog/blog-post-5-3/<h3> The Past Week </h3> <p> Last week was spent improving documentation about the final feature - batch triggers. Hopefully the feature is now in a merge-able state. This feature can be improved a lot though so I'll continue to work on it. </p> <h3> Phasing Out.. </h3> <p> This week brings an end to GSoC. It has been a wonderful experience for me! I'd like to thank my mentors Adrián Chaves and Aditya Kumar for their support and help. Scrapy has been a cool project to work on and I hope I continue to contribute to it. </p>siddharthadr11@gmail.com (siddharthadr)Sun, 15 Aug 2021 19:19:00 +0000https://blogs.python-gsoc.org/en/siddharthadrs-blog/blog-post-5-3/Weekly Check-in #6https://blogs.python-gsoc.org/en/siddharthadrs-blog/weekly-check-in-6-22/<p> <b>What did you do this week?</b><br> A. I am almost done with soft limits for batch delivery triggers. </p> <p> <b>What is coming up next?</b><br> A. Up next is converting the soft limits to hard limits. </p> <p> <b>Did you get stuck anywhere?</b><br> A. Nothing major. </p>siddharthadr11@gmail.com (siddharthadr)Sun, 08 Aug 2021 18:36:19 +0000https://blogs.python-gsoc.org/en/siddharthadrs-blog/weekly-check-in-6-22/Blog Post #4https://blogs.python-gsoc.org/en/siddharthadrs-blog/blog-post-4-3/<h3> The Past Week </h3> <p> My University classes have finally began and with that, placement season as well so I am kinda getting behind my schedule. Though I managed to write tests for file-size and duration batch delivery triggers. As these are soft limits, creating tests for them was tricky. </p> <h3> Work Done So Far </h3> <p> The added triggers are called soft limits because the limits are only checked when an item is scraped. So the triggers will not always be triggered accurately. I was a little unsure how to proceed with the tests but I affirmed with my mentor's suggestion and proceeded with that. So tests for soft limits can have 3 different general cases(presuming we have 2 total items): i) when the limit is zero(so no limits are imposed), ii) the smallest unit is the limit(so only 1 item is accepted), iii) when the limit causes at most 1 item per batch iv) when the limit causes at most 2 items per batch. </p> <h3> Work Ahead </h3> <p> I have yet to add tests for custom batch triggers. With addition of documentations I will finalize my soft limit triggers. After that I will have to convert the soft limits to hard ones. </p>siddharthadr11@gmail.com (siddharthadr)Sun, 01 Aug 2021 18:17:55 +0000https://blogs.python-gsoc.org/en/siddharthadrs-blog/blog-post-4-3/Weekly Check-in #5https://blogs.python-gsoc.org/en/siddharthadrs-blog/weekly-check-in-5-22/<p> <b>What did you do this week?</b><br> A. I added soft limits for file size and time duration as batch triggers. I tried figuring out ways to find hard limits for them as well. </p> <p> <b>What is coming up next?</b><br> A. Finalize plans for hard limits and start its implementation. </p> <p> <b>Did you get stuck anywhere?</b><br> A. I am still in the middle of figuring out hard limits for time duration and file size. </p>siddharthadr11@gmail.com (siddharthadr)Sun, 25 Jul 2021 18:39:47 +0000https://blogs.python-gsoc.org/en/siddharthadrs-blog/weekly-check-in-5-22/Blog Post #3https://blogs.python-gsoc.org/en/siddharthadrs-blog/blog-post-3-2/<p> After finalizing the Post-Processing feature, I have moved onto implementing my third proposed feature - Batch Delivery Triggers. This feature will be the toughest of the three. Further I will talk about what it really is and how am tackling this. </p> <h3>What is this feature?</h3> <p> In Scrapy 2.3.0, a <a href="https://docs.scrapy.org/en/latest/topics/feed-exports.html?highlight=batch#feed-export-batch-item-count">batch creating feature</a> was introduced which generates multiple output files based on a specified item count constraint. Such that whenever a file's item count exceeds a limit, a new fresh file is created for storing further items. What I intend to do, is add more delivery triggers, namely - time duration limit and file size limit while also providing users with a way to create their own custom trigger. </p> <h3>The Plan</h3> <p> To have the ability to use custom batch delivery triggers, I had planned to use a base <code>BatchHandler</code> class to use as a default handler for batch creation for triggers such as file size, item count and time duration. This will allow to easily replace this base class with a custom class, if the user desires to use one. </p> <h3>My Work So Far</h3> <p> I have created a base class with the planned methods for batch handling. Next was modifying the <code>FeedExporter</code> to so that it can use the <code>BatchHandler</code> classes and use them to determine when to start a new batch. So far it works for item count limits and file size limit but there's been some problems implementing a trigger for time duration.<br> <br> Initially I came with a very naive approach to update the time elapsed for a batch whenever an item is scraped and create a new batch if the limit is passed. Obvious problems can be deduced from that sentence itself. There are no guarantees that an item is scraped instantaneously, so a batch duration may very well cross the limit by a lot by the time an item is scraped.<br> <br> One possible solution I have been thinking is to "schedule" or call the function to create a new batch for the feed after the specified duration has elapsed. But if some other declared constraint has breached, a new batch will be created then and timer will be reset. Using Twisted framework's <code>task.deferLater</code> or <code>reactor.callLater</code> could be the answer to my problem. But I think these methods are not thread-safe and will need some sort of mutexes to guarantee safety to the code. I will need to come up with some control flow plan to ensure the batches are created timely without threatening the safety of the code. </p>siddharthadr11@gmail.com (siddharthadr)Sun, 18 Jul 2021 19:10:40 +0000https://blogs.python-gsoc.org/en/siddharthadrs-blog/blog-post-3-2/Weekly Check-in #4https://blogs.python-gsoc.org/en/siddharthadrs-blog/weekly-check-in-4-19/<p> <b>1. What did you do this week?</b><br> A. I extended the tests for Post-Processing and completed the documentation for it. The feature was completed and approved by my mentor. </p> <p> <b>2. What is coming up next?</b><br> A. Now the last proposed feature Batch Trigger remains. I will polish its user API and start the implementation process. </p> <p> <b>3. Did you get stuck anywhere?</b><br> A. Nothing major this week. </p>siddharthadr11@gmail.com (siddharthadr)Mon, 12 Jul 2021 07:56:26 +0000https://blogs.python-gsoc.org/en/siddharthadrs-blog/weekly-check-in-4-19/Blog Post #2https://blogs.python-gsoc.org/en/siddharthadrs-blog/blog-post-2-2/<p> Last week I completed most of my proposed #2 feature which is "Post-Processing". Let me run you through how it happened. </p> <h3> The Idea </h3> <p> So this feature is actually an extension of a compression support idea. Initially it was supposed to be just an addition of compression feature to compress the feeds with the help of different compression algorithms. While discussing this idea with my mentor, he suggested a post-processing sort of thing rather than just a compression thing which I quite liked and expanded it a little. A few back and forth discussion with my mentor and this idea grew somewhat concrete. </p> <h3> The Plan </h3> <p> The finalized plan was that, the processing of feed would be done by "plugins", which would work in a pipeline-ish way by transferring the processed data from one plugin to another until it was finally written to the target file. These plugins will be managed by a single plugin manager which would load all the user declared plugins and connect their outputs to next-in-line plugin's input. The plugin manager will act as a wrapper to the feed's storage, so whenever a write event is invoked on the storage, it first goes through the manager which will then send the data sequentially to all the plugins until finally it reaches the destination file, processed as intended. </p> <h3> The Implementation </h3> <p> The plugin's interface was first decided and subsequently the builtin plugins were created: <code>GzipPlugin</code>, <code>LZMAPlugin</code> and <code>Bz2Plugin</code>. These plugins are, as you can see, compression based. Parameter passing was achieved through <code>feed_options</code> dictionary, the feed-specific options, where the users can declare the parameters. Next came the testing phase, where I initially made an all-in-one test method for the all the post-processing. My mentor suggested to create another elaborate test class, instead of one mega-test-method. Documentation is new for me, so my newbie skills were reflected on my poor documentation attempts. Fortunately, I have a cool mentor with a keen eye for detail so there were valuable inputs in my code reviews from which I got to learn a lot. </p>siddharthadr11@gmail.com (siddharthadr)Sun, 04 Jul 2021 20:26:53 +0000https://blogs.python-gsoc.org/en/siddharthadrs-blog/blog-post-2-2/Weekly Check-in #3https://blogs.python-gsoc.org/en/siddharthadrs-blog/weekly-check-in-3-23/<p> <b>1. What did you do this week?</b><br> A. This week I created plugins' manager <code>PostProcessingManager</code>, <code>Plugin</code> interface and builtin compression plugins (gzip, bzip2, lzma). PR availabe here: <a href="https://github.com/scrapy/scrapy/pull/5190">#5190</a>. </p> <p> <b>2. What is coming up next?</b><br> A. This week I will focus on the tests for the created plugins and manager. </p> <p> <b>3. Did you get stuck anywhere?</b><br> A. While implementing the manager and builtin plugins I had assumed that only two methods will be used ever: <code>write</code> and <code>close</code>. Well it turned out one of the <a href="https://docs.scrapy.org/en/latest/topics/exporters.html#module-scrapy.exporters">exporters</a> (used for exporting a scraped item to a file), <code>CsvItemExporter</code> also wraps the storage file around <code>io.TextIOWrapper</code> and <b>that</b> actually expects an object with at least the methods and attributes of that of <code>io.IOBase</code>. The problem was fixed by making <code>PostProcessingManager</code> subclass <code>io.IOBase</code> and also providing a method <code>writable</code> to indicate that the object is open for writing. </p>siddharthadr11@gmail.com (siddharthadr)Sun, 27 Jun 2021 19:39:19 +0000https://blogs.python-gsoc.org/en/siddharthadrs-blog/weekly-check-in-3-23/Blog Post #1https://blogs.python-gsoc.org/en/siddharthadrs-blog/blog-post-1-2/<p> Going into the 3rd week of GSoC, the <a href="https://github.com/scrapy/scrapy/pull/5178">Item Filter feature</a> is almost complete and ready to go. In this post I'll give a rundown how I implemented it. </p> <br> <h3>What is this feature?</h3> <p> When Scrapy scrapes some data from the interwebs, it saves the scraped item to the declared <a href="https://docs.scrapy.org/en/latest/topics/feed-exports.html#feed-exports">feed storages</a>. Scrapy provides a lot of flexibility in choosing what this storage can be, ranging from simple csv files to S3 cloud storages. But currently everytime some data is scraped, it is stored in all storages. Some users may require storing certain types of data in certain storage. So let's make it convenient for the users to do so. </p> <h3>How to implement such a feature?</h3> <p> We can introduce a certain type of <i>filter</i> for each storage, so that whenever some data is scraped, the storage can check if the data is acceptable and if it is then we can export it to the storage. Well that's really simple, right? So how to do that?<br> Let's create a class called <i>ItemFilter</i> that will handle the filtering process and assign this class to a declared storage. As we can have more than one data storage, we can assign different <i>ItemFilter</i> classes to different storages with each filter having their own logic to filter scraped data. So we create our filter, assign it to one of the storages in <a href="https://docs.scrapy.org/en/latest/topics/feed-exports.html#settings">settings.py</a> file and voila. Another convenient thing we can do is allow users to just declare what sort of <a href="https://docs.scrapy.org/en/latest/topics/items.html">data <i>Item</i></a>, they wish to accept for a particular storage. So they can just put a list of Items for a storage in settings.py file instead of creating a filter class. </p> <h3>My Work</h3> <p> As this was a straightforward implementation, there wasn't much difficulty in implementing this feature apart from some design changes. Though testing phase was a little different for this feature than what I'm familiar with. For testing the Feed Storages, we need to decorate every test method with a <a href="https://twistedmatrix.com/documents/13.2.0/api/twisted.internet.defer.inlineCallbacks.html"><i>@defer.inlineCallbacks</i></a>. It is part of Twisted framework and is used when you want to write a sequential code while using Deferred objects. This was new to me and I had to read about it and why it was used. I got to learn about Deferred objects and how to write code for it. Also this was the first time I was writing documentation in reStructuredText format. </p>siddharthadr11@gmail.com (siddharthadr)Sun, 20 Jun 2021 16:07:27 +0000https://blogs.python-gsoc.org/en/siddharthadrs-blog/blog-post-1-2/Weekly Check-in #2https://blogs.python-gsoc.org/en/siddharthadrs-blog/weekly-check-in-2-12/<p> Weather's just great here now, with the Monsoon arriving here. I can now enjoy the rains while sipping coffee and coding for GSoC with some cozy tunes to go along with. </p> <p> 1. What did you do this week?<br> A. So the ItemChecker class that I proposed has been included in the codebase with the appropriate modifications in FeedExport, so that it can use those filter classes. I pushed the code and created a draft PR. It can be accessed here: <a href="https://github.com/scrapy/scrapy/pull/5178">#5178</a>. My mentor left some feedback which I agreed with. </p> <p> 2. What is coming up next?<br> A. I will look into mentor's feedback and improve the code. After that I'll continue with adding tests and documentations. </p> <p> 3. Did you get stuck anywhere?<br> A. I did once got stuck when I was manually testing the new implementation. I had actually added a class instance attribute called item_classes to ItemChecker class which I intended to be changed for every Feed slot. The problem I was facing was whenever I changed any one of the item_classes attribute it reflected on other instance's attribute as well. The problem was quite trivial actually and I should have known why it was happening. The reason why it was happening was simply because class attributes are shared amongst all the classes. And not only that I also learned about how default parameter values worked in Python which explains why class attributes work the way they do. For the curious, follow this <a href="https://stackoverflow.com/questions/1132941/least-astonishment-and-the-mutable-default-argument">Stackoverflow thread</a> to understand more about default parameter values in Python.</p>siddharthadr11@gmail.com (siddharthadr)Mon, 14 Jun 2021 18:40:15 +0000https://blogs.python-gsoc.org/en/siddharthadrs-blog/weekly-check-in-2-12/Weekly Check-in #1https://blogs.python-gsoc.org/en/siddharthadrs-blog/weekly-check-in-1-8/<p> Alright then as this is the first ever post for GSoC let me introduce myself and what I'm doing. My name is D R Siddhartha, friends call me DR(yeah those two letters). I like running and hiking(hopefully I get to hike Iceland one day). I'll be working on Scrapy, a web-crawling framework. Specifically I'll be working on its Feed features, adding enhancements. </p> <p> Now about the 3 mandatory questions:<br> 1. What did you do this week?<br> A. I started the week by getting a vaccine shot. The side effects manifested the next day and was horrendous. So I had to delay the scheduled meeting. But me and my mentor have already been discussing on the API designs for the enhancements. We made the discussions public in Github so other people can discuss as well. The discussions are available here: <a href="https://github.com/scrapy/scrapy/issues/5168">#5168</a>, <a href="https://github.com/scrapy/scrapy/issues/5161">#5161</a> and <a href="https://github.com/scrapy/scrapy/issues/5169">#5169</a>. </p> <p> 2. What is coming up next?<br> A. The coding phase starts now. I'll be starting my work on the first of the 3 features that I'll be implementing - Item Filters. Some API discussions have already been done. So now the implementation work starts. </p> <p> 3. Did you get stuck anywhere?<br> A. Not so far. </p>siddharthadr11@gmail.com (siddharthadr)Mon, 07 Jun 2021 19:06:01 +0000https://blogs.python-gsoc.org/en/siddharthadrs-blog/weekly-check-in-1-8/Testhttps://blogs.python-gsoc.org/en/siddharthadrs-blog/test/Test [link](www.site.com)siddharthadr11@gmail.com (siddharthadr)Fri, 21 May 2021 14:36:18 +0000https://blogs.python-gsoc.org/en/siddharthadrs-blog/test/