GSoC Final Report

Summary

GSoC comes to an end this week. Here I have presented my contributions to the Scrapy project that I made this summer. All the features that I have proposed were implemented. The last feature had to be divided into 2 parts: soft limits and hard limits. The soft limits part has been implemented but hard limit is yet to be completed.

Contributions

1. Issues

• Feeds Enhancement: Item Filters
• Feeds Enhancement: Post-Processing
• Feeds Enhancement: Batch Delivery Triggers

2. Pull Requests

• Item Filters
• Post-Processing
• Batch Triggers (Soft Limits)

3. Blog Posts: All my blog posts can be found here.

Future Work

Though the features have been implemented, there are scope of improvements:

• Make hard limits for batch delivery triggers
• Switching batches based on item content
• batch scheduling based on cron expression