Last week I completed most of my proposed #2 feature which is "Post-Processing". Let me run you through how it happened.
The Idea
So this feature is actually an extension of a compression support idea. Initially it was supposed to be just an addition of compression feature to compress the feeds with the help of different compression algorithms. While discussing this idea with my mentor, he suggested a post-processing sort of thing rather than just a compression thing which I quite liked and expanded it a little. A few back and forth discussion with my mentor and this idea grew somewhat concrete.
The Plan
The finalized plan was that, the processing of feed would be done by "plugins", which would work in a pipeline-ish way by transferring the processed data from one plugin to another until it was finally written to the target file. These plugins will be managed by a single plugin manager which would load all the user declared plugins and connect their outputs to next-in-line plugin's input. The plugin manager will act as a wrapper to the feed's storage, so whenever a write event is invoked on the storage, it first goes through the manager which will then send the data sequentially to all the plugins until finally it reaches the destination file, processed as intended.
The Implementation
The plugin's interface was first decided and subsequently the builtin plugins were created: GzipPlugin, LZMAPlugin and Bz2Plugin. These plugins are, as you can see, compression based. Parameter passing was achieved through feed_options dictionary, the feed-specific options, where the users can declare the parameters. Next came the testing phase, where I initially made an all-in-one test method for the all the post-processing. My mentor suggested to create another elaborate test class, instead of one mega-test-method. Documentation is new for me, so my newbie skills were reflected on my poor documentation attempts. Fortunately, I have a cool mentor with a keen eye for detail so there were valuable inputs in my code reviews from which I got to learn a lot.