Hello! The third week of GSoC coding period is coming to an end. Here is an update on what I achieved in the past week and what I am looking forward to.
What did you do this week?
-
Created separate tox testing environments for testing integration with third-party parsers like Robotexclusionrulesparser and Reppy.
-
Made Travis use the new tox environments.
-
Described these parsers in Scrapy documentation.
-
Got Robotexclusionrulesparser to work with unicode user agents.
What is coming up next?
I will be working on creating a python based robots.txt parser which compliant with spec and supports modern conventions.
Did you get stuck anywhere?
Nothing major.