Weekly Check-in #4: (14 Jun - 20 Jun )

Published: 06/18/2019

Hello! The third week of GSoC coding period is coming to an end. Here is an update on what I achieved in the past week and what I am looking forward to.


What did you do this week?

  • Created separate tox testing environments for testing integration with third-party parsers like Robotexclusionrulesparser and Reppy.

  • Made Travis use the new tox environments.

  • Described these parsers in Scrapy documentation.

  • Got Robotexclusionrulesparser to work with unicode user agents.


What is coming up next?

I will be working on creating a python based robots.txt parser which compliant with spec and supports modern conventions.


Did you get stuck anywhere?

Nothing major.