<meta charset="utf-8">
Hello! This is my second last blog post for GSoC 2019 - time has gone by so quickly. I spend this week documenting Protego’s API in detail. I opened a pull request to add Protego integration in Scrapy. I added PyPy test environment and modified Protego to treat non-terminal dollar sign as ordinary character.
Up next, I will start the process to transfer Protego to Scrapy organisation on GitHub. I would modify `SitemapCrawler` in Scrapy to use the new interface, and implement a `ROBOTSTXT_USER_AGENT` setting in Scrapy.
I faced minor problems trying to setup PyPy environment in Travis. With the help from mentors, I was able to resolve the issue.