Articles on anubhavp's Blog

[Blog #6] Part of the journey is the end.

anubhavp28@gmail.com (anubhavp) — Sun, 25 Aug 2019 20:07:38 +0000

Part of the journey is the end. It is time for me to work on my final work report for final evaluation of Google Summer of Code 2019. This week, I will devote my time mainly to write my final report and final blog post. If time permits, I will work on my PRs from last week.

Last week, I worked on getting Travis to push automatically to PyPI and I redid benchmarking.

[Blog #5] Time just seems to fly.

anubhavp28@gmail.com (anubhavp) — Sun, 25 Aug 2019 19:52:41 +0000

Hello! This is my second last blog post for GSoC 2019 - time has gone by so quickly. I spend this week documenting Protego’s API in detail. I opened a pull request to add Protego integration in Scrapy. I added PyPy test environment and modified Protego to treat non-terminal dollar sign as ordinary character.

Up next, I will start the process to transfer Protego to Scrapy organisation on GitHub. I would modify `SitemapCrawler` in Scrapy to use the new interface, and implement a `ROBOTSTXT_USER_AGENT` setting in Scrapy.

I faced minor problems trying to setup PyPy environment in Travis. With the help from mentors, I was able to resolve the issue.

Weekly Check-in #13 : ( 16 Aug - 22 Aug )

anubhavp28@gmail.com (anubhavp) — Tue, 20 Aug 2019 13:10:39 +0000

What did you do this week?

I worked on getting Travis to push releases automatically to PyPI, adding a new `ROBOTSTXT_USER_AGENT` setting in Scrapy, and improvements to SitemapSpider.

What is coming up next?

I am going to work on the final PSF blog post in which I will focus on my experience of GSoC 2019 working with my awesome mentors, and Scrapy.
Next, I will write a final report for third evaluation of Google Summer of Code.
Next up, I will work on the changes suggested on my this week's work.

Did you get stuck anywhere?

Nothing Major.

Weekly Check-in #12: (9 Aug - 15 Aug)

anubhavp28@gmail.com (anubhavp) — Tue, 13 Aug 2019 12:59:47 +0000

What did you do this week?

Benchmarking Protego (again). This time we crawled multiple domains (~1,100 domains) and downloaded links to pages as the crawler encounter them. We downloaded 111, 824 links in total.
- Next we made each robots.txt parser - parse and answer query (we made parsers answer each query 5 times) in an order similar to how they would need to in a broad crawl. Here are the results :

Protego :

25th percentile : 0.002419 seconds
50th percentile : 0.006798 seconds
75th percentile : 0.014307 seconds
100th percentile : 2.546978 seconds
Total Time : 19.545984 seconds

RobotFileParser (default in Scrapy) :

25th percentile : 0.002188 seconds
50th percentile : 0.005350 secondsstyle
75th percentile : 0.010492 seconds
100th percentile : 1.805923 seconds
Total Time : 13.799954 seconds

Rerp Parser :
25th percentile : 0.001288 seconds
50th percentile : 0.005222 seconds
75th percentile : 0.014640 seconds
100th percentile : 52.706880 seconds
Total Time : 76.460496 seconds

Reppy Parser :
25th percentile : 0.000210 seconds
50th percentile : 0.000492 seconds
75th percentile : 0.000997 seconds
100th percentile : 0.129440 seconds
Total Time: 1.405558 seconds

Removing an hack used in Protego due to lack of an option to ignore characters in `urllib.parse.unquote`. Added few new features to Protego as well.
Protego has been moved to Scrapy organisation.

What is coming up next?

Configuring Travis to push to PyPI automatically.
Introducing a new `ROBOTSTXT_USER_AGENT` setting in Scrapy.
Making `SitemapCrawler` use the new interface.

Did you get stuck anywhere?

I got blocked by StackExchange for few hours. I think they don't like crawlers on their websites. "It is a temporary automatic ban that is put up by our HAProxy instance when you hit our rate limiter." they answered to one of the questions on their website.

Weekly Check-in #11: ( 2 Aug - 8 Aug )

anubhavp28@gmail.com (anubhavp) — Tue, 06 Aug 2019 13:08:03 +0000

What did you do this week?

Added API description and more usage examples to readme.
Added PyPy test environment.
Opened a pull request to add Protego integration to Scrapy.
Modified Protego to treat non-terminal dollar signs as ordinary characters.
Minor aesthetic changes.

What is coming up next?

Transferring the Protego repository to Scrapy organisation on GitHub. It seems that write permissions are necessary for initiating the transfer process.
Would modify Protego to treat wildcards such as `*` and `$` as ordinary characters as well.
Would modify `SitemapCrawler` to use the new interface.
Implementing support for `host` & `crawl-delay` directives in Scrapy.
Some performance improvements might be possible by using a custom pattern matching logic (in place of regex), but I am not sure. I will need to test it.

Did you get stuck anywhere?

Faced problems setting up PyPy test environment. With help from mentors, I was able to solve the issue.

Weekly Check-in #10 : ( 26 July - 1 Aug )

anubhavp28@gmail.com (anubhavp) — Tue, 30 Jul 2019 13:10:59 +0000

What did you do this week?

Improved performance of Protego by implementing lazy regex compilation.
Benchmark Results :
- Time took to parse 570 `robots.txt` files :

Protego :
25th percentile : 0.000134
50th percentile : 0.000340
75th percentile : 0.000911
100th percentile : 0.345727
Total Time : 0.999360

Rerp :
25th percentile : 0.000066
50th percentile : 0.000123
75th percentile : 0.000279
100th percentile : 0.101409
Total Time : 0.317715

Reppy :
25th percentile : 0.000028
50th percentile : 0.000038
75th percentile : 0.000063
100th percentile : 0.015579
Total Time : 0.055850

Time took to parse 570 `robots.txt` and answer 1000 queries per `robots.txt` :

Protego :
25th percentile : 0.009057
50th percentile : 0.012806
75th percentile : 0.023660
100th percentile : 9.033481
Total Time : 21.999680

Rerp :
25th percentile : 0.006096
50th percentile : 0.011864
75th percentile : 0.041876
100th percentile : 35.027233
Total Time : 68.811635

Reppy :
25th percentile : 0.000858
50th percentile : 0.001018
75th percentile : 0.001472
100th percentile : 0.236081
Total Time : 1.132098

What is coming up next?

Will depend on the review from the mentors. If everything looks good to them, I would shift my focus back on Scrapy.

Did you get stuck anywhere?

Nothing major.

[Blog #4] Need For Speed

anubhavp28@gmail.com (anubhavp) — Wed, 24 Jul 2019 18:59:44 +0000

The most of week 7 was spent making Protego compatible with Google's parser. I also worked on the documentation, since Protego codebase is small enough, proper comments and a good readme was sufficient. I uploaded Protego to PyPI - `pip install Protego` that's all it takes to install Protego.

Week 8 was quite interesting. For Protego to become default in Scrapy, it is necessary that it doesn’t throw any kind of error while parsing `robots.txt` files. To make sure that, I decided to download `robots.txt` from top 10,000 websites. I added tests to see if Protego throws any exceptions while parsing the downloaded `robots.txt`. I benchmarked Protego, and the results were quite disappointing. You can see the result here.

We decided to spend the next week improving performance of Protego. I am going to try profiling and heuristics, and see if the performance can be improved.

Weekly Check-in #9 : ( 19 July - 26 July )

anubhavp28@gmail.com (anubhavp) — Tue, 23 Jul 2019 13:10:46 +0000

What did you do this week?

I added tests to make sure Protego doesn't throw exceptions on `robots.txt` of top 10,000 most popular websites.
Utilised Scrapy to create a tool to download `robots.txt` of top 10,000 most popular websites.
Benchmarked Protego : I ran Protego(written in Python), Robotexclusionrulesparser(written in Python), Reppy(written in C++ but has Python interface) on 570 `robots.txt` downloaded from top 1000 websites, and here are the results.
- Time spend parsing the `robots.txt`
  - Protego : 79.00128873897484 seconds
  - Robotexclusionrulesparser : 0.30100024401326664 seconds
  - Reppy : 0.05821833698428236 seconds
- Time spend answering queries (1000 queries (all were identical) per `robots.txt`)
  - Protego : 14.736387926022871 seconds
  - Robotexclusionrulesparser : 67.33521732398367 seconds
  - Reppy : 1.0866852040198864 seconds
Added logging to Protego.

What is coming up next?

Will depend on the review from the mentors. If everything looks good to them, I would shift my focus back on Scrapy.
Make `SitemapSpider` use the new interface for `robots.txt` parsers.
Implement Crawl-delay & Host directive support in Scrapy.

Did you get stuck anywhere?

Nothing major.

[Blog #3] Google open-sourced its robots.txt parser

anubhavp28@gmail.com (anubhavp) — Mon, 22 Jul 2019 11:46:30 +0000

Hey! This is my fourth blog post for GSoC 2019, covering week 5 and 6.

Few interesting things have happened, Google has open-sourced its robots.txt parser, and have also taken the lead role in pushing the community and enterprises to create an official specification for `robots.txt`. I spend a good amount of time making Protego compatible with Google’s parser. This required rewriting a good chunk of Protego to support Google’s parser specific things such as merging record group, supporting misspellings, etc.

I am scared of reading or writing C++ code that uses STL or pointers heavily. So really going through the source code of Google’s parser was kind of uncomforting, but I was able to understand a good chunk of it, after a few days of struggle.

Next up, I will work on making Protego 100% compatible with Google’s parser. I will have to document Protego. I will collect robots.txt from top 1000 websites to understand usage patterns.

[Blog #2] Protego parse!

anubhavp28@gmail.com (anubhavp) — Sun, 21 Jul 2019 17:57:00 +0000

Hey! This is my third blog post for GSoC 2019, covering week 3 and 4.

The first part of my project concerning interface for `robots.txt` parsers is almost complete.

I have started working on a pure-Python `robots.txt` parser which I have named Protego. The name is borrowed from Harry Potter universe, where Protego is a charm that creates a shield to protect the caster. The end goal for Protego is to support all of the popular directives, wildcard matching, and a good number of less popular directives. Also, we aim to make Protego 100% compatible with Google's robots.txt parser. We intend Protego to become the deafult `robots.txt` parser in Scrapy.

I have implemented support for all major directives in Protego. I have also implemented support for wildcard matching. I utilised pytest and tox to automate testing Protego on every version of Python. Furthur used Travis to run tests automatically on code push and pull requests. I borrowed tests from other parsers to check Protego on. Protego currently passes all tests borrowed from `reppy`, `rep-cpp` and `robotexlusionrulesparser`.

[Blog #1 - Week 3] We are going faster than I predicted

anubhavp28@gmail.com (anubhavp) — Sun, 21 Jul 2019 09:38:36 +0000

Hey, welcome back! It has been about 3 weeks since the beginning of the coding period, and I am making progress at a pace faster than I thought we could. Here are the details of the work I did in the first three weeks.

My week prior to the beginning of the coding period was spent setting up a development environment on my computer, and learning about the internals of Scrapy. Since, Scrapy is python project, setting up an development environment is fairly straightforward. Clone the Scrapy repository `git clone https://github.com/scrapy/scrapy`, and create a virtual environment using `python3 -m venv venv` (it creates a Python 3 virtual environment on Ubuntu). Now, activate the environment using `source venv/bin/activate`. Install python dependencies by running `pip install -r requirements-py3.txt`. If you are planning to run tests locally, install test dependencies by running `pip install -r tests/requirements-py3.txt`. The setup is complete.

Scrapy uses Tox for testing. `tox` is virtual environment management tool that can be used to run tests using multiple versions of python. `tox` also reduces the work required for integrating CIs. To run tests using Python 2.7 and Python 3.7, use `tox -e py27 -e py37`. I had minor difficulty (I was unable to run `tox` using any version other than 2.7) as I was using `tox` for the first time, but I was easily able to solve it using online documentation and help from mentors. You can learn more about `tox` here.

First and second weeks were spend deciding on a interface specification, and implementing the interface on three existing `robots.txt` parsers. Using feedback from the mentors, I improved upon the specification I proposed in my GSoC proposal. Documenting the interface turned out to be harder than implementing it. Both Sphinx and `reStructuredText` was new for me. I also found it difficult to occasionally describe things clearly (documentation is hard). While implementing the interface on top of `RobotFileParser`, I had to take deep dive into its source code. For some reason, I always had this belief that reading through the implementation of python (or any language) and its inbuilt modules, would be difficult and not really useful, and code would mostly be complex and cryptic. This doesn’t seem to be the case (at least with python). I should do more of this, looking at a module’s implementation for fun :smile:. I modified Scrapy to use the new interface instead of directly calling `RobotFileParser`.

Next, I will work on creating extra test environments in Travis for testing third-party parser integration. I haven't work with Travis or `tox` before. Also, I will document the functionalities provided by these parser in Scrapy documentation. I will also need to the make the code for testing integration more maintanble.

Weekly Check-in #8 : (12 July - 18 July)

anubhavp28@gmail.com (anubhavp) — Thu, 18 Jul 2019 17:09:00 +0000

What did you do this week?

Protego is now compatible with Google's parser. Protego pass all testcase in Google's parser testsuite.
Documented Protego - functions now have docstrings, and I have created a readme file.
Protego is now on PyPI. To install Protego, simply type `pip install protego`.
Made few changes to improve performance as suggested by mentors.

What is coming up next?

Additions to CI - making sure Protego doesn't throw exceptions on robots.txt of top 1000 alexa websites.
Benchmark Protego.

Did you get stuck anywhere?

Nothing major.

Weekly Check-in #7: (5 July - 11 July)

anubhavp28@gmail.com (anubhavp) — Tue, 09 Jul 2019 11:57:10 +0000

Hey! here is an update on what I have achieved so far.

What did you do this week?

Protego now passes all tests borrowed from reppy, rep-cpp and robotexclusionrulesparser.
Made few changes to Protego to make it compatible with google's parser.
Worked on changes suggest on the interface pull request.
Wrote code to fetch robots.txt files from top 1000 websites, and generate statistics we need. ( link )
Looked at the code of Google's robots.txt parser for the purpose of creating a python interface on top of it. I might need to modify its code as currently it parses the robots.txt file for answering every query. (Working on anything in C++ that uses pointers or STL heavily makes me feel uncomfortable).

What is coming up next?

Modify protego to make it behave similar to Google's parser (will need to add few more features like record group merging), and add more tests.
Document Protego.
Benchmarking Protego's performance.
I would need to read how to call C/C++ code from python, for creating an interface on top Google's parser. I am currently thinking of using Cython.
Would work on blog posts (planning to write 3 blog posts within this week).

Did you get stuck anywhere?

No, I got to work with some data science tools like jupyter notebook & pandas.

Weekly Check-in #6: (28 Jun - 4 July)

anubhavp28@gmail.com (anubhavp) — Tue, 02 Jul 2019 11:40:42 +0000

Hello! It has been more than a month since the beginning of GSoC coding period, and I am completely satisfied with my progress. I am glad that I choose to work remotely this summer, since it has been raining heavily for last 5-6 days in Mumbai (place where I live).

What did you do this week?

Implemented support for wildcard matching and `request-rate` directive.
Increased the number test cases, some of them were borrowed from `rep-cpp`, another robots.txt parser. The new test cases now ensure that the parser behaviour conforms to the google specification for robots.txt.

PS: I really overestimated the amount of work I would be able to do in a week while writing the last check-in - Hofstadter's law.

What is coming up next?

Google open-sourced their robots.txt parser - here. Since Google is the most popular search engine in the world, it is likely that for most websites, the largest percentage of crawling requests they receive originates from Google. Hence, there out be more robots.txt written for Google's crawler than any other crawler. It would make sense to make Protego behave in a similar way to Google's parser.
Documenting Protego.
Increasing the number of test cases (we can borrow some from Google's parser).
Benchmarking Protego's performance against other parsers.
Regarding the collecting statistics on robots.txt usage, I am not completely sure if it would be a good idea to invest time into it, now that I have found a blog, which describes the popularity of individual directives, and could help us prioritise which ones to implement.

Did you get stuck anywhere?

Nothing major. I found it simpler than I had initially expected.

Weekly Check-in #5 : ( 21 Jun - 27 Jun )

anubhavp28@gmail.com (anubhavp) — Tue, 25 Jun 2019 09:51:27 +0000

Hello! The fourth week of GSoC coding period is coming to an end. Here is an update on what I achieved in the past week and what I am looking forward to.

What did you do this week?

Implemented minor changes suggested by scrapy maintainers.
Started working on a new pure python robots.txt parser (which lives here https://github.com/anubhavp28/protego currently). It will eventually be moved to Scrapy organisation.
Implemented support for standard robots.txt directives in the new parser.
Integrated the code with pytest and tox for testing.
Integrated the repo with Travis CI to trigger tests automatically on pull requests.

What is coming up next?

Implement support for modern conventions like wildcard matching, clear-param etc.
Add a lot of tests (mostly borrowed from existing parsers).
Performance benchmarking of the new parser (against existing parsers).
Collecting statistics related to use of robots.txt. On suggestion of a mentor, I am planing to use robots.txt files of Top 1000 websites in alexa rankings, and collect stats such as how many of them use robots.txt, how many records on average a record group contain, how many times a certain directives is mentioned, etc. This could help use make better choices for improving performance - such as whether to use a trie for prefix matching, etc.

Did you get stuck anywhere?

Oh, actually naming the parser was the hardest part . I am still not satisfied with the name. I just ran out of ideas.

Weekly Check-in #4: (14 Jun - 20 Jun )

anubhavp28@gmail.com (anubhavp) — Tue, 18 Jun 2019 11:33:04 +0000

Hello! The third week of GSoC coding period is coming to an end. Here is an update on what I achieved in the past week and what I am looking forward to.

What did you do this week?

Created separate tox testing environments for testing integration with third-party parsers like Robotexclusionrulesparser and Reppy.
Made Travis use the new tox environments.
Described these parsers in Scrapy documentation.
Got Robotexclusionrulesparser to work with unicode user agents.

What is coming up next?

I will be working on creating a python based robots.txt parser which compliant with spec and supports modern conventions.

Did you get stuck anywhere?

Nothing major.

Weekly Check-in #3 : ( 7 Jun - 13 Jun )

anubhavp28@gmail.com (anubhavp) — Sat, 08 Jun 2019 13:21:16 +0000

Hello, wandering pythonistas! The second week of GSoC coding period is coming to an end. Here is an update on what I achieved in the past week and what I am looking forward to.

What did you do this week?

I made few changes to the interface according to the feedback received from the mentors.
I implemented the interface on top of third party parsers like Robotexclusionrulesparser and Reppy.
Wrote tests for testing the implementation of interface on top of the two parsers. The tricky part was reducing duplication of code and keeping the test maintainable.
Modified Scrapy to use the new interface (instead of directly calling Python’s inbuilt RobotFileParser).
I had the weekly meeting with my mentors, where we discussed new stretch goals for the project.

What is coming up next?

It will depend on the feedback of the mentors. If everything seems good to them, I will focus my attention on writing a pure python robots.txt parser.

Did you get stuck anywhere?

Nothing major, though I had little difficulty due my lack of knowledge of difference between Python 2 and Python 3. I knew Python 3 uses unicode string by default, what I didn’t know is that in Python 3 `bytes` and `str` type are different. Hence, encoding a string produces an object of type `bytes`. This actually makes sense, having different types for string and arbitrary binary data.

[Blog #0] Accepted for GSoC 2019 : 3 Months of Open Source Ahead

anubhavp28@gmail.com (anubhavp) — Fri, 07 Jun 2019 14:20:15 +0000

---

Hello! I have been selected as a student for Google Summer of Code 2019. For those of you who are unaware, Google Summer of Code is a global program focused on bringing more student developers into open source software development. Students developers get an opportunity to work with an open source organization on a 3 month programming project.

I am working on Scrapy - an open-source scraping and web crawling framework. My task is to implement an interface for `robots.txt` parsers in Scrapy. The stretch goal of the project is to write a fully spec compliant pure python `robots.txt` parser.

This blog is part of a series of blog posts where I will go in depth to describe my work. Since, this is the first blog post, I have dedicated it to explain about my project. My project has a lot to do with `robots.txt` and Robot Exclusion Standard. Let’s take a look at what these things are.

Web crawlers (also called spiders) are bots that systematically browse the internet, generally for the purpose of extracting data (called scraping) or indexing. Search engines (eg. google) use web crawlers to index pages on the internet, this index is then used to serve users with relevant results.

When web crawlers visit a website, they too consume resources of the web servers, similar to a normal user. A crawler can make multiple requests per second. Therefore, unrestricted crawling can lower the performance of a website, and cause annoyance to its users. A solution to this problem is robots exclusion standard, which makes it possible to specify which parts of the website a crawler should not access, and how frequently a crawler should make requests for a resource on the server. The standard allows websites to specify how “polite” a crawler should be.

Under robots exclusion standard, instructions for crawlers are specified in a file named `robots.txt` placed at the root of the website. These instructions have to be specified in a certain format.

`robots.txt` standard is not owned by any standard body, and is not in active development since 1997. So, the standard has not been revised for more than two decades which, I will argue, had led it to become outdated. Meanwhile, search giants like google, bing, ~~yahoo!~~ have collaborated to create an informal extension of the standard which their crawlers adhere to. Since usually the majority of crawling requests originates from search engines, most web administrators write `robots.txt` adhering to the informal extension of the specification.

Scrapy uses RobotFileParser (Python’s inbuilt `robots.txt` parser), which strictly follows the old specification, and has not been updated. Hence, Scrapy is likely to misinterpret the crawling rules for majority of websites on the internet. There was an effort to switch to another parser, but since there isn’t a fully compliant pure python parser available, and it is difficult to package non-python code with Scrapy because its wide user base consisting of people using a wide variety of platforms and implementations of python. Including non-python code would need dropping the support of pypy, and would require users to install compilers for other languages (which may not be easy on every platform).

The short term solution to overcome this could be to allow users to switch to a different parser if they wish too, and keep `RobotFileParser` as the default in Scrapy. It has additional benefit of giving more power to the users. For this, we are planning to create an interface for `robots.txt` parsers in Scrapy, and implementing this interface on top few popular parsers. This the first goal of my project. In the end, we would like to have a fully spec compliant pure python parser which Scrapy could use by default. This is the stretch goal of my project.

Hope, to have an incredible 3 months ahead :smile:. If you need any help regarding Google Summer of Code or you just want to learn more about my work, feel free email me at anubhavp28@gmail.com.

Weekly Check-in #2 [31 May - 6 Jun]

anubhavp28@gmail.com (anubhavp) — Sun, 02 Jun 2019 02:29:44 +0000

Hello everyone. The first week of GSoC coding period is coming to an end. Here is an update on what I achieved in the past week and what I am looking forward to.

What did you do this week?

I submitted my first pull request related to GSoC. This week mostly involved discussion on interface specification. I learned that designing an interface involves considering several small but quite important details, and a good practice is to question every choice you make. Also, I had a meeting with my mentors where we discussed about weekly milestones and decided to have weekly meetings, every Tuesdays. I implemented the interface on top of Python’s in-built robots.txt parser and worked on documentation related to the interface.

I got an opportunity to deep dive into the source code of Python’s in-built robots.txt parser. For some reason, I always had this belief that reading through the implementation of python (or any language) and its inbuilt modules, would be difficult and not really useful, and code would mostly be complex and cryptic (to a beginner like me). This doesn’t seem to be the case (at least with python). I should do more of this, looking at a module’s implementation for fun •ᴗ• .

What is coming up next?

In the next week, I am looking to finalize the interface, and modify Scrapy to use the interface to communicate with the parsers. I would also work on documenting the interface, and if time permits will implement the interface on top of few other parsers.

Did you get stuck anywhere?

Nope. I learned a lot from constant feedback from my mentors. It was an awesome week •ᴗ•

Weekly Check-in #1 [24 May - 30 May]

anubhavp28@gmail.com (anubhavp) — Thu, 23 May 2019 09:02:57 +0000

Hey everyone. I am Anubhav, and this summer I am working to implement an interface for robots.txt parsers in scrapy. This is the first of many upcoming weekly blog posts where I will describe in brief the work I have done in the previous week and my plans for the upcoming week. So, let's get started.

What did you do this week?

Most of time was spent on configuring a local development environment, and learning to use tox and how to run tests locally. For the patches I have submitted before, I didn't run tests locally beforehand, and relied solely on CI to do it. Running tests locally could have saved a lot of time. Also, I went through scrapy contribution guide, learned about twisted (scrapy uses it heavily) and PEP8, and worked on a pull request I had opened before.

What is comping up next?

I will have my first meeting with mentors of the project.
I will work on few pull requests I had opened before.
Maybe, since this is the last week of community bonding period, looking to have discussion with the mentors regarding interface specification.

Did you get stuck anywhere?

I had minor difficulties understanding how to run tests using tox. When I followed the instructions given in scrapy documentation to run tests, I could only run tests using a Python 2.7 environment. Thankfully, tox has an incredible documentation that allowed to me understand settings inside of tox.ini config file. In the end, I just had to make few edits to my tox.ini file, and I was able to run tests using a Python 3 environment as well.