Articles on tommyli3318's Blog

Google Summer of Code with Nuitka 7th Weekly Check-in

taofengl@uci.edu (tommyli3318) — Mon, 26 Aug 2019 08:24:13 +0000

1. What did you do this week?

As GSoC is wrapping up, I wrote a summary of my work which can be found here.

In addition, I have finalized my pull requests #495 and #484 which are now ready for merge.

2. What is coming up next?

After GSoC ends, I plan on continuing my contributions to Nuitka and PSF.

3. Did you get stuck anywhere?

I do not remember being stuck anywhere this week :)

Google Summer of Code with Nuitka 6th Blog Post

taofengl@uci.edu (tommyli3318) — Mon, 19 Aug 2019 21:53:49 +0000

This week, I focused mostly on bug fixes and improving my PyPI-pytest automation script while also working on adding more standalone tests.

I created a pull request https://github.com/Nuitka/Nuitka/pull/483 to add support for py_modules compilation support in Nuitka. The pull request turned out very well, adding support for not only py_modules compilation but also the combination of package and module. In addition, a new distutil example was also added to showcase the compilation of package and module
In the process of the above pull request, I discovered a regression in Nuitka and https://github.com/Nuitka/Nuitka/pull/487 was created as a hotfix.
In addition, Issue https://github.com/Nuitka/Nuitka/issues/479 is fixed by #483 and is now closed
I continued to improve my PyPI-pytest automated testing script, including functional improvements, minor speed improvements, readability, and documentation improvements. My work could be found at https://github.com/Nuitka/Nuitka/pull/495.
More standalone tests were also added to my pull request at https://github.com/Nuitka/Nuitka/pull/484. This is a side goal and is less important

I am very proud of the work I have done this week. The goal for this final week is to finish up pull request #495, and I plan on finishing #484 after GSoC ends. I also discussed with my mentor and plan on working for Nuitka long term, even after GSoC :).

Google Summer of Code with Nuitka 6th Weekly Check-in

taofengl@uci.edu (tommyli3318) — Mon, 12 Aug 2019 22:01:32 +0000

1. What did you do this week?

This week, I finished my script which automates the testing of nuitka-wheel pytest. Details can be found on my pull request: https://github.com/Nuitka/Nuitka/pull/440

The script automates the manual testing of comparing pytest results of a nuitka compiled wheel using `python setup.py bdist_nuitka` to the pytest results of an uncompiled wheel built using `python setup.py bdist_wheel` for the most popular PyPI packages. Testing is done to ensure that nuitka is building the wheel correctly. If the pytests pass/fail in the same way, that means Nuitka built the wheel properly. Else if the tests differ, then something is wrong. Virtualenv is used to create a clean environment with no outside pollution.

Many improvements have been made this week, including the addition of more PyPI packages. The pull request was merged after some final touches.

In addition, standalone tests are now being created for the PyPI packages. Details can be found in my pull request: https://github.com/Nuitka/Nuitka/pull/484

2. What is coming up next?
Add more packages to standalone tests. Add support for py_modules only compilation with Nuitka. Work on documentation (README.rst) for the automated testing script.

3. Did you get stuck anywhere?
I ran into some issues with the Jinja2 package. I’ve skipped over it for now for the sake of time but will hopefully get back to it in the future.

Google Summer of Code with Nuitka 5th Blog Post

taofengl@uci.edu (tommyli3318) — Tue, 06 Aug 2019 04:06:17 +0000

This week, I continued to work on my script which automates the testing of nuitka-wheel pytest. Details can be found on my pull request: https://github.com/Nuitka/Nuitka/pull/440

A lot was done this week, including:

Fixing automation for various pypi packages
Figuring out failures for many packages
Created a new issue for package pyyaml - https://github.com/Nuitka/Nuitka/issues/476
Created a new issue for packages decorator, ipaddress, and pyparsing - https://github.com/Nuitka/Nuitka/issues/479
Created a new issue for packages pycparser and numpy - https://github.com/Nuitka/Nuitka/issues/477
Created a py_modules only example of `distutils` (can be seen in my main PR)
Summary improvement and run-time improvements
Fixed pylint errors

Very proud of the work I have done this week. The pull request is pretty much ready to be merged at this point. The plan for next week is to find bugs and probably work on documentation for the test suite.

Google Summer of Code with Nuitka 5th Weekly Check-in

taofengl@uci.edu (tommyli3318) — Tue, 30 Jul 2019 20:47:43 +0000

1. What did you do this week?
This week, I continued to work on my script which automates the testing of nuitka-wheel pytest. Details can be found on my pull request: https://github.com/Nuitka/Nuitka/pull/440

Testing has been improved and extended to many more PyPI packages.

2. What is coming up next?
Perfect the script in preparation for merging and work on documentations.

3. Did you get stuck anywhere?
Some PyPI packages are special and require special handling during automation. I skipped these for the sake of speed, but I will need to get back to them in the future.

Google Summer of Code with Nuitka 4th Blog Post

taofengl@uci.edu (tommyli3318) — Mon, 22 Jul 2019 21:27:28 +0000

This week, I continued to work on my script which automates the testing of nuitka-wheel pytest. Details can be found on my pull request: https://github.com/Nuitka/Nuitka/pull/440

There were a lot of changes made this week.

Added an `__import__` check to make sure pytest used the correct (uncompiled or compiled) package
Added a mechanism to ignore certain tests which are unimportant (fixed the comparison issue for urllib3)
Added coloring to output for better visuals
Added a summary of all packages at the very end of testing
Extended testing to many more packages
Added local caching of `git clone`
Many run time improvements

Very proud of the work I have done this week. The plan for next week is to extend the automation to more PyPI packages.

Google Summer of Code with Nuitka 4th Weekly Check-in

taofengl@uci.edu (tommyli3318) — Tue, 16 Jul 2019 05:01:29 +0000

1. What did you do this week?
This week, I worked on my script which automates the testing of nuitka-wheel pytest. Details can be found on my pull request: https://github.com/Nuitka/Nuitka/pull/440
Automated testing compares the pytest results of a nuitka wheel built using python setup.py bdist_nuitka to the uncompiled bdist_wheel pytest of each PyPI package. Testing is done to ensure that nuitka is building the wheel correctly. If the pytests pass/fail in the same way, that means Nuitka built the wheel properly. Else if the tests differ, then something is wrong. Virtualenv is used to create a clean environment with no outside pollution.
I used subprocess to capture the output from terminal for comparison. I also implemented mechanics to ignore the time differences found in pytests. Furthermore, the testing has been extended to three more packages.

2. What is coming up next?
Perfect the automation and extend the testing to even more PyPI packages. There are also bugs regarding PermissionError on the deletion of the .git folder that need to be fixed. Furthermore, there are inconsistencies between different packages which require special handling.

3. Did you get stuck anywhere?
Yes, output comparison took account of the time differences from pytests when that should have been ignored. I came up with a solution using regular expressions to find and standardize the time differences.

Google Summer of Code with Nuitka 3rd Blog Post

taofengl@uci.edu (tommyli3318) — Mon, 08 Jul 2019 23:07:23 +0000

This week, I created a script to automate the testing of nuitka-wheel pytest. Details can be found on my pull request: https://github.com/Nuitka/Nuitka/pull/440
Automated testing compares the pytest results of a nuitka wheel built using python setup.py bdist_nuitka to the regular python wheel pytest of each PyPI package. Testing is done to ensure that nuitka is building the wheel correctly. If the pytests pass/fail in the same way, that means Nuitka built the wheel properly. Else if the tests differ, then something is wrong. Virtualenv is used to create a clean environment with no outside pollution.

In addition, a post documenting manual test results of dateutil was made to nuitka.net with a pull request: https://github.com/Nuitka/Nuitka-website/pull/11

I ran into many bugs this week but I was proud of myself for being able to overcome all of them with the help of mentors. In particular, getting stdout and stderr from the virtualenv was not easy. There are also some PermissionErrors that I will still need to look more into. Nevertheless, automation now fully works for the urllib3 and dateutil packages and I am very happy with this week’s progress.

The plan for next week is to extend the automation to more PyPI packages.

Google Summer of Code with Nuitka 3rd Weekly Check-in

taofengl@uci.edu (tommyli3318) — Tue, 02 Jul 2019 00:16:02 +0000

1. What did you do this week?
Created a script to automate the testing of nuitka-wheel pytest. Details can be found on my pull request: https://github.com/Nuitka/Nuitka/pull/440
Automated testing compares the pytest results of a nuitka wheel built using python setup.py bdist_nuitka to the regular pytest of each PyPI package. Testing is done to ensure that nuitka is building the wheel correctly. If the pytests pass/fail in the same way, that means Nuitka built the wheel properly. Else if the tests differ, then something is wrong. Virtualenv is used to create a clean environment with no outside pollution.

2. What is coming up next?
Keep working to fully automate nuitka-wheel testing and fix current errors for urllib3. Then, extend the testing to other PyPI packages.

3. Did you get stuck anywhere?
Yes, testing creates two virtualenv directories which needs to be removed after testing. However, I ran into PermissionError on Windows because you cannot delete a binary that has been executed. I will need to look more into that to fix it. Furthermore, I ran into problems trying to get the virualenv to run correctly, but that has been fixed.

Google Summer of Code with Nuitka 2nd Blog Post

taofengl@uci.edu (tommyli3318) — Mon, 01 Jul 2019 23:22:47 +0000

This week, I created nuitka standalone tests for python-dateutil and pyyaml packages. Details can be found here: https://github.com/Nuitka/Nuitka/pull/418

In addition, I performed manual virtualenv pytest testing for the urllib3 package wheel built using nuitka. A pull request was made to document my work on the official nuitka.net website: https://github.com/Nuitka/Nuitka-website/pull/10. The pull request was then merged, the full posting can be found here: https://nuitka.net/posts/urllib3-wheel-with-nuitka.html#

In summary, the tests passed with urllib3 and the results of uncompiled pytest was very similar to the nuitka-compiled pytest. However, there were a few minor differences. My aim is to make this process automatic to avoid having to manually perform testing for every PyPI package.

The plan for next week is to start automating the wheel nuitka tests with the top PyPI packages.

Google Summer of Code with Nuitka 2nd Weekly Check-in

taofengl@uci.edu (tommyli3318) — Mon, 17 Jun 2019 23:02:18 +0000

1. What did you do this week?
Created nuitka standalone tests for python-dateutil and pyyaml packages. Details can be found here: https://github.com/Nuitka/Nuitka/pull/418

Performed manual virtualenv pytest testing for the urllib3 package wheel built using nuitka. Details can be found here: https://github.com/Nuitka/Nuitka-website/pull/10

2. What is coming up next?
Finish up urllib3 wheel test report, then move on to manual virtualenv wheel tests for python-dateutil and pyyaml.

3. Did you get stuck anywhere?
Yes, the nuitka-built urllib3 wheel originally had import problems that was crashing pytest execution. The C code of nuitka was changed to fix the bug.

Google Summer of Code with Nuitka 1st Blog Post

taofengl@uci.edu (tommyli3318) — Mon, 10 Jun 2019 01:15:18 +0000

This week, I created nuitka standalone tests for boto3 and made changes to urllib3. Details can be seen in my PR: https://github.com/Nuitka/Nuitka/pull/339

I also created standalone tests for the python-dateutil and pyyaml packages. My pull request for that can be seen here: https://github.com/Nuitka/Nuitka/pull/418

In addition, I tested urllib3 bdist_nuitka compatibility with Nuitka by building the urllib3 dist wheel via Nuitka inside a virtual environment and then running urllib3’s pytests with the compiled version. It ended up raising an error which I reported here: https://github.com/Nuitka/Nuitka/issues/413

The plan for next week is to look more into this issue and also work on testing the other packages, python-dateutil and pyyaml are next on the list.

Google Summer of Code with Nuitka 1st Weekly Check-in

taofengl@uci.edu (tommyli3318) — Tue, 04 Jun 2019 23:44:40 +0000

1. What did you do this week?
Created nuitka standalone tests for boto3 and made changes to urllib3. Details can be seen in my PR: https://github.com/Nuitka/Nuitka/pull/339

2. What is coming up next?
Compatibility test for Nuitka distutils with urllib3, then move on to the next package, which will be python-dateutil.

3. Did you get stuck anywhere?
Yes, the boto3 standalone test was not passing with nuitka compilation due to unknown errors. My guess was that it has to do with the AWS mocking package moto. If this is too complicated to figure out, we will probably skip this test.

Google Summer of Code with Nuitka

taofengl@uci.edu (tommyli3318) — Tue, 28 May 2019 22:38:40 +0000

Hi all! :) My name is Taofeng Li and I go by the nickname of Tommy, github handle tommyli3318. I'm a Computer Science major at the University of California, Irvine in the United States. I am very excited to announce that I'll be contributing to Nuitka this summer under the Google Summer of Code program! My project is "Nuitka support for PyPI top 50", it ensures Nuitka’s compatibility with the 50 most popular PyPI packages by setting up automated testing for each package. These automated tests will serve to be very important tools for the development of Nuitka, as they will be used to assure Nuitka’s compatibility with the most used Python packages every time Nuitka receives a new update. My mentors will be Vaibhav Tulsyan, Kamran Ghanaat, and Kay Hayen. My project abstract can be seen at https://summerofcode.withgoogle.com/projects/#4874692681269248 and the full details of my project can be found at https://docs.google.com/document/d/1tdbXyMy5xjVEeWociI_t2f6tlmge6czT0xmfCLyDXYg/edit?usp=sharing This week, I am mainly working on the first standalone tests for urllib3 while trying to get the test for boto3 to work. I have been struggling for a while trying to get boto3 tests, getting a weird build error when compiled with nuitka. I plan on using custom plugins to see if that will fix the errors. Furthermore, if the originally proposed 50 tests of my project seems to be too much to do, my mentor says I might have to lower that number. I am greatly looking forward to the many things that I'll learn this summer!