Articles on McSinyx's Blog

Outro

vn.mcsinyx@gmail.com (McSinyx) — Mon, 31 Aug 2020 14:48:23 +0000

Note: This article's HTML source is exported from reST. Without necessary CSS, some part might look hideous. Please consider viewing on my personal blog.

The Look

At the time of writing, implementation-wise parallel download is ready:

Does this mean I’ve finished everything just-in-time? This sounds to good to be true! And how does it perform? Welp…

The Benchmark

Here comes the bad news: under a decent connection to the package index, using fast-deps does not make pip faster. For best comparison, I will time pip download on the following cases:

Average Distribution

For convenience purposes, let’s refer to the commands to be used as follows

legacy-resolver: pip --no-cache-dir download {requirement}
2020-resolver: pip --use-feature=2020-resolver --no-cache-dir download {requirement}
fast-deps: pip --use-feature=2020-resolver --use-feature=fast-deps --no-cache-dir download {requirement}

In the first test, I used axuy and obtained the following results

legacy-resolver	2020-resolver	fast-deps
7.709s	7.888s	10.993s
7.068s	7.127s	11.103s
8.556s	6.972s	10.496s

Funny enough, running pip download with fast-deps in a directory with downloaded files already took around 7-8 seconds. This is because to lazily download a wheel, pip has to make many requests which are apparently more expensive than actual data transmission on my network.

Note

With unstable connection to PyPI (for some reason I am not confident enough to state), this is what I got

2020-resolver	fast-deps
1m16.134s	0m54.894s
1m0.384s	0m40.753s
0m50.102s	0m41.988s

As the connection was unstable and that the majority of pip networking is performed as CI/CD with large and stable bandwidth, I am unsure what this result is supposed to tell (-;

Large Distribution

In this test, I used TensorFlow as the requirement and obtained the following figures:

legacy-resolver	2020-resolver	fast-deps
0m52.135s	0m58.809s	1m5.649s
0m50.641s	1m14.896s	1m28.168s
0m49.691s	1m5.633s	1m22.131s

Distribution with Conflicting Dependencies

Some requirement that will trigger a decent amount of backtracking by the current implementation of the new resolver oslo-utils==1.4.0:

2020-resolver	fast-deps
14.497s	24.010s
17.680s	28.884s
16.541s	26.333s

What Now?

I don’t know, to be honest. At this point I’m feeling I’ve failed my own (and that of other stakeholders of pip) expectation and wasted the time and effort of pip’s maintainers reviewing dozens of PRs I’ve made in the last three months.

On the bright side, this has been an opportunity for me to explore the codebase of package manager and discovered various edge cases where the new resolver has yet to cover (e.g. I’ve just noticed that pip download would save to-be-discarded distributions, I’ll file an issue on that soon). Plus I got to know many new and cool people and idea, which make me a more helpful individual to work on Python packaging in the future, I hope.

Final Check-In

vn.mcsinyx@gmail.com (McSinyx) — Mon, 24 Aug 2020 15:14:05 +0000

Hello there!

What did I do last week?

Not much, but seemingly implementation-wise I have finished my GSoC project:

Finish the implementation of wheels' parallel download (GH-8771)
Help make pip's CI green again (GH-8790)
Reformat a few spots in user guide (GH-8795)

Did I get stuck anywhere?

I got sick, but I am recovering now!

What is coming up next?

I will try to spend the time I got left within the scope of GSoC to improve cache usage of the fast-deps feature.

Parallelizing Wheel Downloads

vn.mcsinyx@gmail.com (McSinyx) — Mon, 17 Aug 2020 15:14:09 +0000

And now it's clear as this promise
That we're making
Two progress bars into one

Hello there! It has been raining a lot lately and my wisdom tooth has decided to start growing today, causing me a mild fever. To whoever reading this, I hope it wouldn't happen to you.

Download Parallelization

I've been working on pip's download parallelization for quite a while now. As distribution download in pip was modeled as a lazily evaluated iterable of chunks, parallelizing such procedure is as simple as submitting routines that write files to disk to a worker pool.

Or at least that is what I thought.

Progress Reporting UI

pip is currently using customly defined progress reporting classes, which was not designed to working with multithreading code. Firstly, I want to try using these instead of defining separate UI for multithreaded progresses. As they use system signals for termination, one must the progress bars has to be running the main thread. Or sort of.

Since the progress bars are designed as iterators, I realized that we can call next on them. So quickly, I throw in some queues and locks, and prototyped the first working implementation of progress synchronization.

Performance Issues

Welp, I only said that it works, but I didn't mention the performance, which is terrible. I am pretty sure that the slow down is with the synchronization, since the map_multithread call doesn't seem to trigger anything that may introduce any sort of blocking.

This seems like a lot of fun, and I hope I'll get better tomorrow to continue playing with it!

Sixth Check-In

vn.mcsinyx@gmail.com (McSinyx) — Mon, 10 Aug 2020 16:53:19 +0000

Hello there!

What did I do last week?

It has been a quite fun week for me, given the current state of development and the newly dicovered bugs thanks to pip 20.2 release:

Initiate discussion with the maintainers of pip on isolating networking code for late download in parallel (GH-8697)
Discuss the UI of parallel download (GH-8698)
Log debug information relating lazy wheel decision (GH-8710)
Disable caching for range requests (GH-8716)
Dedent late download logs (GH-8722)
Add a hook for batch downloading (third attempt I think) (GH-8737)
Test hash checking for fast-deps (GH-8743)

Did I get stuck anywhere?

Not exactly, everything is going smoothly and I'm feeling awesome!

What is coming up next?

I'll try to solve GH-8697 and GH-8698 within the next few days. I am optimistic that the parallel download prototype will be done within this week.

Sorting Things Out

vn.mcsinyx@gmail.com (McSinyx) — Mon, 03 Aug 2020 18:58:09 +0000

Hi! I really hope that everyone reading this is still doing okay, and if that isn't the case, I wish you a good day!

`pip` 20.2 Released!

Last Wednesday, pip 20.2 was released, delivering the 2020-resolver as well as many other improvements! I was lucky to be able to get the fast-deps feature to be included as part of the release. A brief description of this experimental feature as well as testing instruction can be found on Python Discuss.

The public exposure of the feature also remind me of some further optimization to make on the lazy wheel. Hopefully without download parallelization it would not be too slow to put off testing by concerned users of pip.

Preparation for Download Parallelization

As of this moment, we already have:

Multithreading pool fallback working
An opt-in to use lazy wheel to optain dependency information, and thus getting a list of wheels at the end of resolution ready to be downloaded together

What's left is only to interject a parallel download somewhere after the dependency resolution step. Still, this struggles me way more than I've ever imagined. I got so stuck that I had to give myself a day off in the middle of the week (and study some Rust), then I came up with something what was agreed upon as difficult to maintain.

Indeed, a large part of this is my fault, for not communicating the design thoroughly with pip's maintainers and not carefully noting stuff down during (verbal) discussions with my mentor. Thankfully Chris Hunt came to the rescue and did a refactoring that will make my future work much easier and cleaner.

Fifth Check-In

vn.mcsinyx@gmail.com (McSinyx) — Mon, 27 Jul 2020 17:15:15 +0000

Hello and I hope y'all are still doing well!

What did I do last week?

I was not really productive last week—most of the following tickets are fillers to make use of the spare cycles I had when I was still trying to figure out the way to implement the main work.

Finalize the --use-feature=fast-deps flag (GH-8588)
Improve mocking of environment variables in the test suit (GH-8614)
Finalize the fix for verbose/quiet options specified via configuration files and environment variables (GH-8578)
Clean up a tiny bit in the resolver internal API (GH-8629)
Start working on seperating the download of wheels from dependency resolution (GH-8638)

Did I get stuck anywhere?

I'm struggling on refactoring the code to support separate download. pip's codebase was not intended for this and thus there are many execution paths and other details entangled around the relevant area.

What is coming up next?

pip 20.2 is going to be released within the next few days with --use-feature=fast-deps included and I'm mentally prepare to fix any undiscovered problem. At the same time, I will continue working on GH-8638 and hopefully get it done soon enough to begin drafting download parallelization strategies, mostly with the UI.

I've Walked 500 Miles…

vn.mcsinyx@gmail.com (McSinyx) — Mon, 20 Jul 2020 18:04:30 +0000

… and I would walk 500 more
Just to be the man who walks a thousand miles
To fall down at your door

The Main Road

Hi, have you met fast-deps? It's (going to be) the name of pip's experimental feature that may improve the speed of dependency resolution of the new resolver. By avoid downloading whole wheels to just obtain metadata, it is especially helpful when pip has to do heavy backtracking to resolve conflicts.

Thanks to Chris Hunt's review on GH-8537, my mentor Pradyun Gedam and I worked out a less hacky approach to inteject the call to lazy wheel during the resolution process. A new PR GH-8588 was filed to implement it—I could have just worked on top of the old PR and rebased, but my git skill is far from gud enough to confidently do it.

Testing this one has been a lot of fun though. At first, integration tests were added as a rerun of the tests for the new resolver, with an additional flag to use feature fast-deps. It indeed made me feel guilty towards Travis, who has to work around 30 minutes more every run. Per Chris Hunt's suggestion, in the new PR, I instead write a few functional tests for the area relating the most to the feature, namely pip's subcommands wheel, download and install.

It was also suggested that a mock server with HTTP range requests support might be better (in term of performance and reliablilty) than for testing. However, I have yet to be able to make Werkzeug do it.

Why did I say I'm half way there? With the parallel utilities merged and a way to quickly get the list of distribution to be downloaded being really close, what left is only to figure out a way to properly download them in parallel. With no distribution to be added during the download progress, the model of this will fit very well with the architecture in my original proposal. A batch downloader can be implemented to track the progress of each download and thus report them cleanly as e.g. progress bar or percentage. This is the part I am second-most excited about of my GSoC project this summer (after the synchronization of downloads written in my proposal, which was then superseded by fast-deps) and I can't wait to do it!

The Side Quests

As usual, I make sure that I complete every side quest I see during the journey:

GH-8568: Declare constants in configuration.py as such
GH-8571: Clean up Configuration.unset_value and nit the class' __init__
GH-8578: Allow verbose/quite level to be specified via config file and env var
GH-8599: Replace tabs by spaces for consistency

Snap Back to Reality

A bit about me, I actually walked 500 meters earlier today to a bank and walked 500 more to another to prepare my Visa card for purchasing the upcoming Pinephone prototype. It's one of the first smartphones to fully support a GNU/Linux distribution, where one can run desktop apps (including proper terminals) as well as traditional services like SSH, HTTP server and IPFS node because why not? Just a few hours ago, I pre-ordered the postmarketOS community edition with additional hardware for convergence.

If you did not come here for a Pinephone ad, please take my apologies though d-; and to ones reading this, I hope you all can become the person who walks a thousand miles to fall down at the door opening to all what you ever wished for!

Fourth Check-In

vn.mcsinyx@gmail.com (McSinyx) — Mon, 13 Jul 2020 10:43:31 +0000

Hello there! I'm having my second year's last exam tomorrow, but it feels like summer already! I've been finalizing quite a few things to get them ready for pip 20.2b2.

What did I do last week?

I've spent most of the time on getting the opt-in of obtaining dependency information via lazy wheels ready. It will be available as --use-feature=fast-deps and only has effect when --use-feature=2020-resolver also presents.

While waiting for reviews and suggestions, I made some patches for internal cleansing, namely GH-8568, GH-8571 and GH-8578. Some of the similar patches I made earlier were also merged last week: GH-8456 and GH-8538.

Did I get stuck anywhere?

Not really, everything was going as expected for me.

What is coming up next?

After GH-8532, I'll work on the parallel download of the postponed wheels. My main current concern is with how the download progress will be reported to the users, but I think I'll figure it out soon.

I'm Not Drowning On My Own

vn.mcsinyx@gmail.com (McSinyx) — Mon, 06 Jul 2020 19:09:15 +0000

Cold Water

Hello there! My schoolyear is coming to an end, with some final assignments and group projects left to be done. I for sure underestimated the workload of these and in the last (and probably next) few days I'm drowning in work trying to meet my deadlines.

One project that might be remotely relevant is cheese-shop, which tries to manage the metadata of packages from the real Cheese Shop. Other than that, schoolwork is draining a lot of my time and I can't remember the last time I came up with something new for my GSoC Project )-;

Warm Water

On the bright side, I received a lot of help and encouragement from contributors and stakeholders of pip. In the last week alone, I had five pull requests merged:

GH-8332: Add license requirement to _vendor/README.rst
GH-8320: Add utilities for parallelization
GH-8504: Parallelize pip list --outdated and --uptodate
GH-8411: Refactor operations.prepare.prepare_linked_requirement
GH-8467: Add utitlity to lazily acquire wheel metadata over HTTP

In addition to helping me getting my PRs merged, my mentor Pradyun Gedam also gave me my first official feedback, including what I'm doing right (and wrong too!) and what I should keep doing to increase the chance of the project being successful.

GH-7819's roadmap (Danny McClanahan's discoveries and works on lazy wheels) is being closely tracked by hatch's maintainter Ofek Lev, which really makes me proud and warms my heart, that what I'm helping build is actually needed by the community!

Learning How To Swim

With GH-8467 and GH-8530 merged, I'm now working on GH-8532 which aims to roll out the lazy wheel as the way to obtain dependency information via the CLI flag --use-feature=lazy-wheel.

GH-8532 was failing initially, despite being relatively trivial and that the commit it used to base on was passing. Surprisingly, after rebasing it on top of GH-8530, it suddenly became green mysteriously. After the first (early) review, I was able to iterate on my earlier code, which used the ambiguous exception RuntimeError.

The rest to be done is just adding some functional tests (I'm pretty sure this will be either overwhelming or underwhelming) to make sure that the command-line flag is working correctly. Hopefully this can make it into the beta of the upcoming release this month.

In other news, I've also submitted a patch improving the tests for the parallelization utilities, which was really messy as I wrote them. Better late than never!

Metaphors aside, I actually can't swim d-:

Dive Plan

After GH-8532, I think I'll try to parallelize downloads of wheels that are lazily fetched only for metadata. By the current implementation of the new resolver, for pip install, this can be injected directly between the resolution and build/installation process.

Third Check-In

vn.mcsinyx@gmail.com (McSinyx) — Mon, 29 Jun 2020 15:49:58 +0000

Holla, holla, holla! Last seven days has not been a really productive week for me, though I think there are still some nice things to share with you all here! The good news is that I've finish my last leçon as a somophore, the bad news is that I have a bunch of upcoming tests, mainly in the form of group projects and/or presentation (phew!). Enough about me, let's get back to pip:

What did I do last week?

Not much, actually )-:

Write some tests for the HTTP range mapping for wheel.
Try to bring back multithreaded pip list --outdated and --uptodate}, as the parallel map was merged earlier today
Nitpick GH-8332 (yep it's a new low for me to include this to the list (-:).

Did I get stuck anywhere?

Not exactly, since I didn't do much d-; Many of my PRs are stalling though. On one hand the maintainers of pip are all volunteers working in there free time, on the other hand I don't think I have tried hard enough to get their attention on my PRs.

What is coming up next?

I'll try my best getting the following merged upstream before the upcoming beta release:

Parallel networking for pip list: GH-8504
Lazy wheel for dependency information: GH-8467, GH-8411 (to determine if hashing is required) and a new patch introducing this as an unstable feature

The Wonderful Wizard of O'zip

vn.mcsinyx@gmail.com (McSinyx) — Mon, 22 Jun 2020 19:35:31 +0000

Never give up… No one knows what's going to happen next.

Preface

Greetings and best wishes! I had a lot of fun during the last week, although admittedly nothing was really finished. In summary, these are the works I carried out in the last seven days:

Finilizing utilities for parallelization
Continuing experimenting on using lazy wheels for dependency resolution
Polishing up the patch refactoring operations.prepare.prepare_linked_requirement
Adding flake8-logging-format to linter
Splitting the linting patch from the PR adding the license requirement to vendor README

The `multiprocessing[.dummy]` wrapper

Yes, you read it right, this is the same section as last fortnight's blog. My mentor Pradyun Gedam gave me a green light to have GH-8411 merged without support for Python 2 and the non-lazy map variant, which turns out to be troublesome for multithreading.

The tests still needs to pass of course and the flaky tests (see failing tests over Azure Pipeline in the past) really gave me a panic attack earlier today. We probably need to mark them as xfail or investigate why they are undeterministic specifically on Azure, but the real reason I was all caught up and confused was that the unit tests I added mess with the cached imports and as pip's tests are run in parallel, who knows what it might affect. I was so relieved to not discover any new set of tests made flaky by ones I'm trying to add!

The file-like object mapping ZIP over HTTP

This is where the fun starts. Before we dive in, let's recall some background information on this. As discovered by Danny McClanahan in GH-7819, it is possible to only download a potion of a wheel and it's still valid for pip to get the distribution's metadata. In the same thread, Daniel Holth suggested that one may use HTTP range requests to specifically ask for the tail of the wheel, where the ZIP's central directory record as well as where usually dist-info (the directory containing METADATA) can be found.

Well, usually. While PEP 427 does indeed recommend

Archivers are encouraged to place the .dist-info files physically at the end of the archive. This enables some potentially interesting ZIP tricks including the ability to amend the metadata without rewriting the entire archive.

one of the mentioned tricks is adding shared libraries to wheels of extension modules (using e.g. auditwheel or delocate). Thus for non-pure Python wheels, it is unlikely that the metadata lie in the last few megabytes. Ignoring source distributions is bad enough, we can't afford making an optimization that doesn't work for extension modules, which are still an integral part of the Python ecosystem )-:

But hey, the ZIP's directory record is warrantied to be at the end of the file! Couldn't we do something about that? The short answer is yes. The long answer is, well, yessssssss! That, plus magic provided by most operating systems, this is what we figured out:

We can download a realatively small chunk at the end of the wheel until it is recognizable as a valid ZIP file.
In order for the end of the archive to actually appear as the end to zipfile, we feed to it an object with seek and read defined. As navigating to the rear of the file is performed by calling seek with relative offset and whence=SEEK_END (see man 3 fseek for more details), we are completely able to make the wheels in the cloud to behave as if it were available locally.
For large wheels, it is better to store them in hard disks instead of memory. For smaller ones, it is also preferable to store it as a file to avoid (error-prony and often not really efficient) manual tracking and joining of downloaded segments. We only use a small potion of the wheel, however just in case one is wonderring, we have very little control over when tempfile.SpooledTemporaryFile rolls over, so the memory-disk hybrid is not exactly working as expected.
With all these in mind, all we have to do is to define an intermediate object check for local availability and download if needed on calls to read, to lazily provide the data over HTTP and reduce execution time.

The only theoretical challenge left is to keep track of downloaded intervals, which I finally figured out after a few trials and errors. The code was submitted as a pull request to pip at GH-8467. A more modern (read: Python 3-only) variant was packaged and uploaded to PyPI under the name of lazip. I am unaware of any use case for it outside of pip, but it's certainly fun to play with d-:

What's next?

I have been falling short of getting the PRs mention above merged for quite a while. With pip's next beta coming really soon, I have to somehow make the patches reach a certain standard and enough attention to be part of the pre-release—beta-testing would greatly help the success of the GSoC project. To other GSoC students and mentors reading this, I also hope your projects to turn out successful!

Second Check-In

vn.mcsinyx@gmail.com (McSinyx) — Mon, 15 Jun 2020 17:47:43 +0000

Hi everyone and may the odds ever in your favor, especially during this tough time!

What did I do last week?

Not as much I wished, apparently (-:

Finalizing the refactoring patch of operations.prepare.prepare_linked_requirement.
Nitpicking some logging calls. This (as well as the next one) was to fill up the time my brain not being as productive as I want it to XD
Beginning to migrate from %- to {}-style logging. The amount of tests failing due to this was way beyond my imagination, but I got functional tests for pip install and unit tests passing now!
Mocking up a working partial wheel download during dependency resolution for the new resolver.

Did I get stuck anywhere?

Yes, of course! parallel maps are still stalling as well as other small PRs listed above. The failure related to logging are still making me pulling my hair out and the proof of concept for partial wheel downloading is too ugly even for a PoC. I imagine that I will have a lot of clean up to do this week (yay!).

What is coming up next?

I'm trying get the multi-{threading,processing} facilities merged ASAP to start rolling it out in practice. The first thing popping out of my head is to get back the multi-threaded pip list -o.

The other experimental improvement (this phrase does not sound right!) I would like to get done is the partial wheel download. It would be really nice if I can get both of these included as unstable-features in the upcoming beta release of pip 20.2.

Unexpected Things When You're Expecting

vn.mcsinyx@gmail.com (McSinyx) — Mon, 08 Jun 2020 20:56:54 +0000

Hi everyone, I hope that you are all doing well and wishes you all good health! The last week has not been really kind to me with a decent amount of academic pressure (my school year is lasting until early Jully). It would be bold to say that I have spent 10 hours working on my GSoC project since the last check-in, let alone the 30 hours per week requirement. That being said, there were still some discoveries that I wish to share.

The `multiprocessing[.dummy]` wrapper

Most of the time I spent was to finalize the multi{processing,threading} wrapper for map function that submit tasks to the worker pool. To my surprise, it is rather difficult to write something that is not only portable but also easy to read and test.

By the latest commit, I realized the following:

The multiprocessing module was not designed for the implementation details to be abstracted a way entirely. For example, the lazy maps could be really slow without specifying suitable chunk size (to cut the input iterable and distribute them to workers in the pool). By suitable, I mean only an order smaller than the input. This defeats half of the purpose of making it lazy: allowing the input to be evaluated lazily. Luckily, in the use case I'm aiming for, the length of the iterable argument is small and the laziness is only needed for the output (to pipeline download and installation).
Mocking import for testing purposes can never be pretty. One reason is that we (Python users) have very little control over the calls of import statements and its lower-level implementation __import__. In order to properly patch this built-in function, unlike for others of the same group, we have to monkeypatch the name from builtins (or __builtins__ under Python 2) instead of the module that import stuff. Furthermore, because of the special namespacing, to avoid infinite recursion we need to alias the function to a different name for fallback.
To add to the problem, multiprocessing lazily imports the fragile module during pools creation. Since the failure is platform-specific (the lack of sem_open), it was decided to check upon the import of the pip's module. Although the behavior is easier to reason in human language, testing it requires invalidating cached import and re-import the wrapper module.
Last but not least, I now understand the pain of keeping Python 2 compatibility that many package maintainers still need to deal with everyday (although Python 2 has reached its end-of-life, pip, for example, will still support it for another year).

The change in direction

Since last week, my mentor Pradyun Gedam and I set up weekly real-time meeting (a fancy term for video/audio chat in the worldwide quarantine era) for the entire GSoC period. During the last session, we decided to put parallelization of download during resolution on hold, in favor of a more beneficial goal: partially download the wheels during dependency resolution.

As discussed by Danny McClanahan and the maintainers of pip, it is feasible to only download a few kB of a wheel to obtain enough metadata for the resolution of dependency. While this is only applicable to wheels (i.e. prebuilt packages), other packaging format only make up less than 20% of the downloads (at least on PyPI), and the figure is much less for the most popular packages. Therefore, this optimization alone could make the upcoming backtracking resolver's performance par with the legacy one.

During the last few years, there has been a lot of effort being poured into replacing pip's current resolver that is unable to resolve conflicts. While its correctness will be ensured by some of the most talented and hard-working developers in the Python packaging community, from the users' point of view, it would be better to have its performance not lagging behind the old one. Aside from the increase in CPU cycles for more rigorous resolution, more I/O, especially networking operations is expected to be performed. This is due to the lack of a standard and efficient way to acquire the metadata. Therefore, unlike most package managers we are familiar with, pip has to fetch (and possibly build) the packages solely for dependency informations.

Fortunately, PEP 427 recommends package builders to place the metadata at the end of the archive. This allows the resolver to only fetch the last few kB using HTTP range requests for the relevant information. Simply appending Range: bytes=-8000 to the request header in pip._internal.network.download makes the resolution process lightning fast. Of course this breaks the installation but I am confident that it is not difficult to implement this optimization cleanly.

One drawback of this optimization is the compatibility. Not every Python package warehouse support range requests, and it is not possible verify the partial wheel. While the first case is unavoidable, for the other, hashes checking is usually used for pinned/locked-version requirements, thus no backtracking is done during dependency resolution.

Either way, before installation, the packages selected by the resolver can be downloaded in parallel. This warranties a larger crowd of packages, compared to parallelization during resolution, where the number of downloads can be as low as one during trail of different versions of the same package.

Unfortunately, I have not been able to do much other than a minor clean up. I am looking forward to accomplishing more this week and seeing what this path will lead us too! At the moment, I am happy that I'm able to meet the blog deadline, at least in UTC!

First Check-in

vn.mcsinyx@gmail.com (McSinyx) — Mon, 01 Jun 2020 10:32:28 +0000

Hi everyone, I am McSinyx, a Vietnamese undergraduate student who loves free software. This summer I am working with the maintainers and the contributors of pip to make the package manager download in parallel.

What did I do during the community bonding period?

Aside from bonding with pip's maintainers and contributors as well as with my mentors, I was also experimenting on the theoretical and technical obstacles blocking this GSoC project. Pradyun Gedam (a mentor of mine) suggested making a proof of concept to determine if parallel downloading can play nicely with ResolveLib's abstraction and we are reviewing it together. On the technical side, we pip's committers are exploring available options for parallelization and I made an attempt to make use of Python's standard worker pool in a portable way.

Did I get stuck anywhere?

Yes, of course! Neither of the experiments above is finished as of this moment. Though, I am optimistic that the issues will not be real blockers and we will figure that out in the next few days.

What is coming up next?

As planned, this week I am going to refactor the package downloading code pip. The main purpose is to decouple the networking code from the package preparation operation and make sure that it is thread-safe.

In addition, I am also continuing mentioned experiments to have a better confidence on the future of this GSoC project.

To other GSoC students, mentors and admins reading this, I am wishing you all good health and successful projects this summer!

Articles on McSinyx's Blog

Outro

The Look

The Benchmark

Average Distribution

Large Distribution

Distribution with Conflicting Dependencies

What Now?

Final Check-In

What did I do last week?

Did I get stuck anywhere?

What is coming up next?

Parallelizing Wheel Downloads

Download Parallelization

Progress Reporting UI

Performance Issues

Sixth Check-In

What did I do last week?

Did I get stuck anywhere?

What is coming up next?

Sorting Things Out

pip 20.2 Released!

Preparation for Download Parallelization

Fifth Check-In

What did I do last week?

Did I get stuck anywhere?

What is coming up next?

I've Walked 500 Miles…

The Main Road

The Side Quests

Snap Back to Reality

Fourth Check-In

What did I do last week?

Did I get stuck anywhere?

What is coming up next?

I'm Not Drowning On My Own

Cold Water

Warm Water

Learning How To Swim

Dive Plan

Third Check-In

What did I do last week?

Did I get stuck anywhere?

What is coming up next?

The Wonderful Wizard of O'zip

Preface

The multiprocessing[.dummy] wrapper

The file-like object mapping ZIP over HTTP

What's next?

Second Check-In

What did I do last week?

Did I get stuck anywhere?

What is coming up next?

Unexpected Things When You're Expecting

The multiprocessing[.dummy] wrapper

The change in direction

First Check-in

What did I do during the community bonding period?

Did I get stuck anywhere?

What is coming up next?

`pip` 20.2 Released!

The `multiprocessing[.dummy]` wrapper

The `multiprocessing[.dummy]` wrapper