For fork's sake - Fixing multiprocessing issues

Published: 07/15/2019

What did you do this week?

It was also reported that scipy.fft does not interact well with python's multiprocessing library. I was able to track down the issue and resolve it within a few hours, adding a unit test to ensure this wouldn't be broken again. However, when I went to update my uarray PR to use the new fast C++ version, the test started to fail. It turns out, the uarray multimethod objects needed to be pickleable in a very specific way that mimicks the behaviour of python Function types. Thankfully, after applying this fix the issue was resolved.

I've also this week:

  • added Hermitian symmetric transforms to scipy.fft (merged in scipy#10425)
  • added scipy.fft to the 1.4.0 release notes
  • updated my PR adding uarray support to scipy.fft
  • started work on adding a `scipy.fft` compatible interface to pyfftw in pyFFTW#269. This will be one of the first useful backend implementations for the new backend system.

What is coming up next?

The backend system PR is set to be merged very soon so I should be able to focus on adding the uarray protocol to pyfftw and possibly contributing compatible backends to other FFT libraries.

I can also work on pre-planned transforms.

Did you get stuck anywhere?

The multiprocessing incompatibilities posed an unexpected challange but I was able to work though the problems alright.