What did you do this week?

With the coding period coming to an end, this week I've wound down coding and focused on writing up my final work submission. This means I've been looking back at the work I've been doing; reading through my old PRs and comparing against my project's original goals.

My original proposal focused solely on adding backend support to scipy.fftpack, yet the scope of my work over the past 3 months was not limited there. The resulting scipy.fft subpackage is a complete rewrite of scipy.fftpack from the ground-up featuring not just the backend system but also an improved interface and the replacement of FFTPACK with pypocketfft under the hood.

My contributions were also not just limited to SciPy; I've also made contributed to pypocketfft, uarray, pyFFTW and cupy. I've really enjoyed getting to collaborate with this range of open source projects and am very pleased with how my work has gone over GSoC. I would thoroughly recommend that any students interested in open source software should consider doing a project themself.

What did you do this week?

The main feature I've been working on this week has been adding pre-planned transforms to pypocketfft in (pypocketfft!28). This will allow users to speed-up cases where many transforms of the same shape are done, without needing to rely on any internal caches. This gives the users a small performance improvement in pypocketfft but adding this to the scipy.fft interface will also allow other backends to benefit. e.g. pyFFTW can have a significantly more expensive planning phase.

Plans have now been added into the c++ interface of pypocketfft but I have still got to expose these in the python interface. Additionally, I've made a number of other contributions in several other places:

• Refactored the real transform tests to use pytest and to test all the transforms in long double precision as well (scipy#10669)
• Fixed a race condition in my custom threadpool code that was causing crashes on macOS (pypocketfft!27)
• Added benchmarks to my multithreading PR (scipy#10614)
• Opened an issue in mkl_fft about adding a scipy.fft interface (mkl_fft#42). Unfortunately, mkl_ffl doesn't seem to be very active and the issue hasn't received any response yet.
• Fixed a compile warning in uarray (uarray#195)

What is coming up next?

The coding period has officially ended and it's now time to "submit the code". Most of my code has already been merged into the SciPy master branch so this will just involve writing a summary of my work over the whole GSoC project.

I would also like to finish up the pre-planned transform code but it's likely too late to get this merged in time for the evaluation.

Did you get stuck anywhere?

Fixing the race condition in pypocketfft was quite challenging. The only symptom was a SEGFAULT that occurred only rarely and only on macOS, a platform which I don't use and so couldn't reproduce the issue locally. This made diagnosing and solving the issue very tricky. Thankfully, Martin (pypocketfft's author) had access to apple hardware and was eventually able to give me enough details that I found an issue from another project that had run into almost exactly the same issue.

What did you do this week?

• Fixed a small inconsistency with the numpy.fft API when the transform shape is given but not the axes (scipy#10594)
• Gotten my CuPy PR into a review ready state. Fixing uarray backend support (scipy#2355)
• Migrated pypocketfft from OpenMP to a custom thread-pool that I wrote to handle fork on POSIX systems. This addresses all the main concerns with adding OpenMP into SciPy (pypocketfft!23, pypocketfft!24)
• Using the updated pypocketfft, I was then able to open a PR adding multithreading support to the scipy.fft API (scipy#10614)

What is coming up next?

- Create, or at least open issue for, a mkl-fft backend
- Expand the scipy.fft tutorial to include installing and using new FFT backends

There are also a number of minor features that could be added to pypocketfft such as shape changing transforms or pre-planned transforms. Though at this late stage in the project, my focus is on polishing the current feature set.

Did you get stuck anywhere?

No significant blockers this week.

What did you do this week?

• I have opened a PR on cupy adding a scipy.fft backend
• Fixed a few small incompatibilities with the numpy.fft interface
• Updated next_fast_len, a utility function to find the fastest lengths for computing the FFT (pypocketfft!17, scipy#10570)
• Various clean-ups and refactoring of pypocketfft's multi-dimensional transform functions (pypocketfft!16, pypocketfft!20)
• Restarted a discussions about use of OpenMP in SciPy. pypocketfft already supports OpenMP for parallelising transforms over multidimensional arrays, however there are a number of issues preventing SciPy from building with this feature enabled.

What is coming up next?

• Replace OpenMP in pypocketfft with a custom thread-pool. The discussions on SciPy-dev have suggested that OpenMP isn't going to be a suitable solution so I will attempt to replace it.
• Finishing up my CuPy backend PR. It is mostly completed but needs a few more tests.
• Implement shape-changing transforms in pypocketfft.

Did you get stuck anywhere?

I found it quite difficult to get cuda and CuPy working properly on my computer and still occasionally have problems with it stopping recognising my cuda device. However, it's at least functional now if not quite perfect.

What did you do this week?

Perhaps the most significant task this week was completely removing the fortran FFTPACK code from SciPy and rewriting scipy.fftpack using scipy.fft's pocketfft backend. This drastically simplifies the code with a diff of -8,856 lines of code.

I've also made a number of smaller improvements to scipy.fft as well as a significant performance improvement to scipy.signal.fftconvolve.

The first scipy.fft backend has also been created after my pyFFTW PR was merged this week. This means that SciPy users can accelerate their FFT transforms with all of pyFFTW's features using nothing more than a few function calls.

What is coming up next?

• Adding a CuPy backend for scipy.fft
• Implementing efficient shape-changing transforms in pypocketfft
• Writing tests for the pyFFTW real-to-real transforms PR
• Adding the ability to query active backends for uarray

Did you get stuck anywhere?

No blockers this week.