Week 11: Debugging F2PY's parsing signature files

Debugging F2PY's parsing signature files bug

What did I do this week?

This week I encountered a bug while trying to build SciPy where F2PY was not parsing SciPy's signature files correctly. Without sounding too technical I want to write how I found the cause of the bug and implemented its fix:

Encountering the bug: 

• I tried to build and test SciPy with dev.py using the command 

  python dev.py test

  This command builds SciPy from scratch using Meson and runs the tests. 

  • Using setup.py file instead would have been of no use since it uses f2py2e APIs instead of using f2py directly from the command line which would expose our frontend. The f2py2e APIs are yet to be refactored to incorporate the new frontend.

• The build was successful, but while testing I was constantly getting import errors from scipy/linalg/flapack_module. I inspected meson.build file on linalg level and to know the files it was trying to build using F2PY.

• So the problem was while building flapack.pyf.src. I compared the postlists produced by my frontend and old f2py2e and the difference was clear. 

Cause of the bug:

• The above linked f2py signature file mentions module name in it. I was parsing this file and passing the module name without knowing this is already handled it is already handled in the backend.
• So F2PY was creating 2 modules with same name, but mine was produced empty because F2PY assumed my module is different from the module mentioned in the pyf file 

Fixing the bug:

• I referred to the logic in f2p2e to name the produced modules and simply replaced mine with it. It fixed the bug and SciPy built and tested successfully.

Did I get stuck anywhere?

Yep, I am currently stuck trying to build NumPy. My changes when incorporated with NumPy v1.24.0 dev version produce a build which fails to import throwing the error:

Original error was: /home/namami2011/mambaforge/envs/scipy-dev/lib/python3.10/site-packages/numpy/core/_multiarray_umath.cpython-310-x86_64-linux-gnu.so: undefined symbol: npy_sqrt

The problem is with distutils build system probably. I have to debug and add a fix for this.

Interestingly, when my changes are added to NumPy version 1.23.2 the the build is produced correctly and Meson successfully builds SciPy with all the tests passing.

What will I do next week?

I will try to fix the above mention bug so that the latest NumPy version can build SciPy successfully. Then I will update my frontend PR and add a new PR for the new Meson backend.