NamamiShanker's Blog

Week 12: Successfully compiling SciPy with the new F2PY frontend

Published: 09/09/2022

The new F2PY frontend compiles SciPy successfully

This last week of my GSoC journey was mainly fucussed on making F2PY production ready. And for any change in F2PY, it is important that the change doesn't break SciPy. For the last 2 weeks I have been trying to build SciPy but was being constantly bombarded by edge cases and compilation errors. Finally this week, after a few changes to my frontend parser, SciPy compiled successfully and all the tests passed successfully except for one test case. You can have a look at my Pull request here with all the changes. I'll go into details of compilation and testing method below:-

Building SciPy for testing F2PY changes

If you go through installation instructions for SciPy it suggests that you build and install SciPy with its file. However, that was misleading in my case. The file actually uses F2PY APIs, bypassing the frontend entirely. Since my frontend was not even being called, my changes weren't being tested and for a week I was under the false impression that my humungous changes consisting of 1600 lines of code were working perfectly. Alas, I late relised my misconception and resorted to other methods of building SciPy which would rely on the new frontend.

Enter - scipy/ scipt was added in effort to move away from to-be-deprecated distutils and building SciPy with Meson instead. I noticed that this script called F2PY through the command line which would be handled by my new frontend. I tried to build and test SciPy with this scipt using the command

python test

Obviously the first attempt wasn't successful. The first problem I encountered was that the subprocess command was not comprehensible by argparse frontend. Let me explain why:

f2py filename.pyf --build-dir ./    # This is incorrect as the positional argument "filename.pyf" is present before the flags/optional arguments
f2py --build-dir ./ filename.pyf    # This is correct. All the optional arguments/flags must be placed before positional arguments

This limitation is due to how the argparse works. Currently, I have edited the frontend parser such that it transfers all the filenames to the end. But this is a temporary fix, and we should change F2PY documentation so that it clearly mentions this requirement and changed examples should should reflect this requirement from the user side.

There were a few bugs related to parsing module name from the pyf files but I won't go into unneccessary details here.


F2PY now successfully build SciPy's fortran modules. All the test cases pass on my local machine except for one:- 

/home/namami2011/mambaforge/envs/scipy-dev/lib/python3.10/ in inner
    return func(*args, **kwds)
E   AssertionError: 
E   Arrays are not almost equal to 5 decimals
E   Mismatched elements: 1 / 9 (11.1%)
E   Max absolute difference: 1.77845359e-05
E   Max relative difference: inf
E    x: array([[ 1.77845e-05,  1.13063e-05, -2.25008e-06],
E          [ 1.13025e-05,  7.33137e-06, -1.53296e-06],
E          [-2.24821e-06, -1.53482e-06,  3.44589e-07]])
E    y: array([[0., 0., 0.],
E          [0., 0., 0.],
E          [0., 0., 0.]])
        args       = (<function assert_array_almost_equal.<locals>.compare at 0x7f23cc887e20>, array([[ 1.77845359e-05,  1.13062561e-05, -2...[-2.24821270e-06, -1.53481960e-06,  3.44589353e-07]]), array([[0., 0., 0.],
       [0., 0., 0.],
       [0., 0., 0.]]))
        func       = <function assert_array_compare at 0x7f23e8b495a0>
        kwds       = {'err_msg': '', 'header': 'Arrays are not almost equal to 5 decimals', 'precision': 5, 'verbose': True}
        self       = <contextlib._GeneratorContextManager object at 0x7f23e8cbe5f0>
================================================================================================== short test summary info ===================================================================================================
FAILED scipy/linalg/tests/ - AssertionError: 
============================================================ 1 failed, 37188 passed, 2172 skipped, 12176 deselected, 133 xfailed, 13 xpassed in 559.92s (0:09:19) ============================================================

The failed testcase is a precision failure, probably due to the old processor on my machine. This testcase also fails using the latest numpy/main branch, so I don't think this error could be arising from my changes.

All my work throughout GSoC has been pushed to my branch linked abpve. Please go through if you are interested and drop a comment if you have any ideas or would like to sugges any changes.

Did I get stuck anywhere?

I have pushed my changes to these two remote branches - Meson integration and F2PY frontend changes. All the testcases have passed except for 2 errors I am getting in the PR check tests. I am lokking into these but probably these errors won't be resolved during the GSoC period as I need my mentor's help in resolving these. I will continue to look into this and resolve the issue.

What will i do next week?

The next week is my last GSoC week, and I have to make work report of my overall work done during this period. I'll probably be busy in that.

View Blog Post

Week 11: Debugging F2PY's parsing signature files

Published: 08/31/2022

Debugging F2PY's parsing signature files bug

What did I do this week?

This week I encountered a bug while trying to build SciPy where F2PY was not parsing SciPy's signature files correctly. Without sounding too technical I want to write how I found the cause of the bug and implemented its fix:

Encountering the bug: 

  • I tried to build and test SciPy with using the command 
    python test

    This command builds SciPy from scratch using Meson and runs the tests. 

    • Using file instead would have been of no use since it uses f2py2e APIs instead of using f2py directly from the command line which would expose our frontend. The f2py2e APIs are yet to be refactored to incorporate the new frontend.

  • The build was successful, but while testing I was constantly getting import errors from scipy/linalg/flapack_module. I inspected file on linalg level and to know the files it was trying to build using F2PY.

  • So the problem was while building flapack.pyf.src. I compared the postlists produced by my frontend and old f2py2e and the difference was clear. 

Cause of the bug:

  • The above linked f2py signature file mentions module name in it. I was parsing this file and passing the module name without knowing this is already handled it is already handled in the backend.
  • So F2PY was creating 2 modules with same name, but mine was produced empty because F2PY assumed my module is different from the module mentioned in the pyf file 

Fixing the bug:

  • I referred to the logic in f2p2e to name the produced modules and simply replaced mine with it. It fixed the bug and SciPy built and tested successfully.

Did I get stuck anywhere?

Yep, I am currently stuck trying to build NumPy. My changes when incorporated with NumPy v1.24.0 dev version produce a build which fails to import throwing the error:

Original error was: /home/namami2011/mambaforge/envs/scipy-dev/lib/python3.10/site-packages/numpy/core/ undefined symbol: npy_sqrt

The problem is with distutils build system probably. I have to debug and add a fix for this.

Interestingly, when my changes are added to NumPy version 1.23.2 the the build is produced correctly and Meson successfully builds SciPy with all the tests passing.

What will I do next week?

I will try to fix the above mention bug so that the latest NumPy version can build SciPy successfully. Then I will update my frontend PR and add a new PR for the new Meson backend.

View Blog Post

Week 10: Testing the Meson Backend building system

Published: 08/25/2022

Bulding SciPy with new F2PY's new Meson backend. 

What did I do this week?

I was busy this week and a lot on my plate other than GSoC. I couldn't do much this week, mostly spent trying to build SciPy using my version of F2PY.

What will I do the next week?

I will integrate --f77exec, --f90exec, --f77flags and --f90flags into meson building. These will be implemented through custom_targets options of Meson. Also I will test building SciPy with F2PY's Meson backend.

View Blog Post

Week 9: Meson backend for F2PY is completed (Mostly)

Published: 08/16/2022

Completing major part of Meson backend for F2PY

What did I do this week?

As you know, F2PY provides compilation options to build the Python module for you. As distutils is being deprecated, F2PY is moving away from using NumPy's distutils for building shared libraries. Therefore, F2PY community had agreed to integrate Meson building option into F2PY. I am happy to say that most of the compilation options F2PY provides have been successfully integrated into Meson. I will release a bigger blog post explaining the changes that have taken place and the decisions made by me.

What will I do next week?

I will integrate --f77exec, --f90exec, --f77flags and --f90flags into meson building. These will be implemented through custom_targets options of Meson.

Did I get stuck anywhere?

No, all this was already planned months ago, I am not stuck anywhere.

View Blog Post

Week 8: Meson backend for F2PY

Published: 08/12/2022

Implementing backend compilation in F2PY in the form of a dependency injection

What did I do this week?

I created a structure for dependency injection of backend compilation framework into F2PY for compiling Fortran sources and C wrapper. The work is still in prototype stage with support of most of features of F2PY. By the next week I aim to complete a working model for building of F2PY generated modules with Meson. However, I would still need to add support for Fortran 77 and Fortran 90 specific custom target building. If you are not able to understand all this jargon just wait for a few days I will release a fill blog post of the changes that have taken place.

What will I do next week?

I will implement remaining F2PY compile options in meson building module.

Did I get stuck anywhere?

No I didn't get stuck anywhere because all this was already planned way before in my first week of GSoC. Link to my complete article on how to fully handle F2PY's compilation options in Meson this here:

View Blog Post