Week 12: Successfully compiling SciPy with the new F2PY frontend

The new F2PY frontend compiles SciPy successfully

This last week of my GSoC journey was mainly fucussed on making F2PY production ready. And for any change in F2PY, it is important that the change doesn't break SciPy. For the last 2 weeks I have been trying to build SciPy but was being constantly bombarded by edge cases and compilation errors. Finally this week, after a few changes to my frontend parser, SciPy compiled successfully and all the tests passed successfully except for one test case. You can have a look at my Pull request here with all the changes. I'll go into details of compilation and testing method below:-

Building SciPy for testing F2PY changes

If you go through installation instructions for SciPy it suggests that you build and install SciPy with its setup.py file. However, that was misleading in my case. The setup.py file actually uses F2PY APIs, bypassing the frontend entirely. Since my frontend was not even being called, my changes weren't being tested and for a week I was under the false impression that my humungous changes consisting of 1600 lines of code were working perfectly. Alas, I late relised my misconception and resorted to other methods of building SciPy which would rely on the new frontend.

Enter - scipy/dev.py. dev.py scipt was added in effort to move away from to-be-deprecated distutils and building SciPy with Meson instead. I noticed that this script called F2PY through the command line which would be handled by my new frontend. I tried to build and test SciPy with this scipt using the command

python dev.py test

Obviously the first attempt wasn't successful. The first problem I encountered was that the subprocess command dev.py was not comprehensible by argparse frontend. Let me explain why:

f2py filename.pyf --build-dir ./    # This is incorrect as the positional argument "filename.pyf" is present before the flags/optional arguments
f2py --build-dir ./ filename.pyf    # This is correct. All the optional arguments/flags must be placed before positional arguments

This limitation is due to how the argparse works. Currently, I have edited the frontend parser such that it transfers all the filenames to the end. But this is a temporary fix, and we should change F2PY documentation so that it clearly mentions this requirement and changed examples should should reflect this requirement from the user side.

There were a few bugs related to parsing module name from the pyf files but I won't go into unneccessary details here.

Results

F2PY now successfully build SciPy's fortran modules. All the test cases pass on my local machine except for one:- 

/home/namami2011/mambaforge/envs/scipy-dev/lib/python3.10/contextlib.py:79: in inner
    return func(*args, **kwds)
E   AssertionError: 
E   Arrays are not almost equal to 5 decimals
E   
E   Mismatched elements: 1 / 9 (11.1%)
E   Max absolute difference: 1.77845359e-05
E   Max relative difference: inf
E    x: array([[ 1.77845e-05,  1.13063e-05, -2.25008e-06],
E          [ 1.13025e-05,  7.33137e-06, -1.53296e-06],
E          [-2.24821e-06, -1.53482e-06,  3.44589e-07]])
E    y: array([[0., 0., 0.],
E          [0., 0., 0.],
E          [0., 0., 0.]])
        args       = (<function assert_array_almost_equal.<locals>.compare at 0x7f23cc887e20>, array([[ 1.77845359e-05,  1.13062561e-05, -2...[-2.24821270e-06, -1.53481960e-06,  3.44589353e-07]]), array([[0., 0., 0.],
       [0., 0., 0.],
       [0., 0., 0.]]))
        func       = <function assert_array_compare at 0x7f23e8b495a0>
        kwds       = {'err_msg': '', 'header': 'Arrays are not almost equal to 5 decimals', 'precision': 5, 'verbose': True}
        self       = <contextlib._GeneratorContextManager object at 0x7f23e8cbe5f0>
================================================================================================== short test summary info ===================================================================================================
FAILED scipy/linalg/tests/test_solvers.py::test_solve_discrete_are - AssertionError: 
============================================================ 1 failed, 37188 passed, 2172 skipped, 12176 deselected, 133 xfailed, 13 xpassed in 559.92s (0:09:19) ============================================================

The failed testcase is a precision failure, probably due to the old processor on my machine. This testcase also fails using the latest numpy/main branch, so I don't think this error could be arising from my changes.

All my work throughout GSoC has been pushed to my branch linked abpve. Please go through if you are interested and drop a comment if you have any ideas or would like to sugges any changes.

Did I get stuck anywhere?

I have pushed my changes to these two remote branches - Meson integration and F2PY frontend changes. All the testcases have passed except for 2 errors I am getting in the PR check tests. I am lokking into these but probably these errors won't be resolved during the GSoC period as I need my mentor's help in resolving these. I will continue to look into this and resolve the issue.

What will i do next week?

The next week is my last GSoC week, and I have to make work report of my overall work done during this period. I'll probably be busy in that.