Articles on NamamiShanker's Blog

Week 12: Successfully compiling SciPy with the new F2PY frontend

namami2011@gmail.com (NamamiShanker) — Fri, 09 Sep 2022 13:22:32 +0000

The new F2PY frontend compiles SciPy successfully

This last week of my GSoC journey was mainly fucussed on making F2PY production ready. And for any change in F2PY, it is important that the change doesn't break SciPy. For the last 2 weeks I have been trying to build SciPy but was being constantly bombarded by edge cases and compilation errors. Finally this week, after a few changes to my frontend parser, SciPy compiled successfully and all the tests passed successfully except for one test case. You can have a look at my Pull request here with all the changes. I'll go into details of compilation and testing method below:-

Building SciPy for testing F2PY changes

If you go through installation instructions for SciPy it suggests that you build and install SciPy with its setup.py file. However, that was misleading in my case. The setup.py file actually uses F2PY APIs, bypassing the frontend entirely. Since my frontend was not even being called, my changes weren't being tested and for a week I was under the false impression that my humungous changes consisting of 1600 lines of code were working perfectly. Alas, I late relised my misconception and resorted to other methods of building SciPy which would rely on the new frontend.

Enter - scipy/dev.py. dev.py scipt was added in effort to move away from to-be-deprecated distutils and building SciPy with Meson instead. I noticed that this script called F2PY through the command line which would be handled by my new frontend. I tried to build and test SciPy with this scipt using the command

python dev.py test

Obviously the first attempt wasn't successful. The first problem I encountered was that the subprocess command dev.py was not comprehensible by argparse frontend. Let me explain why:

f2py filename.pyf --build-dir ./    # This is incorrect as the positional argument "filename.pyf" is present before the flags/optional arguments
f2py --build-dir ./ filename.pyf    # This is correct. All the optional arguments/flags must be placed before positional arguments

This limitation is due to how the argparse works. Currently, I have edited the frontend parser such that it transfers all the filenames to the end. But this is a temporary fix, and we should change F2PY documentation so that it clearly mentions this requirement and changed examples should should reflect this requirement from the user side.

There were a few bugs related to parsing module name from the pyf files but I won't go into unneccessary details here.

Results

F2PY now successfully build SciPy's fortran modules. All the test cases pass on my local machine except for one:-

/home/namami2011/mambaforge/envs/scipy-dev/lib/python3.10/contextlib.py:79: in inner
    return func(*args, **kwds)
E   AssertionError: 
E   Arrays are not almost equal to 5 decimals
E   
E   Mismatched elements: 1 / 9 (11.1%)
E   Max absolute difference: 1.77845359e-05
E   Max relative difference: inf
E    x: array([[ 1.77845e-05,  1.13063e-05, -2.25008e-06],
E          [ 1.13025e-05,  7.33137e-06, -1.53296e-06],
E          [-2.24821e-06, -1.53482e-06,  3.44589e-07]])
E    y: array([[0., 0., 0.],
E          [0., 0., 0.],
E          [0., 0., 0.]])
        args       = (<function assert_array_almost_equal.<locals>.compare at 0x7f23cc887e20>, array([[ 1.77845359e-05,  1.13062561e-05, -2...[-2.24821270e-06, -1.53481960e-06,  3.44589353e-07]]), array([[0., 0., 0.],
       [0., 0., 0.],
       [0., 0., 0.]]))
        func       = <function assert_array_compare at 0x7f23e8b495a0>
        kwds       = {'err_msg': '', 'header': 'Arrays are not almost equal to 5 decimals', 'precision': 5, 'verbose': True}
        self       = <contextlib._GeneratorContextManager object at 0x7f23e8cbe5f0>
================================================================================================== short test summary info ===================================================================================================
FAILED scipy/linalg/tests/test_solvers.py::test_solve_discrete_are - AssertionError: 
============================================================ 1 failed, 37188 passed, 2172 skipped, 12176 deselected, 133 xfailed, 13 xpassed in 559.92s (0:09:19) ============================================================

The failed testcase is a precision failure, probably due to the old processor on my machine. This testcase also fails using the latest numpy/main branch, so I don't think this error could be arising from my changes.

All my work throughout GSoC has been pushed to my branch linked abpve. Please go through if you are interested and drop a comment if you have any ideas or would like to sugges any changes.

Did I get stuck anywhere?

I have pushed my changes to these two remote branches - Meson integration and F2PY frontend changes. All the testcases have passed except for 2 errors I am getting in the PR check tests. I am lokking into these but probably these errors won't be resolved during the GSoC period as I need my mentor's help in resolving these. I will continue to look into this and resolve the issue.

What will i do next week?

The next week is my last GSoC week, and I have to make work report of my overall work done during this period. I'll probably be busy in that.

Week 11: Debugging F2PY's parsing signature files

namami2011@gmail.com (NamamiShanker) — Wed, 31 Aug 2022 14:59:36 +0000

Debugging F2PY's parsing signature files bug

What did I do this week?

This week I encountered a bug while trying to build SciPy where F2PY was not parsing SciPy's signature files correctly. Without sounding too technical I want to write how I found the cause of the bug and implemented its fix:

Encountering the bug:

I tried to build and test SciPy with dev.py using the command
```
python dev.py test
```
This command builds SciPy from scratch using Meson and runs the tests.
- Using setup.py file instead would have been of no use since it uses f2py2e APIs instead of using f2py directly from the command line which would expose our frontend. The f2py2e APIs are yet to be refactored to incorporate the new frontend.
The build was successful, but while testing I was constantly getting import errors from scipy/linalg/flapack_module. I inspected meson.build file on linalg level and to know the files it was trying to build using F2PY.
So the problem was while building flapack.pyf.src. I compared the postlists produced by my frontend and old f2py2e and the difference was clear.

Cause of the bug:

The above linked f2py signature file mentions module name in it. I was parsing this file and passing the module name without knowing this is already handled it is already handled in the backend.
So F2PY was creating 2 modules with same name, but mine was produced empty because F2PY assumed my module is different from the module mentioned in the pyf file

Fixing the bug:

I referred to the logic in f2p2e to name the produced modules and simply replaced mine with it. It fixed the bug and SciPy built and tested successfully.

Did I get stuck anywhere?

Yep, I am currently stuck trying to build NumPy. My changes when incorporated with NumPy v1.24.0 dev version produce a build which fails to import throwing the error:

Original error was: /home/namami2011/mambaforge/envs/scipy-dev/lib/python3.10/site-packages/numpy/core/_multiarray_umath.cpython-310-x86_64-linux-gnu.so: undefined symbol: npy_sqrt

The problem is with distutils build system probably. I have to debug and add a fix for this.

Interestingly, when my changes are added to NumPy version 1.23.2 the the build is produced correctly and Meson successfully builds SciPy with all the tests passing.

What will I do next week?

I will try to fix the above mention bug so that the latest NumPy version can build SciPy successfully. Then I will update my frontend PR and add a new PR for the new Meson backend.

Week 10: Testing the Meson Backend building system

namami2011@gmail.com (NamamiShanker) — Thu, 25 Aug 2022 16:16:08 +0000

Bulding SciPy with new F2PY's new Meson backend.

What did I do this week?

I was busy this week and a lot on my plate other than GSoC. I couldn't do much this week, mostly spent trying to build SciPy using my version of F2PY.

What will I do the next week?

I will integrate --f77exec, --f90exec, --f77flags and --f90flags into meson building. These will be implemented through custom_targets options of Meson. Also I will test building SciPy with F2PY's Meson backend.

Week 9: Meson backend for F2PY is completed (Mostly)

namami2011@gmail.com (NamamiShanker) — Tue, 16 Aug 2022 19:31:42 +0000

Completing major part of Meson backend for F2PY

What did I do this week?

As you know, F2PY provides compilation options to build the Python module for you. As distutils is being deprecated, F2PY is moving away from using NumPy's distutils for building shared libraries. Therefore, F2PY community had agreed to integrate Meson building option into F2PY. I am happy to say that most of the compilation options F2PY provides have been successfully integrated into Meson. I will release a bigger blog post explaining the changes that have taken place and the decisions made by me.

What will I do next week?

I will integrate --f77exec, --f90exec, --f77flags and --f90flags into meson building. These will be implemented through custom_targets options of Meson.

Did I get stuck anywhere?

No, all this was already planned months ago, I am not stuck anywhere.

Week 8: Meson backend for F2PY

namami2011@gmail.com (NamamiShanker) — Fri, 12 Aug 2022 17:58:01 +0000

Implementing backend compilation in F2PY in the form of a dependency injection

What did I do this week?

I created a structure for dependency injection of backend compilation framework into F2PY for compiling Fortran sources and C wrapper. The work is still in prototype stage with support of most of features of F2PY. By the next week I aim to complete a working model for building of F2PY generated modules with Meson. However, I would still need to add support for Fortran 77 and Fortran 90 specific custom target building. If you are not able to understand all this jargon just wait for a few days I will release a fill blog post of the changes that have taken place.

What will I do next week?

I will implement remaining F2PY compile options in meson building module.

Did I get stuck anywhere?

No I didn't get stuck anywhere because all this was already planned way before in my first week of GSoC. Link to my complete article on how to fully handle F2PY's compilation options in Meson this here: https://namamishanker.github.io/posts/f2py-meson-backend/

Week 6: Continuing working on the frontend argparse

namami2011@gmail.com (NamamiShanker) — Wed, 27 Jul 2022 13:21:10 +0000

Continuing working on the frontend in the draft PR

What did I do this week?

I implemented some changes suggested by reviewers on my draft PR.

Firstly, I added typing support to the functions I have made. Therefore, currently: f2pyarg.py, service.py, utils.py have typing support. Initially I had added typing library for type annotations. But Bas Van Beek suggested to replace them with buitlins type annotations. These are not currently supported in Python versions below 3.9, but can be used by from __future__ import annotations statement. The __future__ module is somewhat unique in that import therefrom enable behaviours that will only become default in future python versions. Older examples of this include the print functions and with statement, but a more recent addition is the postponed evaluation of annotations (PEP 563), which implicitly stringifies all annotations during runtime (e.g. a: int automatically becomes a: "int").

However, for Any type annotation I would still need to rely on typing module. Also, the Optional type annotation can also be replaced by builtins by piping the type with None keyword. (e.g. arr: Optional[list] can be replaced by arr: list | None). The tests work fine with the changes and the frontend looks more complete. The mid evaluation starts from the next week and I would need to present the changes to writers of F2PY. I hope my work is ready.

What will I do next week?

I will fill out the mid evaluation form, and make changes according to the review.

Did I get stuck anywhere?

No, I mostly applied changes requested in the review. These were mostly clear requests with helpful reviewers who clarified any doubts I had.

Week 5: The argparse F2PY is here

namami2011@gmail.com (NamamiShanker) — Wed, 27 Jul 2022 12:38:25 +0000

Finishing with F2PY's new frontend

What did I do this week?

I am so happy to say that I have completed a working and functioning argparse frontend for F2PY. If you had read my last week's post, I had said that I still had to debug and implement a few changes for F2PY.

Specifically:-

1. F2PY's compilation tests were failing, because there were some bug in --f2cmap flag.
2. F2PY would not compile .pyf.src files.
3. I forgot to add version mentioning flag.

I have fixed all the bugs and updated my one and only draft PR: https://github.com/numpy/numpy/pull/21923

All the tests are working correctly and the new f2pyarg should work exactly as f2py2e used to work. Also I found a few bugs that I will report and add fix for. These bugs are:

1. No test for verifying F2PY will work with .pyf.src files.
2. F2PY's test_kind test fails on Max OSX M1 chip.

Also, Rohit and I decided to drop one of our original goals. Before GSoC started we thought it would be a good idea to replace the current class based tests of F2PY to modern fixtures based tests. But we dropped it as the class based structure was performing very well, and the classes were responsible for compiling modules for testing.

What will I do next week?

I need to add some documentation changes and a release note for the changes have taken place. That's probably all I need to do for the next week, if another bug doesn't pop up.

Did I get stuck anywhere?

No, this week was comparatively a smooth sailing. I had already built the framework last week and, all I needed to do was to fix the bugs.

Week 4: Continuing F2PY's frontend

namami2011@gmail.com (NamamiShanker) — Wed, 27 Jul 2022 12:03:12 +0000

Reimplemeting F2PY's frontend with `argparse`

What did I do this week?

I continued with implementing F2PY's frontend with argparse library. This week I implemented compilation functionality provided by F2PY in the frontend framework. I have written a detailed blog here: https://namamishanker.github.io/posts/argparse-f2py/.

It was much more difficult that we expected, mostly because of the fact that we needed to maintain backwards compatibility with numpy's distutils. The full integration is still on the way and should be completed by the mid of the next week.

What will I do the next week?

I will continue to implement compilation functionality in F2PY. The base work has been done this week, but I still need to debug and verify the working of argparse F2PY.

Did I get stuck anywhere?

I'm glad you asked that. I did get stuck a lot this week. The way f2py2e integrated with numpy.distutils was a bit messy and much more complicated than I earlier expected. Please go through my linked blog if you are interested in the details. But then again, you probably aren't. Just know that it was a bit difficult.

Week 3: Renovating F2PY's frontend

namami2011@gmail.com (NamamiShanker) — Wed, 06 Jul 2022 11:25:46 +0000

Reimplmenting F2PY's frontend with `argparse` library

What did I do this week?

This week I worked on re-implementing `f2py2e's` run_main() function in with `argparse` library. The aim is to split up one big f2py2e.py into 2 files. One file `f2pyarg.py` is responsible for parsing arguments file while other `service.py` deals with f2py core functionalities.

I have opened a draft PR here - https://github.com/numpy/numpy/pull/21923. I will continue to add my frontend contributions to this branch.

I have tried to be keep my distance from F2PY's core functionalities. These functions are convoluted with each other and a small change in one core functionality might break another.

What will I do next week?

Some upcoming goals are:-

Implement run_main(). It is the function responsible for generating wrappers and .pyf files.
Implement run_compile() responsible for compiling and building shared libraries.
Decide and implement deprecation of flags. For ex. - `--f2py-wrapper-output`, `--skip-empty-wrappers`
Code cleanup. Will involve refactoring and deciding which flags belong to which subparser.

Did I get stuck anywhere?

It was difficult to integrate a new frontend to an existing and very old codebase. I kept getting stuck many times, but bebugged my way out of it. Currently some older tests are failing, but I think thats probably because of installation problems. Will look into it in the next meeting with my mentor Rogit Goswami.

Week 2: A study of F2PY's backend build system

namami2011@gmail.com (NamamiShanker) — Wed, 06 Jul 2022 11:14:19 +0000

Week 2 Moving F2PY to Meson build system

What did I do this week?

I did an extensive study on F2PY's build system functionality and what it would take to implement them in Meson. I wrote this extensive blog on Moving F2PY to Meson build system providing details.

I did not contribute any code this week as I was mostly studying Meson's documentation and finding f2py equivalent functionality.

What will I be doing next week?

I will start with implementing F2PY's frontend with `argparse` library.

Did I get stuck anywhere?

No, everything proceeds smoothly.

Week 1: A study of F2PY's frontend

namami2011@gmail.com (NamamiShanker) — Mon, 20 Jun 2022 15:52:54 +0000

Community bonding period and Week 1

What did I do this week?

During the community bonding period, I mostly scoured through the codebase, understanding F2PY's existing frontend, the on-going reimplementation of the frontend. I went through the existing tests and some of the tests that need to be added.

This first week of GSoC I understood the backend compilation system and the new compilation method that will follow post addition of meson. I didn't write much code this week because I want to understand and structure out the changes need first. F2PY is a big tool and it need to be modernised in a lot of aspects. I think I will go in with a complete knowledge of what I am building towards.

I have written a blog post on restructuring F2PY's frontend - F2PY's frontend

What will I be doing the next week?

I will go through the rest of the flags in F2PY's frontend and find out Meson's equivalent functionality that we will use. I will create a full report of the same and upload it on my website's blog. Also, I will make pull request of the documentation and test changes I have proposed.

Did I get stuck somewhere?

No not really, my mentor Rohit Goswami is very available and helps me out a lot. We didn't get stuck anywhere this week.