Fourth week of GSoC: Second half of "short GSoC pause" (due to summer school), and getting back to work - Issues with the BIDS-validator
The past week I finished the summer school that I was running from the 11th of June until the 19th of June. It was a success given the reactions from all participants1. However I was very happy when I could return to my GSoC project and coding starting last Thursday.
I started by adding a feature to MNE-BIDS: When reading a raw file into a python object, we want to automatically scan an accompanying meta data file (channels.tsv) to populate the python object with information about bad channels in the data. When implementing the feature, most of the problems I encountered were due to an interaction with the BIDS-validator, which I want to dedicate today's blog post to.
The BIDS-validator
I have written about BIDS before: It's a standard for organizing neuroimaging data. A standard can only be a standard when there is a set of testable rules to follow. The BIDS-validator is a software that automatically checks a dataset for its compliance with the BIDS set of testable rules. The current BIDS-validator is written in JavaScript, which offers a unique advantage: It can be run locally inside a browser (see here), that is: No files are uploaded. This way, users of BIDS can employ the BIDS-validator without having to download software. Yet, for users with some programming experience, the BIDS-validator can also be downloaded as a command line tool to run on nodejs.
Alas, the big advantage of having the BIDS-validator instantiated in Javascript also comes at a cost: The programming language itself. With BIDS being a standard for scientific data, most of the user base consists of researchers. Only a fraction of researchers in the field of neuroscience is well versed in Javascript, with the lingua francas of the field being Matlab and Python (and according to my experiences increasingly Python and less and less Matlab). This means, that open source contributions from the researchers to the BIDS-validator are limited and the BIDS-validator development relies on a small core of contributors and to some extend on contributions from a commercial company, funded through grants given to BIDS. The resulting problem is that the BIDS-validator often lags behind the development of the standard ... or that not all rules are tested to an appropriate extend.
Some rules of BIDS are implemented in the form of regular expressions (see here), and are thus"programming language agnostic". This is a great starting point for BIDS-validators implemented in other languages, and there is some limited Python support.
Thinking about this, I often find myself going down the road of writing a complete BIDS-validator in Python. The advantage would be obvious: Much easier development! And the expense that it wouldn't be as easily available from the Browser as the current Javascript implementation would be negligible for anyone who can install a python package. Yet, as soon as we have more than one validator, we need to ensure that they produce exactly the same results ... and that could lead to another set of problems very soon.
It seems there is no easy way out of this situation. For me, that means that while I develop features with Python during my GSoC, I will have to occasionally spend disproportionate amounts of time debugging and enhacing the BIDS-validator with its codebase in Javascript.
1Although Eric from MNE-Python told me that "Success is measured by how many people you convinced to use and contribute to MNE-Python :)"