tomasb's Blog

Weekly blog #6 (week 12): 12/08 to 18/08

Published: 08/19/2019

Hello there! We are at the end of GSoC! Week 12 was the last week where we’d do any major coding. We still have the next week where we do a submission (and a final check-in?), so let me keep it short and tell you what I worked on this week and what I got stuck on.


This week I focused on my two open PR’s - the text PR and the PyTorch PR.


For the text PR, I did a couple of things:

  • I updated the image tutorial and image tests with new features introduced in this PR, such as explanation “modifiers”.

  • I fleshed out (padded with comments) the text tutorial so that it looks nice.

  • I opened issues on GitHub for some TODO’s and FIXME’s.

  • One of the mentors reviewed the PR so I made any appropriate changes, including:

    • API changes - instead of letting an argument take multiple types, have separate arguments for each type.

    • An interesting change - when trying to choose a hidden layer for Grad-CAM, we will now default to one of the latter layers (closer to output). This is closer to how Grad-CAM is defined (since picking earlier layers would be like using some other gradient method for explanations).


One interesting issue that arose as a result of the above change (defaulting to “higher level” layers) was that my integration tests broke and the tutorial explanations didn’t look as good. With the guidance of my mentor I resolved the failing tests by explicitly picking a layer that works. For the tutorials I made a comment that using earlier layers gave better results. Interesting issues with text!


Next for the PyTorch PR I did the following:

  • Added code that can extract an image from an input tensor (a simple transform in torchvision), like what we do for Keras.

  • Added a very short image tutorial.


I got stuck on the text part of PyTorch because my test model was quite large (large word embeddings). It looks like I will have to train my own embedding layer with a small vocabulary.


What’s next

  • Final submission coming soon.

  • Slowly work on getting text and pytorch PR’s merged as a regular open source contributor!

    • Text PR still has some TODO’s and reviews to resolve.

    • PyTorch PR needs a text tutorial, integration tests, and unit tests. We’ll scale down so hopefully this doesn’t take too much time.


That’s all the technical details! I think we have one more blog next week, so I can talk about GSoC in broader terms then?


Thanks for reading once again!

Tomas Baltrunas

View Blog Post

Weekly check-in #8 (week 11): 05/08 to 11/08

Published: 08/12/2019

Hi! We’re heading for the last week of coding in GSoC. The pressure is on. 

What did you do this week?

After mentors’ reviews the image-related code got merged into ELI5 master. Firstly the old image PR got merged. Then I opened a second PR to make API improvements to the first. The improvements ended up being reversed, but they were still made. For example we no longer require RGBA-only images. We can take images with any “mode”.


Since this has occupied me for most of the week, not much work was done on the other two PR’s. Nevertheless, for PyTorch I added the ability to specify a model’s layer as a string. For the text PR I made the automatic layer chooser more advanced, and added unit tests.

What is coming up next?

Week 12 will be a rush to complete the project. Specifically I will focus on getting the text and PyTorch PR’s as close to merging as possible.


For text, the major TODO’s are: 

  • Flesh out the text tutorial.

  • Code clean up (names, existing and new comments).

For PyTorch I need to:

  • Add validation as in Keras.

  • Add automated tests.

  • Create image + text tutorial(s)


If we do merge, make a “prepare release” PR that includes the new features in the next ELI5 version.

Did you get stuck anywhere?

The first thing that I got stuck on was resolving many merge conflicts. Since my text PR builds on the image PR, making changes to the image code made the two branches diverge. I went through the conflicts manually, and in some cases the git mergetool helped. The lesson is to merge in smaller blocks (and if possible make your PR’s independent of each other).


Another problem that stuck out was coding up things that we decided to reverse later on. This is fine for small things and can be fixed by discussion, but this is a problem to watch out for when dealing with larger things.


That’s it for the week. Hopefully we can keep the merge count up! I won’t have the time to tinker with ELI5 on a real problem or get user feedback, but that’s okay.

Thanks for reading and stay tuned for one last blog next week!

Tomas Baltrunas

View Blog Post

Weekly blog #5 (week 10): 29/07 to 04/08

Published: 08/05/2019

Hey. It looks like we’re in August now. Summer is coming to an end fast!


What happened this week? One of my mentors came back from holidays and we had a sync up call on Monday, and another one on Thursday (which should’ve been on Wednesday but I overslept. Set your alarms students!). We discussed the state of the project and what else can we do before GSoC ends.


At the start of the week I opened a PyTorch PR that supports both image and text explanations.


One thing that I was stuck on was saving and loading PyTorch models. Unlike in Keras where you can use one function call to load a model, in PyTorch there are multiple steps: get the class for the model, instantiate a model object, load the saved “state dict” into that new instance. In my case I was stuck on instantiating the model object, which required passing pre-trained word embeddings. My workaround was passing a tensor of zeros (instead of for example passing glove vectors, which were in the “state dict” anyways).


For the rest of the week I worked on the text PR. The first major progression was in adding integration tests. It took me some time to train and prepare two quality models from scratch, one for sentiment classification and another for multi-class classification.


Another development was in the documentation - function docstrings, library docs for text, adding new packages to the index, etc.


Manually testing the text PR, everything looked good apart from a few things. There was an issue with explaining “1D” layers (dense/fully-connected and final recurrent layers) that I fixed by adding special case code.


A remaining issue is with multi-label classification (, where the explanations do not explain anything. One of my mentor’s suggested solutions was starting with a simple model and building from there.


These were the major issues and solutions for week 10. For week 11 I hope to follow this rough schedule:



  • In the text PR, move changes to the image API to a new PR. This way we can merge PR’s faster and possibly get early feedback.


  • Text PR remaining unit tests.
  • Text PR tutorial (now that I have two good models from the integration tests).


  • Text PR ad hoc tasks (FIXME’s, look at other implementations).
  • PyTorch PR refactor code (^).


  • PyTorch PR tests (manual and automated?).
  • PyTorch PR docs, tutorial.


  • Contingency (^^^)


That’s all for the week. Thank you for reading!

Tomas Baltrunas

View Blog Post

Weekly check-in #7 (week 9): 22/07 to 28/07

Published: 07/28/2019

Hi all. Hope we all passed evaluation 2 and our work continues!

What did you do this week?

The text PR saw a couple of updates, such as support for more ways to specify certain arguments (take more types), a change to the underlying method used for 1D array resizing (replaced scipy.signal with scipy.interpolate), and unit tests for text helpers.


I moved the code that could be reused by different neural network packages (Keras, PyTorch, Tensorflow, etc) into a new ‘nn’ package.


A major feature was adding PyTorch support in a new branch off the text PR. I got working support for images (torchvision and reusable numpy code helped a lot), but text support is yet to be added.

What is coming up next?

Work will continue as usual: Text PR needs fixes, docs and tests. In particular, I will need to train better models for integration tests.


I should open a PyTorch WIP PR once text support is added. That includes obtaining a text model with which I can do manual tests. A stretch goal would be to also get started with docs and other things that come with PR’s.


Since one of my mentors is back from holidays, I will take any feedback, and we can also plan out the last 3 weeks of GSoC. Besides the current tasks, there is some room in the schedule for other things.

Did you get stuck anywhere?

As mentioned, I need to train text models specifically designed for integration tests – small, covering many cases (kinds of layers used, character/word level tokenization, output classes, etc), and working well with explanations. I tried to add an integration test but the model I used gave dubious explanations for simple sentences, so changing the model is a solution I can try.


I was keen on opening a PyTorch PR this week, but a blocker was adding text support. Specifically I didn’t have an off-the-shelf pretrained text model to test with. The simplest solution I tried was saving a text classifier from a public kaggle notebook (I got an error when trying to save so I will need to retry).


I shall see you next week!

Tomas Baltrunas

View Blog Post

Weekly blog #4 (week 8): 15/07 to 21/07

Published: 07/21/2019

Hey. During week 8 I was working mostly on my own, since one of my mentors was on holidays. I did a few things, but on some days productivity or “throughput” just wasn’t good. Let me tell you about it.


Starting with text, I trained more text classifiers for manual tests, and opened a PR for the text codebase in ELI5 itself. In the codebase I did some refactoring and fixed broken image tests. I also added character-level explanations, and an option to remove pre-padding in text input.


Did I mention “character-level”? Indeed I managed to resolve last blog’s issue of low accuracy for a character-based network. Solution? Increase batch size from 30 to 500. and gave some ideas. (This fix was quite random and I got lucky, so I still have the problem of knowing how to train neural nets!)


I also trained a network with multiple LSTM layers. Training on CPU, first time things didn’t go past epoch 1. My sequences were way too long, so the solution was to reduce their length. In my case I found that LSTM’s realistically only work for about 100 tokens.


An interesting issue regarding refactoring the text codebase was to do with how to handle duplicate function arguments. Specifically we have a dispatcher function and multiple concrete functions. These concrete functions have unique arguments, but also share some. See for an example. I ended up not using any **kwargs magic and typed out all the arguments in the dispatcher and concrete functions myself (readable, but need to watch when making changes).


I followed through the PyTorch 60 minute blitz tutorial (mostly just copy pasting code and looking up docs for some objects). I must admit that I didn’t understand autograd, and I will need to go through it more carefully for gradient calculations.


I went on to load a pretrained model from torchvision and the imagenet dataset (since the dataset is 150 GB, I loaded just a few local samples). I will use this set up when creating Grad-CAM code.


Workflow wise, I discovered that using the PYTHONPATH variable, for example running `PYTHONPATH=$PWD/SOMEPATH jupyter notebook` is a convenient way to make sure Python can find your local modules. This is much easier than relocating your scripts or modifying sys.path.


Work on first image PR has stopped. Improvements to it are now being done through the second text PR.


So what I could’ve done more productivity-wise? I wish I could’ve progressed the text PR more. It is in a very early WIP stage still, with no tests, docs, and sprinkled with TODO’s/FIXME’s. Regarding PyTorch I wish I could’ve coded some experimental Grad-CAM code. Overall it is hard to not get lazy when working without “checks”, like leaving a comment about what you are working on or syncing up in more detail during the week.


For next week we have a second evaluation. I heard that I will have to submit PR’s, so it would be good to do two things: open a PyTorch ‘WIP’ PR once I get some Grad-CAM code, and remove the ‘WIP’ label from the text PR.


That’s it for the week. Hope the next one is productive!

Tomas Baltrunas

View Blog Post