Hi everyone!

It’s time for the last GSOC blog. I must say that it has been nice working with all of you. I have learned a lot and faced many professional challenges. In the end it was worth it. I hope GSOC experience will help me in the future and that my project will be useful to open source. You can check my final report at https://github.com/teabolt/gsoc19-eli5-gradcam.

Shout out to both of my mentors for taking me on board!

• Konstantin, https://github.com/lopuhin

  • Thank you for the constant help, knowledge, and advice! A mentor that I would recommend to any student!

• Mikhail, https://github.com/kmike

  • Thanks for jumping in at the right moments with important observations and help making the right decisions!

Shout out to other students

• Anubhav, https://github.com/anubhavp28

• Vipul, https://github.com/vipulgupta2048

• Leonardo, https://github.com/leovictorsr

• Ashwin, https://github.com/AshwinB-hat

  • Though not here for the final evaluation, thanks for sticking around and hopefully you had other good endeavours this summer!

Finally shout out to all the other participants in Scrapy, especially

• Ian, https://github.com/imduffy15

  • For helping me get into GSOC in the first place, and for all your engineering advice!

• Cathal, https://github.com/cathalgarvey

  • Thanks for your work as an org-admin!

So that’s it. Hope to see you in open source or in another GSOC.

Peace,

Tomas Baltrunas

Hello there! We are at the end of GSoC! Week 12 was the last week where we’d do any major coding. We still have the next week where we do a submission (and a final check-in?), so let me keep it short and tell you what I worked on this week and what I got stuck on.

This week I focused on my two open PR’s - the text PR and the PyTorch PR.

For the text PR, I did a couple of things:

• I updated the image tutorial and image tests with new features introduced in this PR, such as explanation “modifiers”.

• I fleshed out (padded with comments) the text tutorial so that it looks nice.

• I opened issues on GitHub for some TODO’s and FIXME’s.

• One of the mentors reviewed the PR so I made any appropriate changes, including:

  • API changes - instead of letting an argument take multiple types, have separate arguments for each type.

  • An interesting change - when trying to choose a hidden layer for Grad-CAM, we will now default to one of the latter layers (closer to output). This is closer to how Grad-CAM is defined (since picking earlier layers would be like using some other gradient method for explanations).

One interesting issue that arose as a result of the above change (defaulting to “higher level” layers) was that my integration tests broke and the tutorial explanations didn’t look as good. With the guidance of my mentor I resolved the failing tests by explicitly picking a layer that works. For the tutorials I made a comment that using earlier layers gave better results. Interesting issues with text!

Next for the PyTorch PR I did the following:

• Added code that can extract an image from an input tensor (a simple transform in torchvision), like what we do for Keras.

• Added a very short image tutorial.

I got stuck on the text part of PyTorch because my test model was quite large (large word embeddings). It looks like I will have to train my own embedding layer with a small vocabulary.

What’s next

• Final submission coming soon.

• Slowly work on getting text and pytorch PR’s merged as a regular open source contributor!

  • Text PR still has some TODO’s and reviews to resolve.

  • PyTorch PR needs a text tutorial, integration tests, and unit tests. We’ll scale down so hopefully this doesn’t take too much time.

That’s all the technical details! I think we have one more blog next week, so I can talk about GSoC in broader terms then?

Thanks for reading once again!

Tomas Baltrunas

Hi! We’re heading for the last week of coding in GSoC. The pressure is on. 

What did you do this week?

After mentors’ reviews the image-related code got merged into ELI5 master. Firstly the old image PR got merged. Then I opened a second PR to make API improvements to the first. The improvements ended up being reversed, but they were still made. For example we no longer require RGBA-only images. We can take images with any “mode”.

Since this has occupied me for most of the week, not much work was done on the other two PR’s. Nevertheless, for PyTorch I added the ability to specify a model’s layer as a string. For the text PR I made the automatic layer chooser more advanced, and added unit tests.

What is coming up next?

Week 12 will be a rush to complete the project. Specifically I will focus on getting the text and PyTorch PR’s as close to merging as possible.

For text, the major TODO’s are: 

• Flesh out the text tutorial.

• Code clean up (names, existing and new comments).

For PyTorch I need to:

• Add validation as in Keras.

• Add automated tests.

• Create image + text tutorial(s)

If we do merge, make a “prepare release” PR that includes the new features in the next ELI5 version.

Did you get stuck anywhere?

The first thing that I got stuck on was resolving many merge conflicts. Since my text PR builds on the image PR, making changes to the image code made the two branches diverge. I went through the conflicts manually, and in some cases the git mergetool helped. The lesson is to merge in smaller blocks (and if possible make your PR’s independent of each other).

Another problem that stuck out was coding up things that we decided to reverse later on. This is fine for small things and can be fixed by discussion, but this is a problem to watch out for when dealing with larger things.

That’s it for the week. Hopefully we can keep the merge count up! I won’t have the time to tinker with ELI5 on a real problem or get user feedback, but that’s okay.
 

Thanks for reading and stay tuned for one last blog next week!

Tomas Baltrunas

Hey. It looks like we’re in August now. Summer is coming to an end fast!

What happened this week? One of my mentors came back from holidays and we had a sync up call on Monday, and another one on Thursday (which should’ve been on Wednesday but I overslept. Set your alarms students!). We discussed the state of the project and what else can we do before GSoC ends.

At the start of the week I opened a PyTorch PR that supports both image and text explanations.

One thing that I was stuck on was saving and loading PyTorch models. Unlike in Keras where you can use one function call to load a model, in PyTorch there are multiple steps: get the class for the model, instantiate a model object, load the saved “state dict” into that new instance. In my case I was stuck on instantiating the model object, which required passing pre-trained word embeddings. My workaround was passing a tensor of zeros (instead of for example passing glove vectors, which were in the “state dict” anyways).

For the rest of the week I worked on the text PR. The first major progression was in adding integration tests. It took me some time to train and prepare two quality models from scratch, one for sentiment classification and another for multi-class classification.

Another development was in the documentation - function docstrings, library docs for text, adding new packages to the index, etc.

Manually testing the text PR, everything looked good apart from a few things. There was an issue with explaining “1D” layers (dense/fully-connected and final recurrent layers) that I fixed by adding special case code.

A remaining issue is with multi-label classification (https://keras.io/datasets/#reuters-newswire-topics-classification), where the explanations do not explain anything. One of my mentor’s suggested solutions was starting with a simple model and building from there.

These were the major issues and solutions for week 10. For week 11 I hope to follow this rough schedule:

Monday

• In the text PR, move changes to the image API to a new PR. This way we can merge PR’s faster and possibly get early feedback.

Tuesday

• Text PR remaining unit tests.
• Text PR tutorial (now that I have two good models from the integration tests).

Wednesday

• Text PR ad hoc tasks (FIXME’s, look at other implementations).
• PyTorch PR refactor code (^).

Thursday

• PyTorch PR tests (manual and automated?).
• PyTorch PR docs, tutorial.

Friday

• Contingency (^^^)

That’s all for the week. Thank you for reading!

Tomas Baltrunas

Hi all. Hope we all passed evaluation 2 and our work continues!

What did you do this week?

The text PR saw a couple of updates, such as support for more ways to specify certain arguments (take more types), a change to the underlying method used for 1D array resizing (replaced scipy.signal with scipy.interpolate), and unit tests for text helpers.

I moved the code that could be reused by different neural network packages (Keras, PyTorch, Tensorflow, etc) into a new ‘nn’ package.

A major feature was adding PyTorch support in a new branch off the text PR. I got working support for images (torchvision and reusable numpy code helped a lot), but text support is yet to be added.

What is coming up next?

Work will continue as usual: Text PR needs fixes, docs and tests. In particular, I will need to train better models for integration tests.

I should open a PyTorch WIP PR once text support is added. That includes obtaining a text model with which I can do manual tests. A stretch goal would be to also get started with docs and other things that come with PR’s.

Since one of my mentors is back from holidays, I will take any feedback, and we can also plan out the last 3 weeks of GSoC. Besides the current tasks, there is some room in the schedule for other things.

Did you get stuck anywhere?

As mentioned, I need to train text models specifically designed for integration tests – small, covering many cases (kinds of layers used, character/word level tokenization, output classes, etc), and working well with explanations. I tried to add an integration test but the model I used gave dubious explanations for simple sentences, so changing the model is a solution I can try.

I was keen on opening a PyTorch PR this week, but a blocker was adding text support. Specifically I didn’t have an off-the-shelf pretrained text model to test with. The simplest solution I tried was saving a text classifier from a public kaggle notebook (I got an error when trying to save so I will need to retry).

I shall see you next week!

Tomas Baltrunas