Articles on tomasb's Blog

Final Blog Post GSOC19

tomasbaltrunas0@gmail.com (tomasb) — Sat, 24 Aug 2019 12:24:42 +0000

Hi everyone!

It’s time for the last GSOC blog. I must say that it has been nice working with all of you. I have learned a lot and faced many professional challenges. In the end it was worth it. I hope GSOC experience will help me in the future and that my project will be useful to open source. You can check my final report at https://github.com/teabolt/gsoc19-eli5-gradcam.

Shout out to both of my mentors for taking me on board!

Konstantin, https://github.com/lopuhin
- Thank you for the constant help, knowledge, and advice! A mentor that I would recommend to any student!
Mikhail, https://github.com/kmike
- Thanks for jumping in at the right moments with important observations and help making the right decisions!

Shout out to other students

Anubhav, https://github.com/anubhavp28
Vipul, https://github.com/vipulgupta2048
Leonardo, https://github.com/leovictorsr
Ashwin, https://github.com/AshwinB-hat
- Though not here for the final evaluation, thanks for sticking around and hopefully you had other good endeavours this summer!

Finally shout out to all the other participants in Scrapy, especially

Ian, https://github.com/imduffy15
- For helping me get into GSOC in the first place, and for all your engineering advice!
Cathal, https://github.com/cathalgarvey
- Thanks for your work as an org-admin!

So that’s it. Hope to see you in open source or in another GSOC.

Peace,

Tomas Baltrunas

Weekly blog #6 (week 12): 12/08 to 18/08

tomasbaltrunas0@gmail.com (tomasb) — Mon, 19 Aug 2019 10:34:22 +0000

Hello there! We are at the end of GSoC! Week 12 was the last week where we’d do any major coding. We still have the next week where we do a submission (and a final check-in?), so let me keep it short and tell you what I worked on this week and what I got stuck on.

This week I focused on my two open PR’s - the text PR and the PyTorch PR.

For the text PR, I did a couple of things:

I updated the image tutorial and image tests with new features introduced in this PR, such as explanation “modifiers”.
I fleshed out (padded with comments) the text tutorial so that it looks nice.
I opened issues on GitHub for some TODO’s and FIXME’s.
One of the mentors reviewed the PR so I made any appropriate changes, including:
- API changes - instead of letting an argument take multiple types, have separate arguments for each type.
- An interesting change - when trying to choose a hidden layer for Grad-CAM, we will now default to one of the latter layers (closer to output). This is closer to how Grad-CAM is defined (since picking earlier layers would be like using some other gradient method for explanations).

One interesting issue that arose as a result of the above change (defaulting to “higher level” layers) was that my integration tests broke and the tutorial explanations didn’t look as good. With the guidance of my mentor I resolved the failing tests by explicitly picking a layer that works. For the tutorials I made a comment that using earlier layers gave better results. Interesting issues with text!

Next for the PyTorch PR I did the following:

Added code that can extract an image from an input tensor (a simple transform in torchvision), like what we do for Keras.
Added a very short image tutorial.

I got stuck on the text part of PyTorch because my test model was quite large (large word embeddings). It looks like I will have to train my own embedding layer with a small vocabulary.

What’s next

Final submission coming soon.
Slowly work on getting text and pytorch PR’s merged as a regular open source contributor!
- Text PR still has some TODO’s and reviews to resolve.
- PyTorch PR needs a text tutorial, integration tests, and unit tests. We’ll scale down so hopefully this doesn’t take too much time.

That’s all the technical details! I think we have one more blog next week, so I can talk about GSoC in broader terms then?

Thanks for reading once again!

Tomas Baltrunas

Weekly check-in #8 (week 11): 05/08 to 11/08

tomasbaltrunas0@gmail.com (tomasb) — Mon, 12 Aug 2019 00:15:50 +0000

Hi! We’re heading for the last week of coding in GSoC. The pressure is on.

What did you do this week?

After mentors’ reviews the image-related code got merged into ELI5 master. Firstly the old image PR got merged. Then I opened a second PR to make API improvements to the first. The improvements ended up being reversed, but they were still made. For example we no longer require RGBA-only images. We can take images with any “mode”.

Since this has occupied me for most of the week, not much work was done on the other two PR’s. Nevertheless, for PyTorch I added the ability to specify a model’s layer as a string. For the text PR I made the automatic layer chooser more advanced, and added unit tests.

What is coming up next?

Week 12 will be a rush to complete the project. Specifically I will focus on getting the text and PyTorch PR’s as close to merging as possible.

For text, the major TODO’s are:

Flesh out the text tutorial.
Code clean up (names, existing and new comments).

For PyTorch I need to:

Add validation as in Keras.
Add automated tests.
Create image + text tutorial(s)

If we do merge, make a “prepare release” PR that includes the new features in the next ELI5 version.

Did you get stuck anywhere?

The first thing that I got stuck on was resolving many merge conflicts. Since my text PR builds on the image PR, making changes to the image code made the two branches diverge. I went through the conflicts manually, and in some cases the git mergetool helped. The lesson is to merge in smaller blocks (and if possible make your PR’s independent of each other).

Another problem that stuck out was coding up things that we decided to reverse later on. This is fine for small things and can be fixed by discussion, but this is a problem to watch out for when dealing with larger things.

That’s it for the week. Hopefully we can keep the merge count up! I won’t have the time to tinker with ELI5 on a real problem or get user feedback, but that’s okay.

Thanks for reading and stay tuned for one last blog next week!

Tomas Baltrunas

Weekly blog #5 (week 10): 29/07 to 04/08

tomasbaltrunas0@gmail.com (tomasb) — Mon, 05 Aug 2019 12:10:59 +0000

Hey. It looks like we’re in August now. Summer is coming to an end fast!

What happened this week? One of my mentors came back from holidays and we had a sync up call on Monday, and another one on Thursday (which should’ve been on Wednesday but I overslept. Set your alarms students!). We discussed the state of the project and what else can we do before GSoC ends.

At the start of the week I opened a PyTorch PR that supports both image and text explanations.

One thing that I was stuck on was saving and loading PyTorch models. Unlike in Keras where you can use one function call to load a model, in PyTorch there are multiple steps: get the class for the model, instantiate a model object, load the saved “state dict” into that new instance. In my case I was stuck on instantiating the model object, which required passing pre-trained word embeddings. My workaround was passing a tensor of zeros (instead of for example passing glove vectors, which were in the “state dict” anyways).

For the rest of the week I worked on the text PR. The first major progression was in adding integration tests. It took me some time to train and prepare two quality models from scratch, one for sentiment classification and another for multi-class classification.

Another development was in the documentation - function docstrings, library docs for text, adding new packages to the index, etc.

Manually testing the text PR, everything looked good apart from a few things. There was an issue with explaining “1D” layers (dense/fully-connected and final recurrent layers) that I fixed by adding special case code.

A remaining issue is with multi-label classification (https://keras.io/datasets/#reuters-newswire-topics-classification), where the explanations do not explain anything. One of my mentor’s suggested solutions was starting with a simple model and building from there.

These were the major issues and solutions for week 10. For week 11 I hope to follow this rough schedule:

Monday

In the text PR, move changes to the image API to a new PR. This way we can merge PR’s faster and possibly get early feedback.

Tuesday

Text PR remaining unit tests.
Text PR tutorial (now that I have two good models from the integration tests).

Wednesday

Text PR ad hoc tasks (FIXME’s, look at other implementations).
PyTorch PR refactor code (^).

Thursday

PyTorch PR tests (manual and automated?).
PyTorch PR docs, tutorial.

Friday

Contingency (^^^)

That’s all for the week. Thank you for reading!

Tomas Baltrunas

Weekly check-in #7 (week 9): 22/07 to 28/07

tomasbaltrunas0@gmail.com (tomasb) — Sun, 28 Jul 2019 19:37:32 +0000

Hi all. Hope we all passed evaluation 2 and our work continues!

What did you do this week?

The text PR saw a couple of updates, such as support for more ways to specify certain arguments (take more types), a change to the underlying method used for 1D array resizing (replaced scipy.signal with scipy.interpolate), and unit tests for text helpers.

I moved the code that could be reused by different neural network packages (Keras, PyTorch, Tensorflow, etc) into a new ‘nn’ package.

A major feature was adding PyTorch support in a new branch off the text PR. I got working support for images (torchvision and reusable numpy code helped a lot), but text support is yet to be added.

What is coming up next?

Work will continue as usual: Text PR needs fixes, docs and tests. In particular, I will need to train better models for integration tests.

I should open a PyTorch WIP PR once text support is added. That includes obtaining a text model with which I can do manual tests. A stretch goal would be to also get started with docs and other things that come with PR’s.

Since one of my mentors is back from holidays, I will take any feedback, and we can also plan out the last 3 weeks of GSoC. Besides the current tasks, there is some room in the schedule for other things.

Did you get stuck anywhere?

As mentioned, I need to train text models specifically designed for integration tests – small, covering many cases (kinds of layers used, character/word level tokenization, output classes, etc), and working well with explanations. I tried to add an integration test but the model I used gave dubious explanations for simple sentences, so changing the model is a solution I can try.

I was keen on opening a PyTorch PR this week, but a blocker was adding text support. Specifically I didn’t have an off-the-shelf pretrained text model to test with. The simplest solution I tried was saving a text classifier from a public kaggle notebook (I got an error when trying to save so I will need to retry).

I shall see you next week!

Tomas Baltrunas

Weekly blog #4 (week 8): 15/07 to 21/07

tomasbaltrunas0@gmail.com (tomasb) — Sun, 21 Jul 2019 19:54:44 +0000

Hey. During week 8 I was working mostly on my own, since one of my mentors was on holidays. I did a few things, but on some days productivity or “throughput” just wasn’t good. Let me tell you about it.

Starting with text, I trained more text classifiers for manual tests, and opened a PR for the text codebase in ELI5 itself. In the codebase I did some refactoring and fixed broken image tests. I also added character-level explanations, and an option to remove pre-padding in text input.

Did I mention “character-level”? Indeed I managed to resolve last blog’s issue of low accuracy for a character-based network. Solution? Increase batch size from 30 to 500. http://theorangeduck.com/page/neural-network-not-working and http://karpathy.github.io/2019/04/25/recipe/ gave some ideas. (This fix was quite random and I got lucky, so I still have the problem of knowing how to train neural nets!)

I also trained a network with multiple LSTM layers. Training on CPU, first time things didn’t go past epoch 1. My sequences were way too long, so the solution was to reduce their length. In my case I found that LSTM’s realistically only work for about 100 tokens.

An interesting issue regarding refactoring the text codebase was to do with how to handle duplicate function arguments. Specifically we have a dispatcher function and multiple concrete functions. These concrete functions have unique arguments, but also share some. See https://github.com/TeamHG-Memex/eli5/blob/453b0da382db2507972cf31bb25e68dae5674b57/eli5/keras/explain_prediction.py for an example. I ended up not using any **kwargs magic and typed out all the arguments in the dispatcher and concrete functions myself (readable, but need to watch when making changes).

I followed through the PyTorch 60 minute blitz tutorial (mostly just copy pasting code and looking up docs for some objects). I must admit that I didn’t understand autograd, and I will need to go through it more carefully for gradient calculations.

I went on to load a pretrained model from torchvision and the imagenet dataset (since the dataset is 150 GB, I loaded just a few local samples). I will use this set up when creating Grad-CAM code.

Workflow wise, I discovered that using the PYTHONPATH variable, for example running `PYTHONPATH=$PWD/SOMEPATH jupyter notebook` is a convenient way to make sure Python can find your local modules. This is much easier than relocating your scripts or modifying sys.path.

Work on first image PR has stopped. Improvements to it are now being done through the second text PR.

So what I could’ve done more productivity-wise? I wish I could’ve progressed the text PR more. It is in a very early WIP stage still, with no tests, docs, and sprinkled with TODO’s/FIXME’s. Regarding PyTorch I wish I could’ve coded some experimental Grad-CAM code. Overall it is hard to not get lazy when working without “checks”, like leaving a comment about what you are working on or syncing up in more detail during the week.

For next week we have a second evaluation. I heard that I will have to submit PR’s, so it would be good to do two things: open a PyTorch ‘WIP’ PR once I get some Grad-CAM code, and remove the ‘WIP’ label from the text PR.

That’s it for the week. Hope the next one is productive!

Tomas Baltrunas

Weekly check-in #6 (week 7): 08/07 to 14/07

tomasbaltrunas0@gmail.com (tomasb) — Sun, 14 Jul 2019 15:17:28 +0000

Hi! At the start of week 7 I have learned that one of my mentors will be going on holiday during the next two weeks. This means that during that time I will mostly work by myself. Excited to see what’s going to happen!

What did you do this week?

Since this was my mentor’s last week before holidays, we had sync up calls on Wednesday and also on Friday. We talked about how to explain RNN’s and character-based models. An interesting addition to the text codebase this week was resizing or “resampling” 1D arrays. I ended up using scipy.signal.

For the image PR, another mentor joined and left some comments for improvement, mostly about clarifying docs and outdated information (could this be automated or tested?).

Re: Workflow, I did a couple of things on Sunday, such as creating a symlink between my long GSOC path and simply ~/gsoc in my home directory, and removing a commit in the middle of other commits with git rebase.

What is coming up next?

I couldn't get down to doing some PyTorch this week, so I should definitely start with it next week.

It would be good to clean up the text branch (fix regressions, add tests, docs, tutorial) and make a second PR. The code should explain at least some models reasonably well, so I’ll need to train more RNN’s (with more than one LSTM layer) and a character-based model (see below for issues) for testing.

The image PR has more or less settled down. I will implement any suggestions, and once the PR gets merged I might add commits for a new release.

Did you get stuck anywhere?

The first stumbling block this week was the theory behind RNN’s. I read some articles about them, but I still don’t have an exact idea of how they work. And I didn’t even start with LSTM’s!

But the main obstacle this week was training a character-based network. Firstly, the text datasets in Keras (for example IMDB) are made for word tokens, not characters. I had to get the original IMDB texts and build a tokenizer. This took some time but it worked.

Training the network on CPU was so slow that I couldn’t get past one epoch! My mentor suggested picking a reasonable max length for the sequences - say the 99th or 95th percentile. He also suggested to train on GPU. Since I'm broke student that can’t afford a GPU with CUDA 3.5, I used Kaggle kernels.

The training worked but the network had an accuracy close to 50%. This is an issue yet to be resolved using some neural network troubleshooting techniques.

That’s the update of the week. Again I’m excited to work more independently in the next two weeks. Hopefully it won’t end up with no progress and all tests failing, and we’ll get some work done!

Tomas Baltrunas

Weekly blog #3 (week 6): 01/07 to 07/07

tomasbaltrunas0@gmail.com (tomasb) — Sun, 07 Jul 2019 19:15:33 +0000

Hello there. It’s time for a weekly blog. One thing to note right away is that half of GSoC is already gone (week 6 is over)! I really do hope to put in some solid work before everything finishes.

Let me start by saying what I did this week. Firstly, once again the PR got closer to being finished. Just to mention a few of the interesting changes: with the help of my mentor I added a check, though not perfect, for whether a network outputs ‘scores’ (unconstrained values) or ‘probabilities’ (values between 0 and 1). I also had to make one argument optional for a common object in the library, without breaking too many things and without too many corrections (mypy helped massively). An interesting addition to the Grad-CAM code was checking whether differentiation failed (checking whether gradients had any None values) and making an automated test for that.

Outside the PR, the mentors made a new ELI5 release, and I merged those changes into my fork.

Regarding text, I managed to add some code to ELI5 to get a working end-to-end text explanation example (trust me, you don’t want to see the code under the hood - a bunch of hardcoding and if statements!). Improvements were made.

Through the week I came across a few issues.

For example, at the start of Tuesday, just as I was about to do some productive work, I realised that me or Jupyter Notebook had not saved some of the work that I had done on Monday. Big shout out to IPython %history magic that let me recover the commands that I ran. I’ll CTRL+S all the time from now on!

I felt that a big issue this week was a lack of productivity, especially relating to ELI5 text code. Sometimes I found myself making random changes to code, without much planning. I was overwhelmed by how much there is to refactor and implement, and I still have to deal with the problem of reusing existing code from image explanations. A couple of things had helped:

Again make small changes and test. Don’t make changes “into the future” that you can’t test right now.
Create a simple list of TODO items in a text editor, like I did for image code before. This is great for seeing the big picture, taking tasks one-by-one instead of everything at once, and marking tasks as done.

Writing library code - code that interacts with existing code and code that may expose an API to users - is much harder than simple experimental snippets in a Jupyter Notebook. Organisation helps.

On Sunday I got down to adding virtualenvwrapper to my workflow. Goodbye “source big-relative-or-absolute-path-to-my-virtualenv/bin/activate” and hello “workon eli5gradcam”! I also tried to train a ConvNet on a multiple classification problem (reuters newstopics), but the model kept on getting an appalling 30% accuracy. Perhaps I need a minor modification to the hyperparameters, or maybe ConvNet’s are not so good for multiple classification and text tasks?

I believe that next week will be much the same as week 6, topic wise: small PR changes, library code for text, and experimentation to make the code cover more possible models (RNN’s will come - someday!).

Thank you for reading once again, and see you next time!

Tomas Baltrunas

Weekly check-in #5 (week 5): 24/06 to 30/06

tomasbaltrunas0@gmail.com (tomasb) — Mon, 01 Jul 2019 13:14:10 +0000

Hi all. I am running out of ideas for opening sentences, so let me just begin!

What did you do this week?

This week was evaluation, though that was very short and work continued as usual. On Monday and Wednesday I synced up with one of my mentors, discussing how we will incorporate text into existing ELI5 code, our next steps.

However, I wanted to get the PR out of the way, so most of what I did this week was resolving comments and opening new GitHub issues for larger TODO’s and FIXME’s. Finally all the checks went green!

Only on Sunday I got to do a bit of work on text, reusing existing ELI5 code to highlight text importance.

What is coming up next?

Now that I have a roughly working “end-to-end” example of text explanations in a Jupyter Notebook, I need to incorporate it into the ELI5 codebase (on a new branch). The example is indeed “roughly working”, so I will need to experiment and debug, and add explanations for more models (RNN’s come to mind).

Thinking about the next stage - what to do by the second evaluation - I have two top priority tasks in mind. First, explain text. Second, add PyTorch support alongside Keras. Roughly the same goals as in the proposed schedule, with some differences.

Did you get stuck anywhere?

I found myself working on the PR for too long, and I think it’s harder for me to switch between tasks than to work on a single task. From next week a good resolution would be to focus on text and spend less time on the PR.

Remembering all the things I did through the week and putting them on a blog in one day also tended to be hard. My solution was to write every couple of days. This did not happen, but I did manage to write the blog a day earlier (Sunday instead of Monday)!

When working on the “end-to-end” text explanation example, I found the ELI5 objects used for HTML formatting a bit mystic. Fortunately there was a tutorial showing highlighted text, and stepping through ELI5 code with pdb and examining the values helped.

That’s the check-in for the week. Continue on the work!

Tomas Baltrunas

Weekly blog #2 (week 4): 17/06 to 23/06

tomasbaltrunas0@gmail.com (tomasb) — Tue, 25 Jun 2019 13:57:29 +0000

Hi! Welcome to my second “blog post” in which I will talk about my work, problems, and solutions for week 4. Let’s jump into it...

As expected this week was a split between working on my existing Pull Request (Grad-CAM applied to images), and exploring how to apply Grad-CAM to text.

Starting with the Pull Request work, I continued resolving various mentors’ comments on it.

At one point I wanted to make a simple test case that checks whether a warning is raised when certain dependencies, when making a certain function call, are missing. My problem was that I spent way too much time on that. Trying to mock dependencies or do tricks with sys.modules is hard. Since the case was quite specialised and rare, the solution was to simply hack up a manual test. Lesson - use your time economically!

Another problem that I often met was to do with a Jupyter Notebook tutorial for the docs. Whenever I converted the notebook to .rst, I had to make some manual repetitive changes (changing path URI’s of image links). Thankfully in one of the GitHub comments a mentor pointed out that there is a script in the repo dedicated for such manual notebook conversions. Shout out to “sed” and “mv”!

Lastly on the PR, when I was making changes to the Grad-CAM gradient and mean calculator, I had to deal with the concept of “dataflow programming” in Tensorflow (backend of Keras). It was a bit tricky getting used to this, as I sometimes forgot that Tensorflow operations don’t actually “happen” until you call a function to “evaluate” the instructions. After some attempts I got the code to work.

Onto the exciting topic of the week - text! I managed to train a bunch of simple text models in a Jupyter Notebook, and then applied the same Grad-CAM steps to the text models as I used for images. In many cases the results were indeed disappointing, but for one case the explanation does work!

The first issue with text was padding. In some cases it’s useful to have each text sequence be the same length, padding the text with some “padding token” if it is too short. The problem is what to do with this padding once we get a Grad-CAM heatmap (showing the importance of each token for some prediction). My solution was to simply cut the heatmap at the point where padding starts. This has produced an actual working result, though with edge cases. Another issue yet to be resolved was applying Grad-CAM onto text without padding. My models, trained with padding, but given input without padding, were way off. Padding was a big discussion point with my mentor this week.

There are many different model architectures for text, which is another issue. As advised, to not make things more difficult for myself I decided to leave RNN’s, LSTM’s, etc. for later on, sticking to fully connected and Convolutional networks for now. Of course, there are issues, yet to be solved, with such architectures as well. Using a traditional model with Dense layers and a Global Average Pooling layer seemed to give no results at all.

So you can see that there were many technical issues regarding text. Things went slowly. Over the weekend I looked at a Deep Learning book to have some background for text, in topics such as tokenization, vectorization, embedding, etc. To train more models faster I also wanted to install Tensorflow for GPU, but my GPU (compute capability 3.0) is too old to match the current Tensorflow requirements (at least compute capability 3.5 for easiest installation). Google Collab and Kaggle kernels here I come!

I hope this was not too long and you were able to skim through it. Next week hopefully we’ll see an “end-to-end” example of a prediction about text being explained. I might even finish with the PR (I know that I have been saying this for the last two weeks...). Also, enjoy writing for Evaluation 1. Let us pass!

Tomas Baltrunas

Weekly check-in #4 (week 3): 10/06 to 16/06

tomasbaltrunas0@gmail.com (tomasb) — Mon, 17 Jun 2019 21:06:47 +0000

Hi! It’s almost a month into GSoC now.

What did you do this week?

In week 3 I submitted a Work-in Progress Pull Request (https://github.com/TeamHG-Memex/eli5/pull/315) for explaining Keras image classifiers. The rest of the week was making changes to this PR. One of the mentors added plenty of comments to the it so that kept me busy. I have changed many things: the tutorial, docstrings, exception handling, the API (function signatures). It’s maintenance time!

What is coming up next?

Finishing up the PR soon is definitely a priority, so that I can work on other things. There are a few issues left to resolve such as optimisation and managing optional dependencies for images. I need a few more tests for coverage to look good as well!

After that, I hope to get started with text and machine learning. The mentors shared some resources related to that, such as this book and tutorials from Tensorflow and Keras docs. First I will need to get familiar with the area, and only then start applying Grad-CAM.

Did you get stuck anywhere?

I was a bit stuck getting my docs build to work. In our Sphinx automatic documentation builder, when mocking external libraries I had to declare submodules, not just the top level modules, that I have used. The syntax for docstrings was weird too.

When making the tutorial I spent way too much time trying to find a unique picture from ImageNet. The site was often slow and after giving up I returned to the good old ‘cat_dog.jpeg’.

I wanted to work on new features this week, but I was always on the PR, so that was a blocker!

We have a week left before the first evaluation. Thanks for reading and keep up the work!

Tomas Baltrunas

Weekly blog #1 (week 2): 03/06 to 09/06

tomasbaltrunas0@gmail.com (tomasb) — Mon, 10 Jun 2019 21:48:25 +0000

As per the PSF calendar, for this week I will try to write a blog post instead of the usual check-in post. I will be answering three questions: what I am working on, what I struggled with, and what solutions I have come to.

First, a recap of week 1.

I started working with Grad-CAM for Keras and images. Just to explain that, say we have neural network that takes in an input such as an image, and gives an output, for example a category that tells you what is in the image. By using Grad-CAM, we can highlight the pixels in the image that helped the network decide on the category that it picked. We can check where the network “looks”.

Right away I struggled with implementing such “explanations”. Like any respectable student does I found a GitHub repo that contained all the work I needed to do and copy pasted it real fast. In the end this worked well, but my approach was not good. I started out by adding code function-by-function, making “optimisations” as I saw fit. Unfortunately I could not check if I have made any errors, and ended up with some exceptions that I could not resolve.

The solution to this was testing, then making small changes. It’s hard to take something that does not work and make changes to it. It’s much easier to take what works and change it, then check that nothing broke. I thank the mentors for advising me this.

Going forward to week 2, I added automated tests for what I have done in week 1, and as per a sync up call with a mentor made some optimisations to the Grad-CAM implementation.

There were a couple of issues I ran into. Firstly, testing image output is a problem in itself. There were only a few comments on this online, but I talked to my mentors and we agreed that doing a rough check (checking average values in a region) would be good (pixel-by-pixel checks are too fragile). This led to some “integration” tests.

Next, I found it hard to come up with a few “unit tests”. I clarified the API with the mentors and changed some function signatures, and that helped.

I think these were the main “issues” of week 2. It went by fast and I am much happier with the code now. Looking forward to learning about RNN’s and adapting Grad-CAM next week! But first I have to write some docs :(

See you next week,

Tomas Baltrunas

Weekly check-in #3 (week 1): 27/05 to 02/06

tomasbaltrunas0@gmail.com (tomasb) — Mon, 03 Jun 2019 10:56:32 +0000

Hey! So we are done with week 1 ...

What did you do this week?

Finishing this week, I can now feed Keras models and images to ELI5 and make it show visualisations of them. I have wrapped a working Grad-CAM implementation I found on GitHub, and made changes to it such as ability to choose a layer and prediction to do Grad-CAM on. Grad-CAM produces a “heatmap”, so I have added an “image formatter” that takes that heatmap and overlays it over the original image. During the week I caught up with my mentors, going through the Grad-CAM paper, and called with some Scrapy students to get to know each other.

What is coming up next?

Week 2 will be a split between two activities. First, I will do testing. It will be essential to add some automated tests using PyTest and tox. Some manual tests using different datasets and models would also be nice. Secondly, I will need to perform optimisations, refactorings, and improvements to the code added in week 1. One interesting task will be making Grad-CAM work beyond classification-based models, i.e. regression, etc. I hope that by the end of week 2 or 3 my branch will be in a good shape for a Pull Request.

Did you get stuck anywhere?

I had troubles with implementing/wrapping Grad-CAM itself (more on that in the upcoming weekly blog, but testing what works THEN changing things, and mentors’ advice, had certainly helped). A slight blocker was also low responsiveness of my machine when testing with large models in memory. Fortunately smaller models exist so I don’t always have to use VGG16!

1 down and 11 more to go! Thanks for reading!

Tomas Baltrunas

Weekly check-in #2: 21/05 to 26/05 (community bonding)

tomasbaltrunas0@gmail.com (tomasb) — Mon, 03 Jun 2019 10:55:15 +0000

Hi! This is a check-in for the last week of community bonding. I am writing this at the end of week 1, so I hope I still remember most of what I did!

What did you do this week?

Briefly tested out Tox and Sphinx on my machine. Made sure I can run all the environments specified in ELI5’s tox.ini: I had to apt-get some python-dev packages, and comment out the installation for xgboost 0.6a2 due to installation issues on Ubuntu. Set up a few virtualenv’s. Installed PyCharm IDE. Git clone’d existing Grad-CAM implementations for Keras and checked that they work using examples from ImageNet. Commented the source code by stepping through it with pdb. Looked at the authors’ implementation of Grad-CAM in Lua.

What is coming up next?

Week 1 tasks will begin! As scheduled I will implement Grad-CAM for images (working with Keras models). One change will be that I will make automated tests during week 2, not week 1. Instead of testing this week, I will add an image formatter so that I can get immediate feedback of the Grad-CAM algorithm implemented. There were some things that I planned to do during this week, but prioritised them for later, including: Enabling my GPU to use with ML libraries, learning about RNN’s, and setting up virtualenvwrapper/pyenv instead of plain virtualenv.

Did you get stuck anywhere?

At the end of the week I was stuck at understanding the details of Grad-CAM and how is it implemented in Keras. I wish I could’ve spent more time on learning actual Keras and doing some CNN visualisations.

Thanks for reading again!

Tomas Baltrunas

Weekly check-in #1: 13/05 to 20/05

tomasbaltrunas0@gmail.com (tomasb) — Mon, 20 May 2019 20:25:53 +0000

Hello everybody. Checking in for the first time under ELI5 (Scrapinghub), implementing Grad-CAM for neural networks.

What did you do this week?

Briefly looked into some auxiliary tools such as PyTest and mypy. Called with mentors for the first time. Discussed workflow and preparation details, such as when to call, create pull requests, and document code. Mentioned some things to do in the future - i.e. use Google Colab when need hardware to train a network, consider how to test image output, go through the technical Grad-CAM paper together. In the suborg's Slack, discussed how everyone should give updates.

What is coming up next?

Set up a recommended environment: PyCharm IDE, Jupyter notebooks for manual testing, one virtualenv for development and tox for testing on other environments. Enable recent libraries such as CUDA 9 on my local NVIDIA hardware. Install, use, and look over source code of existing Grad-CAM implementations. Look over and do the relevant parts of the recommended course at http://cs231n.stanford.edu/. Briefly learn tox (make sure the project's tox config works for me locally) and look into Sphinx very briefly.

Did you get stuck anywhere?

No specific issues yet, just overall a bit slow to start looking into the actual topic of the project- Grad-CAM.

Thank you for reading!

Tomas Baltrunas