flashtorch 0.1.0

Visualization toolkit for neural networks in PyTorch

FlashTorch

Visualizaion toolkit implemented in PyTorch for inspecting what neural networks learn in image recognition tasks (feature visualizaion).

The project is very much work in progress, and I would appreciate your feedback!

It currently supports visualizaion of saliency maps for all the models available under torchvision.models.

Overview

• Installation
• API guide
• How to use
  • Image handling
  • Saliency maps
  • Activation maximization
• Talks & blog posts
• Papers
• Inspiration
• Citation
• Author

Installation

If you are installing flashtorch for the first time:

$ pip install flashtorch

Or if you are upgrading it:

$ pip install flashtorch -U

API guide

An API guide is under construction, so this is a temporary workaround.

These are currently available modules.

• flashtorch.utils: some useful utility functions for data handling & transformation
• flashtorch.utils.imagenet: ImageNetIndex class for easy-ish retrieval of class index
• flashtorch.saliency.backprop: Backprop class for calculating gradients
• flashtorch.activmax.gradient_ascent: GradientAscent class for activation maximization

You can inspect each module with Python built-in function help. The output of that is available on Quick API Guide for your convenience.

How to use

Here are some handy notebooks showing examples of using flashtorch.

Image handling

Notebook: Image handling

Saliency maps

Saliency in human visual perception is a subjective quality that makes certain things within the field of view stand out from the rest and grabs our attention.

Saliency maps in computer vision provide indications of the most salient regions within images. By creating a saliency map for neural networks, we can gain some intuition on "where the network is paying the most attention to" in an imput image.

Using flashtorch.saliency module, let's visualise image-specific class saliency maps of AlexNet pre-trained on ImageNet classification tasks.

Great gray owl (class index 24): The network is focusing on the sunken eyes and the round head for this owl.

Refer to the notebooks below for more examples:

• Image-specific class saliency map with backpropagation
• Google Colab version: best for playing around

Activation maximization

Activation maximization is one form of feature visualization that allows us to visualize what CNN filters are "looking for", by applying each filter to an input image and updating the input image so as to maximize the activation of the filter of interest (i.e. treating it as a gradient ascent task with filter activation values as the loss).

Using flashtorch.activmax module, let's visualise images optimized with filters from VGG16 pre-trained on ImageNet classification tasks.

We can see that, in the earlier layers (conv1_2, conv2_1), filters get activated by colors and simple patterns such as virtical, horisontal and diagonal lines. In the intermediate layers (conv3_1, conv4_1), we start to see more complex patterns. Then concepts such as 'eyes' (filter 45) and 'entrances (?)' (filter 271) seem to appear in the last layer (conv5_1).

Refer to the notebooks below for more examples:

• Activation maximization
• Google Colab version: best for playing around

Talks & blog posts

• Hopperx1 London, June 2019 - slide deck

• Uncovering what neural nets “see” with FlashTorch

• Gaining insights on transfer learning with FlashTorch

Papers

• Introduction and overview of feature visualizaion: Feature Visualization

• The latest development in feature visualizaion: Exploring Neural Networks with Activation Atlases

• Using backpropagation for gradient visualizaion: Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps

• Guided backprobagation: Striving for Simplicity: The All Convolutional Net

Inspiration

• pytorch-cnn-visualizations by utkuozbulak

• keras-vis by raghakot

Citation

Misa Ogura. (2019, July 8). MisaOgura/flashtorch: 0.0.8 (Version v0.0.8). Zenodo. http://doi.org/10.5281/zenodo.3271410

Author

Misa Ogura

R&D Software Engineer @ BBC

Co-founder of Women Driven Development

Github | Medium | twitter | LinkedIn