Visualization toolkit for neural networks in PyTorch
Project description
FlashTorch
Visualizaion toolkit implemented in PyTorch for inspecting what neural networks learn in image recognition tasks (feature visualizaion).
The project is very much work in progress, and I would appreciate your feedback!
It currently supports visualizaion of saliency maps for all the models available under torchvision.models.
Overview
Installation
If you are installing flashtorch
for the first time:
$ pip install flashtorch
Or if you are upgrading it:
$ pip install flashtorch -U
API guide
An API guide is under construction, so this is a temporary workaround.
These are currently available modules.
flashtorch.utils
: some useful utility functions for data handling & transformationflashtorch.utils.imagenet
:ImageNetIndex
class for easy-ish retrieval of class indexflashtorch.saliency.backprop
:Backprop
class for calculating gradientsflashtorch.activmax.gradient_ascent
:GradientAscent
class for activation maximization
You can inspect each module with Python built-in function help
. The output of that is available on Quick API Guide for your convenience.
How to use
Here are some handy notebooks showing examples of using flashtorch
.
Image handling
Notebook: Image handling
Saliency maps
Saliency in human visual perception is a subjective quality that makes certain things within the field of view stand out from the rest and grabs our attention.
Saliency maps in computer vision provide indications of the most salient regions within images. By creating a saliency map for neural networks, we can gain some intuition on "where the network is paying the most attention to" in an imput image.
Using flashtorch.saliency
module, let's visualise image-specific class saliency maps of AlexNet pre-trained on ImageNet classification tasks.
Great gray owl (class index 24): The network is focusing on the sunken eyes and the round head for this owl.
Refer to the notebooks below for more examples:
- Image-specific class saliency map with backpropagation
- Google Colab version: best for playing around
Activation maximization
Activation maximization is one form of feature visualization that allows us to visualize what CNN filters are "looking for", by applying each filter to an input image and updating the input image so as to maximize the activation of the filter of interest (i.e. treating it as a gradient ascent task with filter activation values as the loss).
Using flashtorch.activmax
module, let's visualise images optimized with filters
from VGG16 pre-trained on ImageNet classification tasks.
We can see that, in the earlier layers (conv1_2, conv2_1), filters get activated by colors and simple patterns such as virtical, horisontal and diagonal lines. In the intermediate layers (conv3_1, conv4_1), we start to see more complex patterns. Then concepts such as 'eyes' (filter 45) and 'entrances (?)' (filter 271) seem to appear in the last layer (conv5_1).
Refer to the notebooks below for more examples:
- Activation maximization
- Google Colab version: best for playing around
Talks & blog posts
-
Hopperx1 London, June 2019 - slide deck
Papers
-
Introduction and overview of feature visualizaion: Feature Visualization
-
The latest development in feature visualizaion: Exploring Neural Networks with Activation Atlases
-
Using backpropagation for gradient visualizaion: Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps
-
Guided backprobagation: Striving for Simplicity: The All Convolutional Net
Inspiration
-
pytorch-cnn-visualizations by utkuozbulak
-
keras-vis by raghakot
Citation
Misa Ogura. (2019, July 8). MisaOgura/flashtorch: 0.0.8 (Version v0.0.8). Zenodo. http://doi.org/10.5281/zenodo.3271410
Author
Misa Ogura
R&D Software Engineer @ BBC
Co-founder of Women Driven Development
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.