A toolbox to innvestigate neural networks' predictions.
Project description
iNNvestigate neural networks!
Table of contents
Introduction
In the recent years neural networks furthered the state of the art in many domains like, e.g., object detection and speech recognition. Despite the success neural networks are typically still treated as black boxes. Their internal workings are not fully understood and the basis for their predictions is unclear. In the attempt to understand neural networks better several methods were proposed, e.g., Saliency, Deconvnet, GuidedBackprop, SmoothGrad, IntergratedGradients, LRP, PatternNet&Attribution. Due to the lack of a reference implementations comparing them is a major effort. This library addresses this by providing a common interface and outofthebox implementation for many analysis methods. Our goal is to make analyzing neural networks' predictions easy!
If you use this code please star the repository and cite the following paper:
@article{JMLR:v20:18540,
author = {Maximilian Alber and Sebastian Lapuschkin and Philipp Seegerer and Miriam H{{\"a}}gele and Kristof T. Sch{{\"u}}tt and Gr{{\'e}}goire Montavon and Wojciech Samek and KlausRobert M{{\"u}}ller and Sven D{{\"a}}hne and PieterJan Kindermans},
title = {iNNvestigate Neural Networks!},
journal = {Journal of Machine Learning Research},
year = {2019},
volume = {20},
number = {93},
pages = {18},
url = {http://jmlr.org/papers/v20/18540.html}
}
Installation
iNNvestigate can be installed with the following commands. The library is based on Keras and therefore requires a supported Kerasbackend o(Currently only the Tensorflow backend is supported. We test with Python 3.6, Tensorflow 1.12 and Cuda 9.x.):
pip install innvestigate # Installing Keras backend pip install [tensorflow  theano  cntk]
To use the example scripts and notebooks one additionally needs to install the package matplotlib:
pip install matplotlib
The library's tests can be executed via:
git clone https://github.com/albermax/innvestigate.git cd innvestigate python setup.py test
Usage and Examples
The iNNvestigate library contains implementations for the following methods:
 function:
 gradient: The gradient of the output neuron with respect to the input.
 smoothgrad: SmoothGrad averages the gradient over number of inputs with added noise.
 signal:
 deconvnet: DeConvNet applies a ReLU in the gradient computation instead of the gradient of a ReLU.
 guided: Guided BackProp applies a ReLU in the gradient computation additionally to the gradient of a ReLU.
 pattern.net: PatternNet estimates the input signal of the output neuron.
 attribution:
 input_t_gradient: Input * Gradient
 deep_taylor[.bounded]: DeepTaylor computes for each neuron a rootpoint, that is close to the input, but which's output value is 0, and uses this difference to estimate the attribution of each neuron recursively.
 pattern.attribution: PatternAttribution applies Deep Taylor by searching rootpoints along the singal direction of each neuron.
 lrp.*: LRP attributes recursively to each neuron's input relevance proportional to its contribution of the neuron output.
 integrated_gradients: IntegratedGradients integrates the gradient along a path from the input to a reference.
 deeplift.wrapper: DeepLIFT (wrapper around original code, slower) computes a backpropagation based on "finite" gradients.
 miscellaneous:
 input: Returns the input.
 random: Returns random Gaussian noise.
The intention behind iNNvestigate is to make it easy to use analysis methods, but it is not to explain the underlying concepts and assumptions. Please, read the according publication(s) when using a certain method and when publishing please cite the according paper(s) (as well as the iNNvestigate paper). Thank you!
All the available methods have in common that they try to analyze the output of a specific neuron with respect to input to the neural network. Typically one analyses the neuron with the largest activation in the output layer. For example, given a Keras model, one can create a 'gradient' analyzer:
import innvestigate model = create_keras_model() analyzer = innvestigate.create_analyzer("gradient", model)
and analyze the influence of the neural network's input on the output neuron by:
analysis = analyzer.analyze(inputs)
To analyze a neuron with the index i, one can use the following scheme:
analyzer = innvestigate.create_analyzer("gradient", model, neuron_selection_mode="index") analysis = analyzer.analyze(inputs, i)
Let's look at an example (code) with VGG16 and this image:
import innvestigate import innvestigate.utils import keras.applications.vgg16 as vgg16 # Get model model, preprocess = vgg16.VGG16(), vgg16.preprocess_input # Strip softmax layer model = innvestigate.utils.model_wo_softmax(model) # Create analyzer analyzer = innvestigate.create_analyzer("deep_taylor", model) # Add batch axis and preprocess x = preprocess(image[None]) # Apply analyzer w.r.t. maximum activated outputneuron a = analyzer.analyze(x) # Aggregate along color channels and normalize to [1, 1] a = a.sum(axis=np.argmax(np.asarray(a.shape) == 3)) a /= np.max(np.abs(a)) # Plot plt.imshow(a[0], cmap="seismic", clim=(1, 1))
Trainable methods
Some methods like PatternNet and PatternAttribution are dataspecific and need to be trained. Given a data set with train and test data, this can be done in the following way:
import innvestigate analyzer = innvestigate.create_analyzer("pattern.net", model) analyzer.fit(X_train) analysis = analyzer.analyze(X_test)
Tutorials
In the directory examples one can find different examples as Python scripts and as Jupyter notebooks:
 Introduction to iNNvestigate: shows how to use iNNvestigate.
 Comparing methods on MNIST: shows how to train and compare analyzers on MNIST.
 Comparing output neurons on MNIST: shows how to analyze the prediction of different classes on MNIST.
 Comparing methods on ImageNet: shows how to compare analyzers on ImageNet.
 Comparing networks on ImageNet: shows how to compare analyzes for different networks on ImageNet.
 Sentiment Analysis.
 Development with iNNvestigate: shows how to develop with iNNvestigate.
 Perturbation Analysis.
To use the ImageNet examples please download the example images first (script).
More documentation
... can be found here:
 Alber, M., Lapuschkin, S., Seegerer, P., Hägele, M., Schütt, K. T., Montavon, G., Samek, W., Müller, K. R., Dähne, S., & Kindermans, P. J. (2019). INNvestigate neural networks! Journal of Machine Learning Research, 20.](https://jmlr.org/papers/v20/18540.html)
@article{JMLR:v20:18540, author = {Maximilian Alber and Sebastian Lapuschkin and Philipp Seegerer and Miriam H{{\"a}}gele and Kristof T. Sch{{\"u}}tt and Gr{{\'e}}goire Montavon and Wojciech Samek and KlausRobert M{{\"u}}ller and Sven D{{\"a}}hne and PieterJan Kindermans}, title = {iNNvestigate Neural Networks!}, journal = {Journal of Machine Learning Research}, year = {2019}, volume = {20}, number = {93}, pages = {18}, url = {http://jmlr.org/papers/v20/18540.html} }
 https://innvestigate.readthedocs.io/en/latest/
Contributing
If you would like to contribute or add your analysis method please open an issue or submit a pull request.
Releases
Project details
Release history Release notifications  RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Filename, size  File type  Python version  Upload date  Hashes 

Filename, size innvestigate1.0.9py3noneany.whl (100.6 kB)  File type Wheel  Python version py3  Upload date  Hashes View 
Filename, size innvestigate1.0.9.tar.gz (80.0 kB)  File type Source  Python version None  Upload date  Hashes View 
Hashes for innvestigate1.0.9py3noneany.whl
Algorithm  Hash digest  

SHA256  6bd93c8ba4ca50900ea5683d0f811aab8412ce263cf7d91c43e6b85fe2c97594 

MD5  211ba3dc97a14b7dd6d6a94102c37fd3 

BLAKE2256  555fd8c1c0136ddaf0720cb663edf6c2361f54d499283b813c1823e5343aff81 