Delve lets you monitor PyTorch model layer saturation during training
Delve: Deep Live Visualization and Evaluation
Delve is a Python package for visualizing deep learning model training.
Use Delve if you need a lightweight PyTorch or Keras extension that:
- Plots live statistics of network layer inputs to TensorBoard or terminal
- Performs spectral analysis to identify layer saturation for network pruning
- Is easily extendible and configurable
Designing a deep neural network involves optimizing over a wide range of parameters and hyperparameters. Delve allows you to visualize your layer saturation during training so you can grow and shrink layers as needed.
pip install delve
delve.CheckLayerSat can be configured as follows:
savefile (str) : destination for summaries, depending on the saving strategy, this may be a directory or file save_to (str) : specifies the saving strategy supported writers are: console : print the stats to console everytime save() is called tensorboard : logs everything in tensorboard format, in this case the savefile must be a directory csv : creates a csv file with each column corresponding to a logged variable. Everytime save() is called a new line in the file is created layerwise_sat (bool) : toggles if layerwise sautration should be saved by the writer average_sat (bool) : toggles if average saturation should be saved by the writer ignore_layer_names (list) : a list of layer names, as specified in the modules. The layers specified will be excluded in the computation. Usefull for excluding layers which are force into a speciic saturation like softmaxes or other output layers. include_conv (bool) : toggle if convolutional layers should be included conv_method (str) : the method used to pool the latent space of convolutional layers. Default is 'median", valid inputs 'median', 'mean' and 'max' sat_threshold (float) : the saturation theshold for computing the dimensionality of the latent representations. Default is .99. This value may be any floating point in 0 and 1. modules (torch modules or list of modules) : layer-containing object (may contain submodules) log_interval (int) : steps between writing summaries stats (list of str): list of stats to collect supported stats are: lsat : layer saturation conv_method : Method for calculating saturation. Use `cumvar99``, or `all`. See https://github.com/justinshenk/playground for a comparison of how they work. include_conv : bool, setting to False includes only linear layers verbose (bool) : print saturation for every layer during training
Pass either a PyTorch model or
torch.nn.Linear layers to
from delve import CheckLayerSat model = TwoLayerNet() # PyTorch network stats = CheckLayerSat('runs', model) #logging directory and input ... # setup data loader for i, data in enumerate(train_loader): stats.saturation() # output saturation
Only fully-connected and convolutional layers are currently supported.
To log the saturation to console, call
stats.saturation(). For example:
Regression - SixLayerNet - Hidden layer size 10 │ loss=0.231825: 68%|████████████████████▎ | 1350/2000 [00:04<00:02, 289.30it/s]│ linear1: 90%|█████████████████████████████████▎ | 90.0/100 [00:00<00:00, 453.47it/s]│ linear2: 18%|██████▊ | 18.0/100 [00:00<00:00, 90.68it/s]│ linear3: 32%|███████████▊ | 32.0/100 [00:00<00:00, 161.22it/s]│ linear4: 32%|███████████▊ | 32.0/100 [00:00<00:00, 161.24it/s]│ linear5: 28%|██████████▎ | 28.0/100 [00:00<00:00, 141.11it/s]│ linear6: 90%|██████████████████████████████████▏ | 90.0/100 [00:01<00:00, 56.04it/s]
Two classes are provided in
CustomTensorBoard takes two parameters:
||location for writing summaries|
||frequency for writing summaries|
SaturationLogger contains two parameters:
||data for passing through the model|
||frequency for printing|
from delve.kerascallback import CustomTensorBoard, SaturationLogger ... # Tensorboard logging tbCallBack = CustomTensorBoard(log_dir='./runs', user_defined_freq=1) # Console logging saturation_logger = SaturationLogger(model, input_data=input_x_train[:2], print_freq=1) ... # Add callback to Keras `fit` method model.fit(x_train, y_train, epochs=100, batch_size=128, callbacks=[saturation_logger]) # can also pass tbCallBack
Epoch 29/100 128/1000 [==>...........................] - ETA: 0s - loss: 2.2783 - acc: 0.1406 dense_1 : %0.83 | dense_2 : %0.79 | dense_3 : %0.67 |
Optimize neural network topology
Ever wonder how big your fully-connected layers should be? Delve helps you visualize the effect of modifying the layer size on your layer saturation.
For example, see how modifying the hidden layer size of this network affects the second layer saturation but not the first. Multiple runs show that the fully-connected "linear2" layer (light blue is 256-wide and orange is 8-wide) saturation is sensitive to layer size:
Log spectral analysis
Writes the top 5 eigenvalues of each layer to TensorBoard summaries:
# PyTorch-only stats = CheckLayerSat('runs', layers, 'spectrum')
View the intrinsic dimensionality of models in realtime:
This comparison suggests that the 8-unit layer (light blue) is too saturated and that a larger layer is needed.
Why this name, Delve?
- reach inside a receptacle and search for something
- to carry on intensive and thorough research for data, information, or the like
Release history Release notifications | RSS feed
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
|Filename, size||File type||Python version||Upload date||Hashes|
|Filename, size delve-0.1.41.tar.gz (25.9 kB)||File type Source||Python version None||Upload date||Hashes View|