Skip to main content

Delve lets you monitor PyTorch model layer saturation during training

Project description

Delve: Deep Live Visualization and Evaluation

PyPI version Build Status

Inspect layer saturation for optimizing your PyTorch models.

Delve is a Python package for visualizing deep learning model training.

Use Delve if you need a lightweight PyTorch extension that:

  • Plots live statistics of network activations to TensorBoard
  • Performs spectral analysis to identify layer saturation for network pruning
  • Is easily extendible and configurable

Motivation

Designing a deep neural network involves optimizing over a wide range of parameters and hyperparameters. Delve allows you to visualize your layer saturation during training so you can grow and shrink layers as needed.

Demo

live layer saturation demo

Getting Started

pip install delve

Layer Saturation

Pass a PyTorch model or Linear layers to CheckLayerSat:

from delve import CheckLayerSat

model = TwoLayerNet() # PyTorch network
stats = CheckLayerSat('runs', model) #logging directory and input

... # setup data loader

for i, data in enumerate(train_loader):    
    stats.saturation() # output saturation

Only fully-connected and convolutional layers are currently supported.

To log the saturation to console, call stats.saturation(). For example:

Regression - SixLayerNet - Hidden layer size 10loss=0.231825:  68%|████████████████████▎         | 1350/2000 [00:04<00:02, 289.30it/s]│
linear1:  90%|█████████████████████████████████▎   | 90.0/100 [00:00<00:00, 453.47it/s]│
linear2:  18%|██████▊                               | 18.0/100 [00:00<00:00, 90.68it/s]│
linear3:  32%|███████████▊                         | 32.0/100 [00:00<00:00, 161.22it/s]│
linear4:  32%|███████████▊                         | 32.0/100 [00:00<00:00, 161.24it/s]│
linear5:  28%|██████████▎                          | 28.0/100 [00:00<00:00, 141.11it/s]│
linear6:  90%|██████████████████████████████████▏   | 90.0/100 [00:01<00:00, 56.04it/s]

Optimize neural network topology

Ever wonder how big your fully-connected layers should be? Delve helps you visualize the effect of modifying the layer size on your layer saturation.

For example, see how modifying the hidden layer size of this network affects the second layer saturation but not the first. Multiple runs show that the fully-connected "linear2" layer (light blue is 256-wide and orange is 8-wide) saturation is sensitive to layer size:

saturation

saturation

Log spectral analysis

Writes the top 5 eigenvalues of each layer to TensorBoard summaries:

stats = CheckLayerSat('runs', layers, 'spectrum')

Other options spectrum

Intrinsic dimensionality

View the intrinsic dimensionality of models in realtime:

intrinsic_dimensionality-layer2

This comparison suggests that the 8-unit layer (light blue) is too saturated and that a larger layer is needed.

Why this name, Delve?

delve (verb):

  • reach inside a receptacle and search for something
  • to carry on intensive and thorough research for data, information, or the like

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Filename, size & hash SHA256 hash help File type Python version Upload date
delve-0.1.5.tar.gz (9.1 kB) Copy SHA256 hash SHA256 Source None

Supported by

Elastic Elastic Search Pingdom Pingdom Monitoring Google Google BigQuery Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN SignalFx SignalFx Supporter DigiCert DigiCert EV certificate StatusPage StatusPage Status page