Building blocks for Continual Inference Networks in PyTorch
Project description
Building blocks for Continual Inference Networks in PyTorch
Install
pip install continual-inference
Simple example
Continual Modules are a weight-compatible drop-in replacement for torch.nn Modules, with the enhanced capability of efficient continual inference.
import torch
import continual as co
# B, C, T, H, W
example = torch.randn((1, 1, 5, 3, 3))
conv = co.Conv3d(in_channels=1, out_channels=1, kernel_size=(3, 3, 3))
# Same exact computation as torch.nn.Conv3d ✅
output = conv(example)
# But can also perform online inference efficiently 🚀
firsts = conv.forward_steps(example[:, :, :4])
last = conv.forward_step(example[:, :, 4])
assert torch.allclose(output[:, :, : conv.delay], firsts)
assert torch.allclose(output[:, :, conv.delay], last)
See also the "Advanced Examples" section.
Continual Inference Networks (CINs)
Continual Inference Networks are a type of neural network, which operate on a continual input stream of data and infer a new prediction for each new time-step. They are ideal for online detection and monitoring scenarios, but can also be used succesfully in offline situations.
All networks and network-modules, that do not utilise temporal information can be used for an Continual Inference Network (e.g. nn.Conv1d
and nn.Conv2d
on spatial data such as an image).
Moreover, recurrent modules (e.g. LSTM
and GRU
), that summarize past events in an internal state are also useable in CINs.
Some example CINs and non-CINs are illustrated below to
CIN:
O O O (output)
↑ ↑ ↑
nn.LSTM nn.LSTM nn.LSTM (temporal LSTM)
↑ ↑ ↑
nn.Conv2D nn.Conv2D nn.Conv2D (spatial 2D conv)
↑ ↑ ↑
I I I (input frame)
However, modules that operate on temporal data with the assumption that the more temporal context is available than the current frame (e.g. the spatio-temporal nn.Conv3d
used by many SotA video recognition models) cannot be directly applied.
Not CIN:
Θ (output)
↑
nn.Conv3D (spatio-temporal 3D conv)
↑
----------------- (concatenate frames to clip)
↑ ↑ ↑
I I I (input frame)
Sometimes, though, the computations in such modules, can be cleverly restructured to work for online inference as well!
CIN:
O O Θ (output)
↑ ↑ ↑
co.Conv3d co.Conv3d co.Conv3d (continual spatio-temporal 3D conv)
↑ ↑ ↑
I I I (input frame)
Here, the ϴ
output of the Conv3D
and ConvCo3D
are identical! ✨
The last conversion from a non-CIN to a CIN is possible due to a recent break-through in Online Action Detection, namely [Continual Convolutions].
Continual Convolutions
Below, principle sketches are shown, which compare regular and continual convolutions during online / continual inference:
Regular Convolution. A regular temporal convolutional layer leads to redundant computations during online processing of video clips, as illustrated by the repeated convolution of inputs (green b,c,d) with a kernel (blue α,β) in the temporal dimen- sion. Moreover, prior inputs (b,c,d) must be stored between time-steps for online processing tasks.
Continual Convolution. An input (green d or e) is convolved with a kernel (blue α, β). The intermediary feature-maps corresponding to all but the last temporal position are stored, while the last feature map and prior memory are summed to produce the resulting output. For a continual stream of inputs, Continual Convolutions produce identical outputs to regular convolutions.
As illustrated, Continual Convolutions can lead to major improvements in computational efficiency when online / frame-by-frame predictions are required! 🚀
For more information, we refer to the seminal paper on Continual Convolutions.
Forward modes
The library components feature three distinct forward modes, which are handy for different situations.
forward_step
Performs a forward computation for a single frame and continual states are updated accordingly. This is the mode to use for continual inference.
O+S O+S O+S O+S (O: output, S: updated internal state)
↑ ↑ ↑ ↑
N N N N (N: nework module)
↑ ↑ ↑ ↑
I I I I (I: input frame)
forward_steps
Performs a layer-wise forward computation using the continual module.
The computation is performed frame-by-frame and continual states are updated accordingly.
The output-input mapping the exact same as that of a regular module.
This mode is handy for initialising the network on a whole clip (multipleframes) before the forward
is usead for continual inference.
O (O: output)
↑
----------------- (-: aggregation)
O O+S O+S O+S O (O: output, S: updated internal state)
↑ ↑ ↑ ↑ ↑
N N N N N (N: nework module)
↑ ↑ ↑ ↑ ↑
P I I I P (I: input frame, P: padding)
forward
Performs a full forward computation exactly as the regular layer would. This method is handy for effient training on clip-based data.
O (O: output)
↑
N (N: nework module)
↑
----------------- (-: aggregation)
P I I I P (I: input frame, P: padding)
Modules
The repository contains custom online inference-friendly versions of common network building blocks, as well as handy wrappers and a global conversion function from torch.nn
to continual
(co
) modules.
-
Convolutions:
co.Conv1d
co.Conv2d
co.Conv3d
-
Pooling:
co.AvgPool1d
co.AvgPool2d
co.AvgPool3d
co.MaxPool1d
co.MaxPool2d
co.MaxPool3d
co.AdaptiveAvgPool1d
co.AdaptiveAvgPool2d
co.AdaptiveAvgPool3d
co.AdaptiveMaxPool1d
co.AdaptiveMaxPool2d
co.AdaptiveMaxPool3d
-
Containers
co.Sequential
- Sequential wrapper for modules. This module automatically performs conversions of torch.nn modules, which are safe during continual inference. These include all batch normalisation and activation function.co.Parallel
- Parallel wrapper for modules.co.Residual
- Residual wrapper for modules.co.Delay
- Pure delay module (e.g. needed in residuals).
-
Converters
co.continual
- conversion function from non-continual modules to continual modulesco.forward_stepping
- functional wrapper, which enhances temporally local non-continual modules with the forward_stepping functions
Advanced examples
Continual 3D MBConv
MobileNetV2 Inverted residual block. Source: https://arxiv.org/pdf/1801.04381.pdf
import continual as co
from torch import nn
mb_conv = co.Residual(
co.Sequential(
co.Conv3d(32, 64, kernel_size=(1, 1, 1)),
nn.BatchNorm3d(64),
nn.ReLU6(),
co.Conv3d(64, 64, kernel_size=(3, 3, 3), padding=(0, 1, 1), groups=64),
nn.ReLU6(),
co.Conv3d(64, 32, kernel_size=(1, 1, 1)),
nn.BatchNorm3d(32),
)
)
Continual 3D Inception module
Inception module with dimension reductions. Source: https://arxiv.org/pdf/1409.4842v1.pdf
import continual as co
from torch import nn
def norm_relu(module, channels):
return co.Sequential(
module,
nn.BatchNorm3d(channels),
nn.ReLU(),
)
inception_module = co.Parallel(
co.Conv3d(192, 64, kernel_size=1),
co.Sequential(
norm_relu(co.Conv3d(192, 96, kernel_size=1), 96),
norm_relu(co.Conv3d(96, 128, kernel_size=3, padding=1), 128),
),
co.Sequential(
norm_relu(co.Conv3d(192, 16, kernel_size=1), 16),
norm_relu(co.Conv3d(16, 32, kernel_size=3, padding=1), 32),
),
co.Sequential(
co.MaxPool3d(kernel_size=3, padding=1, stride=1),
norm_relu(co.Conv3d(192, 32, kernel_size=1), 32),
),
aggregation_fn="concat",
)
For additional full-fledged examples of complex Continual Inference Networks, see:
Compatibility
The library modules are built to integrate seamlessly with other PyTorch projects. Specifically, extra care was taken to ensure out-of-the-box compatibility with:
Citations
This library
@article{hedegaard2021colib,
title={Continual Inference Library},
author={Lukas Hedegaard},
journal={GitHub. Note: https://github.com/LukasHedegaard/continual-inference},
year={2021}
}
@article{hedegaard2021co3d,
title={Continual 3D Convolutional Neural Networks for Real-time Processing of Videos},
author={Lukas Hedegaard and Alexandros Iosifidis},
journal={preprint, arXiv:2106.00050},
year={2021}
}
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for continual-inference-0.2.2.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | 025abdb77a1c6aa45ba714a440e775f6317937409696b637083e9942ab34af58 |
|
MD5 | 0ec5d495352bf9521b7ef3c888b5cfac |
|
BLAKE2b-256 | 762ef156d18850eb60770ab512f7340c694e00ceff2713cb0f8e516fa925d0a0 |
Hashes for continual_inference-0.2.2-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 2b98c11bdc1ad27da19d10cde7726cb04f8a07db9acf46acb924be3d5ec5f6dd |
|
MD5 | b63ebe11e74d526300263f9246a7fcff |
|
BLAKE2b-256 | 7fd7a1425ceb1502ab99cf4edf692e6ba5234283f2248eff9403852393d55ea7 |