Input shape inference and SOTA custom layers for PyTorch.
Project description
Version | Docs | Tests | Coverage | Style | PyPI | Python | PyTorch | Docker |
---|---|---|---|---|---|---|---|---|
torchlayers is a library based on PyTorch
providing automatic shape and dimensionality inference of torch.nn
layers + additional
building blocks featured in current SOTA architectures (e.g. Efficient-Net).
Above requires no user intervention (except single call to torchlayers.build
)
similarly to the one seen in Keras.
Main functionalities:
- Shape inference for most of
torch.nn
module (convolutional, recurrent, transformer, attention and linear layers) - Dimensionality inference (e.g.
torchlayers.Conv
working astorch.nn.Conv1d/2d/3d
based oninput shape
) - Shape inference of custom modules (see examples section)
- Additional Keras-like layers (e.g.
torchlayers.Reshape
ortorchlayers.StandardNormalNoise
) - Additional SOTA layers mostly from ImageNet competitions (e.g. PolyNet, Squeeze-And-Excitation, StochasticDepth)
- Useful defaults (
"same"
padding and defaultkernel_size=3
forConv
, dropout rates etc.) - Zero overhead and torchscript support
Examples
For full functionality please check torchlayers documentation. Below examples should introduce all necessary concepts you should know.
Simple convolutional image and text classifier
- We will use single "model" for both tasks.
Firstly let's define it using
torch.nn
andtorchlayers
:
import torch
import torchlayers
# torch.nn and torchlayers can be mixed easily
model = torch.nn.Sequential(
torchlayers.Conv(64), # specify ONLY out_channels
torch.nn.ReLU(), # use torch.nn wherever you wish
torchlayers.BatchNorm(), # BatchNormNd inferred from input
torchlayers.Conv(128), # Default kernel_size equal to 3
torchlayers.ReLU(),
torchlayers.Conv(256, kernel_size=11), # "same" padding as default
torchlayers.GlobalMaxPool(), # Known from Keras
torchlayers.Linear(10), # Output for 10 classes
)
print(model)
Above would give you model's summary like this (notice question marks for not yet inferred values):
Sequential(
(0): Conv(in_channels=?, out_channels=64, kernel_size=3, stride=1, padding=same, dilation=1, groups=1, bias=True, padding_mode=zeros)
(1): ReLU()
(2): BatchNorm(num_features=?, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(3): Conv(in_channels=?, out_channels=128, kernel_size=3, stride=1, padding=same, dilation=1, groups=1, bias=True, padding_mode=zeros)
(4): ReLU()
(5): Conv(in_channels=?, out_channels=256, kernel_size=11, stride=1, padding=same, dilation=1, groups=1, bias=True, padding_mode=zeros)
(6): GlobalMaxPool()
(7): Linear(in_features=?, out_features=10, bias=True)
)
- Now you can build/instantiate your model with example input (in this case MNIST-like):
mnist_model = torchlayers.build(model, torch.randn(1, 3, 28, 28))
- Or if it's text classification you are after, same model could be built with different
input shape
(e.g. for text classification using300
dimensional pretrained embedding):
# [batch, embedding, timesteps], first dimension > 1 for BatchNorm1d to work
text_model = torchlayers.build(model, torch.randn(2, 300, 1))
- Finally, you can
print
both models after instantiation, provided below side by-side for readability (notice different dimenstionality, e.g.Conv2d
vsConv1d
aftertorchlayers.build
):
# MNIST CLASSIFIER TEXT CLASSIFIER
Sequential( Sequential(
(0): Conv1d(300, 64) (0): Conv2d(3, 64)
(1): ReLU() (1): ReLU()
(2): BatchNorm1d(64) (2): BatchNorm2d(64)
(3): Conv1d(64, 128) (3): Conv2d(64, 128)
(4): ReLU() (4): ReLU()
(5): Conv1d(128, 256) (5): Conv2d(128, 256)
(6): GlobalMaxPool() (6): GlobalMaxPool()
(7): Linear(256, 10) (7): Linear(256, 10)
) )
As you can see both modules "compiled" into original pytorch
layers.
Custom modules with shape inference capabilities
User can define any module and make it shape inferable with torchlayers.infer
function:
# Class defined with in_features
# It might be a good practice to use _ prefix and Impl as postfix
# to differentiate from shape inferable version
class _MyLinearImpl(torch.nn.Module):
def __init__(self, in_features: int, out_features: int):
super().__init__()
self.weight = torch.nn.Parameter(torch.randn(out_features, in_features))
self.bias = torch.nn.Parameter(torch.randn(out_features))
def forward(self, inputs):
return torch.nn.functional.linear(inputs, self.weight, self.bias)
MyLinear = torchlayers.infer(_MyLinearImpl)
# Build and use just like any other layer in this library
layer =torchlayers.build(MyLinear(out_features=32), torch.randn(1, 64))
layer(torch.randn(1, 64))
By default inputs.shape[1]
will be used as in_features
value
during initial forward
pass. If you wish to use different index
(e.g. to infer using
inputs.shape[3]
) use MyLayer = torchlayers.infer(_MyLayerImpl, index=3)
as a decorator.
Autoencoder with inverted residual bottleneck and pixel shuffle
Please check code comments and documentation if needed. If you are unsure what autoencoder is you could see this example blog post.
Below is a convolutional denoising autoencoder example for ImageNet
-like images.
Think of it like a demonstration of capabilities of different layers
and building blocks provided by torchlayers
.
# Input - 3 x 256 x 256 for ImageNet reconstruction
class AutoEncoder(torch.nn.Module):
def __init__(self):
super().__init__()
self.encoder = torchlayers.Sequential(
torchlayers.StandardNormalNoise(), # Apply noise to input images
torchlayers.Conv(64, kernel_size=7),
torchlayers.activations.Swish(), # Direct access to module .activations
torchlayers.InvertedResidualBottleneck(squeeze_excitation=False),
torchlayers.AvgPool(), # shape 64 x 128 x 128, kernel_size=2 by default
torchlayers.HardSwish(), # Access simply through torchlayers
torchlayers.SeparableConv(128), # Up number of channels to 128
torchlayers.InvertedResidualBottleneck(), # Default with squeeze excitation
torch.nn.ReLU(),
torchlayers.AvgPool(), # shape 128 x 64 x 64, kernel_size=2 by default
torchlayers.DepthwiseConv(256), # DepthwiseConv easier to use
# Pass input thrice through the same weights like in PolyNet
torchlayers.Poly(torchlayers.InvertedResidualBottleneck(), order=3),
torchlayers.ReLU(), # all torch.nn can be accessed via torchlayers
torchlayers.MaxPool(), # shape 256 x 32 x 32
torchlayers.Fire(out_channels=512), # shape 512 x 32 x 32
torchlayers.SqueezeExcitation(hidden=64),
torchlayers.InvertedResidualBottleneck(),
torchlayers.MaxPool(), # shape 512 x 16 x 16
torchlayers.InvertedResidualBottleneck(squeeze_excitation=False),
# Randomly switch off the last two layers with 0.5 probability
torchlayers.StochasticDepth(
torch.nn.Sequential(
torchlayers.InvertedResidualBottleneck(squeeze_excitation=False),
torchlayers.InvertedResidualBottleneck(squeeze_excitation=False),
),
p=0.5,
),
torchlayers.AvgPool(), # shape 512 x 8 x 8
)
# This one is more "standard"
self.decoder = torchlayers.Sequential(
torchlayers.Poly(torchlayers.InvertedResidualBottleneck(), order=2),
# Has ICNR initialization by default after calling `build`
torchlayers.ConvPixelShuffle(out_channels=512, upscale_factor=2),
# Shape 512 x 16 x 16 after PixelShuffle
torchlayers.Poly(torchlayers.InvertedResidualBottleneck(), order=3),
torchlayers.ConvPixelShuffle(out_channels=256, upscale_factor=2),
# Shape 256 x 32 x 32
torchlayers.Poly(torchlayers.InvertedResidualBottleneck(), order=3),
torchlayers.ConvPixelShuffle(out_channels=128, upscale_factor=2),
# Shape 128 x 64 x 64
torchlayers.Poly(torchlayers.InvertedResidualBottleneck(), order=4),
torchlayers.ConvPixelShuffle(out_channels=64, upscale_factor=2),
# Shape 64 x 128 x 128
torchlayers.InvertedResidualBottleneck(),
torchlayers.Conv(256),
torchlayers.Dropout(), # Defaults to 0.5 and Dropout2d for images
torchlayers.Swish(),
torchlayers.InstanceNorm(),
torchlayers.ConvPixelShuffle(out_channels=32, upscale_factor=2),
# Shape 32 x 256 x 256
torchlayers.Conv(16),
torchlayers.Swish(),
torchlayers.Conv(3),
# Shape 3 x 256 x 256
)
def forward(self, inputs):
return self.decoder(self.encoder(inputs))
Now one can instantiate the module and use it with torch.nn.MSELoss
as per usual.
autoencoder = torchlayers.build(AutoEncoder(), torch.randn(1, 3, 256, 256))
Installation
pip
Latest release:
pip install --user torchlayers
Nightly:
pip install --user torchlayers-nightly
Docker
CPU standalone and various versions of GPU enabled images are available at dockerhub.
For CPU quickstart, issue:
docker pull szymonmaszke/torchlayers:18.04
Nightly builds are also available, just prefix tag with nightly_
. If you are going for GPU
image make sure you have
nvidia/docker installed and it's runtime set.
Contributing
If you find issue or would like to see some functionality (or implement one), please open new Issue or create Pull Request.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for torchlayers-0.1.1-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 785dd215dd0ca774f776465781ba25bb285245f38ed6ca7dc4ce121012e5ffd2 |
|
MD5 | f709a61b36771384d6c6b16f64ca72aa |
|
BLAKE2b-256 | c73e991c901501dc56b5c19cc7eb8f99523f57cd687fd1dc8b321569aa4d3ec9 |