torch-slices 0.2.0

Structured Linear CDE (SLiCE) layers for sequence modelling in PyTorch

SLiCEs

slices is a small PyTorch package providing Structured Linear CDE (SLiCE) recurrences.

Mathematical form

Given an input sequence $x_i \in \mathbb{R}^D$ for $i=1,\dots,T$, a SLiCE computes hidden states $y_i \in \mathbb{R}^H$ via

$$ y_i = y_{i-1} + A(X_i)y_{i-1} + B(X_i), $$

where $A(\cdot): \mathbb{R}^D \rightarrow \mathbb{R}^{H \times H}$ and $B(\cdot): \mathbb{R}^D \rightarrow \mathbb{R}^H$ are learned linear maps, the initial state $y_0$ is either a function of $X_0$ or a learnt vector, and the input is augmented with an extra channel:

• inc = a constant “increment” channel (all ones)

such that

$$ X_i = [inc_i, x_i] \in \mathbb{R}^{D+1}. $$

Installation

pip install torch-slices

Or install from source:

pip install git+https://github.com/datasig-ac-uk/slices.git

What's included

• SLiCE: the core structured recurrence
• SLiCELayer: a residual layer wrapping SLiCE with a post-activation stage (GLU or tanh)
• StackedSLiCE: stacks multiple SLiCELayers with an embedding + output projection (supports tokens or continuous inputs)

SLiCE supports both:

• Recurrent execution (step-by-step update)
• Parallel chunked scan execution using torch.associative_scan

Structured transition matrices

SLiCE supports different $A(X_i)$ structures:

1) Diagonal (elementwise update)

Set:

• diagonal_dense=False
• block_size=1

Then $A(X_i)$ is diagonal, which aligns with the approach used by Mamba (see here for more details).

2) Block-diagonal

Set:

• diagonal_dense=False
• block_size > 1

Then $A(X_i)$ is block-diagonal with blocks of shape (block_size × block_size).

3) Diagonal + dense tail block

Set:

• diagonal_dense=True
• block_size > 1

Then the first (hidden_dim - block_size) dimensions are diagonal, and the final block_size dimensions are updated via a dense (block_size × block_size) matrix.

Quickstart

Use SLiCE directly

import torch
from slices import SLiCE

x = torch.randn(8, 128, 32)  # (batch, seq, input_dim)

layer = SLiCE(
    input_dim=32,
    hidden_dim=64,
    block_size=4,
    diagonal_dense=False,
    bias=True,
    use_parallel=True,
    chunk_size=256,
)

y = layer(x)  # (8, 128, 64)
print(y.shape)

Execution mode is configured via constructor arguments (use_parallel, chunk_size).

Use SLiCELayer as a residual sequence layer

import torch
from slices import SLiCELayer

x = torch.randn(4, 256, 64)

layer = SLiCELayer(
    input_dim=64,
    block_size=4,
    diagonal_dense=True,
    dropout_rate=0.01,
    use_glu=True,
)

y = layer(x)  # (4, 256, 64)

Stack layers for a full model

Token sequence mode (tokens=True)

Uses an nn.Embedding(data_dim, hidden_dim) front-end.

import torch
from slices import StackedSLiCE

batch, seq_len = 2, 128
vocab_size = 5000

x = torch.randint(0, vocab_size, (batch, seq_len))

model = StackedSLiCE(
    num_layers=4,
    data_dim=vocab_size,
    hidden_dim=256,
    label_dim=vocab_size,
    tokens=True,
    block_size=4,
    diagonal_dense=False,
    use_glu=True,
)

logits = model(x)  # (batch, seq_len, vocab_size)

Continuous time-series mode (tokens=False)

Uses an nn.Linear(data_dim, hidden_dim) front-end.

import torch
from slices import StackedSLiCE

x = torch.randn(16, 100, 12)  # (batch, seq, data_dim)

model = StackedSLiCE(
    num_layers=3,
    data_dim=12,
    hidden_dim=64,
    label_dim=10,
    tokens=False,
    block_size=4,
    diagonal_dense=True,
)

y = model(x)  # (16, 100, 10)

Benchmarking

To compare recurrent vs parallel throughput across sequence lengths and hidden dimensions:

python examples/benchmark_parallel_vs_recurrent.py

This script:

• benchmarks all four SLiCE matrix modes (diagonal, block_diagonal, diagonal_dense, dense)
• prints timing/speedup tables
• saves a combined 3D plot to examples/images/parallel_vs_recurrent_speedup_3d_all_modes.png

For plotting in development, install development dependencies (includes matplotlib):

uv sync --dev

Requirements

• Python ≥ 3.11
• PyTorch ≥ 2.8.0
• NumPy ≥ 2.4.1

License

MIT License. See LICENSE.