Skip to main content

Stackformers

CI PyPI License: MIT

Composable PyTorch transformers for research.

Build vision, spatial, and sequence models with interchangeable attention, positional encodings, normalization, and feedforward components. Start with a preset, then replace individual components as your experiments evolve.

  • Experiment with architecture: explicit, serializable configurations and structural protocols let you supply your own components.
  • Work with variable-length data: run the same encoder weights on padded or packed token batches, with explicit token coordinates.
  • Connect research and deployment: examples cover image patches, point clouds, causal modeling, custom components, and cached ONNX Runtime decoding.

Install

Install from PyPI with pip:

pip install --upgrade stackformers

Or add it to your uv project:

uv add stackformers

Try the latest prerelease

Use --pre to include newer prereleases when upgrading:

pip install --upgrade --pre stackformers

With uv:

uv add --upgrade --prerelease allow stackformers

For an existing virtual environment, uv pip install --upgrade --pre stackformers is also supported. These commands select the newest compatible published version, including prereleases. Pin the version you validate for an experiment; see the research guide.

Run your first encoder

import torch
import stackformers as sf

model = sf.TransformerEncoder(
    sf.plain_encoder_config(dim=64, heads=1, num_layers=2, ff_mult=3.0)
).eval()

sequences = [torch.randn(8, 64), torch.randn(5, 64)]
padded = sf.PaddedTensor.from_list(sequences)
with torch.no_grad():
    padded_output = model(padded)              # (2, 8, 64)
    packed_output = model(padded.to_packed())  # (13, 64)

The two calls share model weights. Packed output contains valid tokens only; padded output retains batch and sequence axes. See the layout parity example for a numerical comparison. Packed input does not guarantee faster execution: backend, hardware, dtype, and length distribution matter.

Explore examples

Browse complete examples for common tasks:

Your task Example
Image patches and 2-D positions Image encoder
Continuous spatial coordinates Point-cloud encoder
Unequal sequence lengths Padded and packed
Replace a model component Custom encoders
Autoregressive token modeling Causal language model
Export an encoder–decoder ONNX Runtime guide

The examples guide includes commands for running the repository examples.

These are integration examples with synthetic inputs, not trained models or task-accuracy benchmarks. Stackformers provides transformer components; your application owns tokenization, data processing, training, and task heads.

What can you change?

Component Included choices
Attention Global self/cross attention, local self-attention, GQA/MQA
Token layout Padded or packed, with explicit positions
Positional encoding RoPE-1D, RoPE-2D, continuous-coordinate RoPE-ND, or none
Feedforward SwiGLU, HardSwishGLU, GEGLU, GELU, ReLU²
Normalization RMSNorm, LayerNorm; pre, post, sandwich, or reordered placement
Model assembly Encoders, decoders, cross-attenders, variable-width encoders
Deployment Projection packing, compile/export paths, cached decoder adapters

Presets build components from discriminated Pydantic configurations. For deeper changes, wire modules directly or implement an owning module's protocol. The configuration and API guide covers both approaches.

Compatibility and limits

  • Global attention uses PyTorch SDPA. Eligible CUDA fp16/bf16 packed workloads use PyTorch's variable-length attention; other paths can fall back to padded execution.
  • Compile and export support depend on the component and input path. Eager list/nested adapters and deployment cache strategies have separate contracts.
  • Cached deployment supports specified built-in classes; custom forward overrides need their own deployment integration.
  • No speedup or downstream accuracy improvement is claimed without a matching benchmark.

See compatibility evidence and inference contracts before selecting a deployment path.

Documentation

Development

git clone https://github.com/Red-Eyed/stackformers.git
cd stackformers
just sync
just check

The Justfile owns formatting, linting, type-checking, tests, and builds.

Citation and license

If Stackformers contributes to your research implementation, cite the software using CITATION.cff and report the exact version or commit used. No associated research paper or conference endorsement is implied.

Released under the MIT license.

Metadata

Release files for stackformers 5.1.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for stackformers 5.1.0
File Size Uploaded Provenance Actions
stackformers-5.1.0.tar.gz 595.0 kB Without provenance Details for stackformers-5.1.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for stackformers 5.1.0
File Interpreter ABI Platform Actions
stackformers-5.1.0-py3-none-any.whl Python 3 none any Details for stackformers-5.1.0-py3-none-any.whl

Total release size: 737.5 kB

Release files / stackformers-5.1.0.tar.gz

Download URL stackformers-5.1.0.tar.gz
Size 595.0 kB
Tags Source
SHA-256 checksum
How to use checksums
811438b90528ee2e260ce5d60c715e58ac39e4a38df940f43c627c88570aa405
BLAKE2b-256 checksum
How to use checksums
a9fb02dbb1bf5b4e3ebdbcb1fab3c326028b5ca2ea06a8c035453b5f87ad753d
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via uv/0.12.1 {"installer":{"name":"uv","version":"0.12.1","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"macOS","version":null,"id":null,"libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}

Release files / stackformers-5.1.0-py3-none-any.whl

Download URL stackformers-5.1.0-py3-none-any.whl
Size 142.5 kB
Tags Python 3
SHA-256 checksum
How to use checksums
0dd29c0f58bd118f35b407b508f72ac5423458e1f02bfcfe0e96b0fcc328862b
BLAKE2b-256 checksum
How to use checksums
e9ba22a677eb23e8ad81821ab272b3bb0aea089a587c01717650501b95009f07
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via uv/0.12.1 {"installer":{"name":"uv","version":"0.12.1","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"macOS","version":null,"id":null,"libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}

Security for stackformers 5.1.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

5.1.0 This release

2 release files No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page