Skip to main content
Pre-release

This release is a pre-release and may not be stable for production use.

Stackformers

CI PyPI License: MIT

Composable PyTorch transformers for research.

Build vision, spatial, and sequence models with interchangeable attention, positional encodings, normalization, and feedforward components. Start with a preset, then replace individual components as your experiments evolve.

  • Experiment with architecture: explicit, serializable configurations and structural protocols let you supply your own components.
  • Work with variable-length data: run the same encoder weights on padded or packed token batches, with explicit token coordinates.
  • Connect research and deployment: examples cover image patches, point clouds, causal modeling, custom components, and cached ONNX Runtime decoding.

Try the current API

Python 3.11 or newer is required. This checkout is on the 5.0 release-candidate series. To run the examples shown here against the matching source:

git clone https://github.com/Red-Eyed/stackformers.git
cd stackformers
uv run examples/padded_and_packed.py

uv run prepares the project environment automatically. For a published release in an existing project:

uv add stackformers

An unpinned install may resolve a different stable release whose API differs from this checkout. Use the documentation matching your installed version. For experiments, pin the exact version or commit you validated; see the research guide.

Run your first encoder

import torch
import stackformers as sf

model = sf.TransformerEncoder(
    sf.plain_encoder_config(dim=64, heads=1, num_layers=2, ff_mult=3.0)
).eval()

sequences = [torch.randn(8, 64), torch.randn(5, 64)]
padded = sf.PaddedTensor.from_list(sequences)
with torch.no_grad():
    padded_output = model(padded)              # (2, 8, 64)
    packed_output = model(padded.to_packed())  # (13, 64)

The two calls share model weights. Packed output contains valid tokens only; padded output retains batch and sequence axes. See the layout parity example for a numerical comparison. Packed input does not guarantee faster execution: backend, hardware, dtype, and length distribution matter.

Choose an example

Examples run on CPU without dataset or model downloads. From a repository checkout, uv run prepares the project environment automatically.

Your task Start here Run
Image patches and 2-D positions Image encoder uv run examples/image_encoder.py
Continuous spatial coordinates Point-cloud encoder uv run examples/point_cloud_encoder.py
Unequal sequence lengths Padded and packed uv run examples/padded_and_packed.py
Replace a model component Custom encoders uv run examples/subclassed_encoder.py
Autoregressive token modeling Causal language model uv run examples/causal_language_model.py
Export an encoder–decoder ONNX Runtime guide uv run examples/encoder_decoder_onnxruntime.py

These are integration examples with synthetic inputs, not trained models or task-accuracy benchmarks. Stackformers provides transformer components; your application owns tokenization, data processing, training, and task heads.

What can you change?

Component Included choices
Attention Global self/cross attention, local self-attention, GQA/MQA
Token layout Padded or packed, with explicit positions
Positional encoding RoPE-1D, RoPE-2D, continuous-coordinate RoPE-ND, or none
Feedforward SwiGLU, HardSwishGLU, GEGLU, GELU, ReLU²
Normalization RMSNorm, LayerNorm; pre, post, sandwich, or reordered placement
Model assembly Encoders, decoders, cross-attenders, variable-width encoders
Deployment Projection packing, compile/export paths, cached decoder adapters

Presets build components from discriminated Pydantic configurations. For deeper changes, wire modules directly or implement an owning module's protocol. The configuration and API guide covers both approaches.

Compatibility and limits

  • Global attention uses PyTorch SDPA. Eligible CUDA fp16/bf16 packed workloads use PyTorch's variable-length attention; other paths can fall back to padded execution.
  • Compile and export support depend on the component and input path. Eager list/nested adapters and deployment cache strategies have separate contracts.
  • Cached deployment supports specified built-in classes; custom forward overrides need their own deployment integration.
  • No speedup or downstream accuracy improvement is claimed without a matching benchmark.

See compatibility evidence and inference contracts before selecting a deployment path.

Documentation

Development

git clone https://github.com/Red-Eyed/stackformers.git
cd stackformers
just sync
just check

The Justfile owns formatting, linting, type-checking, tests, and builds.

Citation and license

If Stackformers contributes to your research implementation, cite the software using CITATION.cff and report the exact version or commit used. No associated research paper or conference endorsement is implied.

Released under the MIT license.

Metadata

Release files for stackformers 5.0.0rc22

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for stackformers 5.0.0rc22
File Size Uploaded Provenance Actions
stackformers-5.0.0rc22.tar.gz 589.9 kB Without provenance Details for stackformers-5.0.0rc22.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for stackformers 5.0.0rc22
File Interpreter ABI Platform Actions
stackformers-5.0.0rc22-py3-none-any.whl Python 3 none any Details for stackformers-5.0.0rc22-py3-none-any.whl

Total release size: 730.2 kB

Release files / stackformers-5.0.0rc22.tar.gz

Download URL stackformers-5.0.0rc22.tar.gz
Size 589.9 kB
Tags Source
SHA-256 checksum
How to use checksums
f6d0b75a983899120f94517a753a8bf8d84b0822b468b024e06e552ec8dc4402
BLAKE2b-256 checksum
How to use checksums
da11ba448138f2cc8b8a9166b591798cc7941494f2936b274d4c38979ff245d3
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via uv/0.12.1 {"installer":{"name":"uv","version":"0.12.1","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"macOS","version":null,"id":null,"libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}

Release files / stackformers-5.0.0rc22-py3-none-any.whl

Download URL stackformers-5.0.0rc22-py3-none-any.whl
Size 140.2 kB
Tags Python 3
SHA-256 checksum
How to use checksums
349ad420ff2d4150bcbc70fa7dbd081cca315aa44624a3a40601b7c02bb804a8
BLAKE2b-256 checksum
How to use checksums
8647ce0ee83e80c20c02a2878bf89386c633b0055522a0463962a2a8f3ea20f6
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via uv/0.12.1 {"installer":{"name":"uv","version":"0.12.1","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"macOS","version":null,"id":null,"libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}

Security for stackformers 5.0.0rc22

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release
Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page