tensortorrent 0.3.4

Heterogeneous execution planning for PyTorch on a single machine.
Profile → plan → simulate → compile → run across CPUs, GPUs, memory, and storage.

Install · Quick start · Benchmarks · When to use · Docs · Contributing

When a model approaches or exceeds accelerator memory, hand-written .to(device) maps and ad-hoc offload get fragile. TensorTorrent profiles the host, searches placements in a native Rust planner, simulates finalists, then compiles and runs the winner — trading PCIe/host bandwidth for capacity when that trade makes sense.

[!NOTE] Alpha. CPU and virtual backends are covered by CI. Validate accelerators on the target host with tensortorrent validate-hardware.

Benchmarks

Primarily aimed at models that approach or exceed accelerator memory. Small models that fit one GPU are usually faster in native PyTorch — planning/runtime overhead shows up there. Under memory pressure or beyond-VRAM, TensorTorrent can be competitive with host-offload runtimes.

Numbers below: RTX 3070 Ti Laptop (~7.66 GiB VRAM). Full tables, figures, raw JSON: benchmarks/evidence/ · methodology.

Qwen3-8B BF16 logits forward (seq_len=16, 16.38 GB params)

Fixed-shape forward only — not autoregressive generation.

DeepMLP 1.5× VRAM (12.35 GB params)

Fit-in-VRAM (native PyTorch wins)

2× GPU / ROCm / XPU / autoregressive generation: SUPPORTED BUT UNMEASURED on this machine.

Install

pip install torch
pip install tensortorrent

Linux · Python 3.10–3.13 · PyTorch 2.4+. Source builds and CUDA/ROCm/XPU notes: Installation.

Quick start

import torch
import torch.nn as nn
import tensortorrent as tt

model = nn.Sequential(
    nn.Linear(256, 1024),
    nn.GELU(),
    nn.Linear(1024, 256),
).eval()

x = torch.randn(32, 256)
compiled = tt.compile(model, example_inputs=(x,))

y = compiled(x)
torch.testing.assert_close(y, model(x), check_device=False)
print(compiled.explain())

Save/reload, objectives, and multi-module graphs: Quickstart.

When to use

Good fit: model does not fit one GPU · unequal devices · transfer cost matters · RAM/VRAM budgets · parameter streaming or activation spill · reproducible plans instead of hand-written device maps.

Not for: multi-node clusters · exhaustive placement search · “use every detected GPU” · replacing PyTorch kernels · treating discovery as production validation.

Full boundary: Product scope.

Docs

tensortorrent doctor
tensortorrent validate-hardware --output artifacts/validation_report.json
make check   # from a source checkout

License

Apache License 2.0. See LICENSE.

_{TensorTorrent}