TensorTorrent

Heterogeneous execution planning for PyTorch on a single machine.
Profile → plan → simulate → compile → run across CPUs, GPUs, memory, and storage.

CI PyPI Python 3.10–3.13 Apache-2.0

Install · Quick start · Benchmarks · When to use · Docs · Contributing

When a model approaches or exceeds accelerator memory, hand-written .to(device) maps and ad-hoc offload get fragile. TensorTorrent profiles the host, searches placements in a native Rust planner, simulates finalists, then compiles and runs the winner — trading PCIe/host bandwidth for capacity when that trade makes sense.

[!NOTE] Alpha. CPU and virtual backends are covered by CI. Validate accelerators on the target host with tensortorrent validate-hardware.

Benchmarks

Primarily aimed at models that approach or exceed accelerator memory. Small models that fit one GPU are usually faster in native PyTorch — planning/runtime overhead shows up there. Under memory pressure or beyond-VRAM, TensorTorrent can be competitive with host-offload runtimes.

Numbers below: RTX 3070 Ti Laptop (~7.66 GiB VRAM). Full tables, figures, raw JSON: benchmarks/evidence/ · methodology.

Qwen3-8B BF16 logits forward (seq_len=16, 16.38 GB params)

Fixed-shape forward only — not autoregressive generation.

Approach Median ms Peak VRAM Notes
GPU eager — — infeasible (params > VRAM)
CPU eager 4131 0 ok
TensorTorrent auto 1678 6.83 GB transfer_evict · cosine 0.9997 · argmax 15/16
Accelerate (device_map=auto) 1616 6.44 GB tested config only

DeepMLP 1.5× VRAM (12.35 GB params)

Approach Median ms Peak VRAM Notes
GPU eager — — OOM
CPU eager 446 0.08 GB ok
TensorTorrent auto 444 0.00 GB chose CPU (direct_export_free)
Accelerate (device_map=auto) 807 5.38 GB tested config only

Fit-in-VRAM (native PyTorch wins)

Workload Eager ms TensorTorrent ms Peak VRAM
MLP 512×8 0.23 0.28 17 MB
Transformer 256 0.26 0.34 20 MB
MLP 2048×8 0.70 0.75 146 MB

2× GPU / ROCm / XPU / autoregressive generation: SUPPORTED BUT UNMEASURED on this machine.

TensorTorrent compilation and execution pipeline

Install

pip install torch
pip install tensortorrent

Linux · Python 3.10–3.13 · PyTorch 2.4+. Source builds and CUDA/ROCm/XPU notes: Installation.

Quick start

import torch
import torch.nn as nn
import tensortorrent as tt

model = nn.Sequential(
    nn.Linear(256, 1024),
    nn.GELU(),
    nn.Linear(1024, 256),
).eval()

x = torch.randn(32, 256)
compiled = tt.compile(model, example_inputs=(x,))

y = compiled(x)
torch.testing.assert_close(y, model(x), check_device=False)
print(compiled.explain())

Save/reload, objectives, and multi-module graphs: Quickstart.

When to use

Good fit: model does not fit one GPU · unequal devices · transfer cost matters · RAM/VRAM budgets · parameter streaming or activation spill · reproducible plans instead of hand-written device maps.

Not for: multi-node clusters · exhaustive placement search · “use every detected GPU” · replacing PyTorch kernels · treating discovery as production validation.

Full boundary: Product scope.

Docs
Getting started Install · Quickstart
Architecture Overview · Planner · Runtime
Ops Large models · Deployment · FAQ 
tensortorrent doctor
tensortorrent validate-hardware --output artifacts/validation_report.json
make check   # from a source checkout

License

Apache License 2.0. See LICENSE.

TensorTorrent icon
TensorTorrent

Project links

Data verified by PyPI on Aug 13, 2026
Data provided by the project maintainers, verified at the time the release was uploaded to PyPI.

Key dates

PyPI data
Data sourced directly from PyPI's database.
  • Released:
Latest release

1 maintainer

PyPI data
Data sourced directly from PyPI's database.
Avatar for ajvendetta from gravatar.com ajvendetta

Credits

Author: TensorTorrent Contributors

GitHub Statistics

Data verified by PyPI on Aug 13, 2026
The GitHub source repository was provided by the project maintainers and verified by PyPI at the time of upload. Stars, forks, and open issues/PRs are derived from that repository and have not been independently verified.

License expression

Apache-2.0
View SPDX License List

Requires

Python >=3.10

Provides Extra

bench dev

Tags

pytorch compiler inference heterogeneous multi-gpu numa streaming

Classifiers

Report project as malware

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

tensortorrent-0.3.4.tar.gz (423.4 kB view details)

Uploaded Source

Built Distributions

If you're not sure about the file name format, learn more about wheel file names.

tensortorrent-0.3.4-cp313-cp313-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.6 MB view details)

Uploaded CPython 3.13manylinux: glibc 2.17+ x86-64

tensortorrent-0.3.4-cp313-cp313-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (1.5 MB view details)

Uploaded CPython 3.13manylinux: glibc 2.17+ ARM64

tensortorrent-0.3.4-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.6 MB view details)

Uploaded CPython 3.12manylinux: glibc 2.17+ x86-64

tensortorrent-0.3.4-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (1.5 MB view details)

Uploaded CPython 3.12manylinux: glibc 2.17+ ARM64

tensortorrent-0.3.4-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.6 MB view details)

Uploaded CPython 3.11manylinux: glibc 2.17+ x86-64

tensortorrent-0.3.4-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.6 MB view details)

Uploaded CPython 3.10manylinux: glibc 2.17+ x86-64

File details

Details for the file tensortorrent-0.3.4.tar.gz.

File metadata

  • Download URL: tensortorrent-0.3.4.tar.gz
  • Upload date:
  • Size: 423.4 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for tensortorrent-0.3.4.tar.gz
Algorithm Hash digest
SHA256 5daefb9980a8ac4bc049bb1065d3199bfbb2b832aa4ae751da0bfc145e2c43db
MD5 8a153d68c1e25af4c17aa3e53904d12e
BLAKE2b-256 9805376c47acd86d9074e498efc5dad0594dbf016eb907472ef3773d07edd89f

See more details on using hashes here.

Provenance

The following attestation bundles were made for tensortorrent-0.3.4.tar.gz:

Publisher: release.yml on alhussein-jamil/TensorTorrent

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file tensortorrent-0.3.4-cp313-cp313-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for tensortorrent-0.3.4-cp313-cp313-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 9d476a28a29cc90ac8363ece7835c480d958bff2079dfdbc9b50e2417969fc8c
MD5 2c04b92c99f04da24cd1c9653f6b099a
BLAKE2b-256 7a577e67397e1be8c4d7309e1ba0c1cefb431f11eb12008571eecf25dd88d7c5

See more details on using hashes here.

Provenance

The following attestation bundles were made for tensortorrent-0.3.4-cp313-cp313-manylinux_2_17_x86_64.manylinux2014_x86_64.whl:

Publisher: release.yml on alhussein-jamil/TensorTorrent

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file tensortorrent-0.3.4-cp313-cp313-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for tensortorrent-0.3.4-cp313-cp313-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 2e0da51fde84e885208f5000297817b7eb19fe0838873507b82344e6b7afcaf4
MD5 81a0b4a63a70519ac1d06ba3b5315423
BLAKE2b-256 1d7f724753105f2d2d7f68e59d94a6d62f0b75f34c2b702be3a8339e0ddac728

See more details on using hashes here.

Provenance

The following attestation bundles were made for tensortorrent-0.3.4-cp313-cp313-manylinux_2_17_aarch64.manylinux2014_aarch64.whl:

Publisher: release.yml on alhussein-jamil/TensorTorrent

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file tensortorrent-0.3.4-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for tensortorrent-0.3.4-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 2fd92c6303d1c841bcd25e68583098fe63eeef6bfdc003bee93c9fca192fe4a7
MD5 0a8af1af2c2e63296302e0d27885594f
BLAKE2b-256 4665c95be848c03e0619777c6ae4efa7119c7d5d1df328c05379e376da787149

See more details on using hashes here.

Provenance

The following attestation bundles were made for tensortorrent-0.3.4-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl:

Publisher: release.yml on alhussein-jamil/TensorTorrent

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file tensortorrent-0.3.4-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for tensortorrent-0.3.4-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 ae8fa1277a2836547720bd96941f9f0f91d080f7a620fe85d71d8ac8c95a7982
MD5 1446a490ffcde6f67d9914069ee142d4
BLAKE2b-256 f2d7488e5f992a9cc2a37f111b9e3e533c5d9094c572efd9089e5b0b17108cbf

See more details on using hashes here.

Provenance

The following attestation bundles were made for tensortorrent-0.3.4-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl:

Publisher: release.yml on alhussein-jamil/TensorTorrent

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file tensortorrent-0.3.4-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for tensortorrent-0.3.4-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 6c1e82dc29f7b9e5a3ef71a7165f3bc9e97c3d40ca1e1ee0362218efba6adb6b
MD5 048ff46aed6e6ed96352c22c720ae013
BLAKE2b-256 9b2d406d21c1cd063b9c30902f8384cb0e46d2477e99a366b9fb2dfff94aeedd

See more details on using hashes here.

Provenance

The following attestation bundles were made for tensortorrent-0.3.4-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl:

Publisher: release.yml on alhussein-jamil/TensorTorrent

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file tensortorrent-0.3.4-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for tensortorrent-0.3.4-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 80a3a333ac1dd6df0baab0dc1569347c4e2396e2db8cf3bdad389b671c618698
MD5 d9734ca6a05f47f4d2684be69613922f
BLAKE2b-256 7c88699d6ce54f121a19332a98ccd3c2e647626191cb2aeced8be44843f79ecc

See more details on using hashes here.

Provenance

The following attestation bundles were made for tensortorrent-0.3.4-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl:

Publisher: release.yml on alhussein-jamil/TensorTorrent

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Depot Continuous Integration Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page