tensortorrent

TensorTorrent

Heterogeneous execution planning for PyTorch on a single machine.
Profile → plan → simulate → compile → run across CPUs, GPUs, memory, and storage.

Python 3.10–3.13

Install · Quick start · Benchmarks · When to use · Docs · Contributing

TensorTorrent is a capacity-oriented heterogeneous runtime for PyTorch. It profiles a host, searches placements in a native Rust planner, simulates finalists, then compiles and runs the winner across unequal CPUs, accelerators, memory tiers, and storage.

Problem: when a model approaches or exceeds accelerator memory, hand-written .to(device) maps and ad-hoc offload become fragile. TensorTorrent can trade PCIe/host transfer bandwidth for execution capacity by streaming state through the accelerator.

[!NOTE] Alpha. CPU and virtual backends are covered by CI. Validate accelerators on the target host with tensortorrent validate-hardware.

Benchmarks

TensorTorrent is primarily designed for models that approach or exceed accelerator memory. Native PyTorch is expected to be faster for small models that fit comfortably on a single GPU — TensorTorrent adds planning and runtime overhead there. On memory-constrained or beyond-VRAM workloads, it provides capacity and can be competitive with host-offload runtimes.

Numbers below are measured on an RTX 3070 Ti Laptop GPU (~7.66 GiB VRAM). Full tables, figures, and raw JSON: benchmarks/evidence/ · methodology.

Qwen3-8B BF16 logits forward (`seq_len=16`, 16.38 GB params)

Fixed-shape forward only — not autoregressive generation.

Approach	Median ms	Peak VRAM	Notes
GPU eager	—	—	infeasible (params > VRAM)
CPU eager	4131	0	ok
TensorTorrent auto	1678	6.83 GB	`transfer_evict` · cosine 0.9997 · argmax 15/16
Accelerate (`device_map=auto`)	1616	6.44 GB	tested config only

DeepMLP 1.5× VRAM (12.35 GB params)

Approach	Median ms	Peak VRAM	Notes
GPU eager	—	—	OOM
CPU eager	446	0.08 GB	ok
TensorTorrent auto	444	0.00 GB	chose CPU (`direct_export_free`)
Accelerate (`device_map=auto`)	807	5.38 GB	tested config only

Fit-in-VRAM (native PyTorch wins)

Workload	Eager ms	TensorTorrent ms	Peak VRAM
MLP 512×8	0.23	0.28	17 MB
Transformer 256	0.26	0.34	20 MB
MLP 2048×8	0.70	0.75	146 MB

2× GPU / ROCm / XPU / autoregressive generation: SUPPORTED BUT UNMEASURED on this machine.

TensorTorrent compilation and execution pipeline

Install

pip install torch
pip install tensortorrent

Linux · Python 3.10–3.13 · PyTorch 2.4+. Source builds and CUDA/ROCm/XPU notes: Installation.

Quick start

import torch
import torch.nn as nn
import tensortorrent as tt

model = nn.Sequential(
    nn.Linear(256, 1024),
    nn.GELU(),
    nn.Linear(1024, 256),
).eval()

x = torch.randn(32, 256)
compiled = tt.compile(model, example_inputs=(x,))

y = compiled(x)
torch.testing.assert_close(y, model(x), check_device=False)
print(compiled.explain())

Save/reload, objectives, and multi-module graphs: Quickstart.

When to use

Good fit: model does not fit one GPU · unequal devices · transfer cost matters · RAM/VRAM budgets · parameter streaming or activation spill · reproducible plans instead of hand-written device maps.

Not for: multi-node clusters · exhaustive placement search · “use every detected GPU” · replacing PyTorch kernels · treating discovery as production validation.

Full boundary: Product scope.

Docs


Getting started	Install · Quickstart
Architecture	Overview · Planner · Runtime
Ops	Large models · Deployment · FAQ

tensortorrent doctor
tensortorrent validate-hardware --output artifacts/validation_report.json
make check   # from a source checkout

License

Apache License 2.0. See LICENSE.

TensorTorrent icon
_{TensorTorrent}

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

tensortorrent-0.3.2.tar.gz (409.4 kB view details)

Uploaded Aug 10, 2026 Source

Built Distributions

If you're not sure about the file name format, learn more about wheel file names.

The dropdown lists show the available interpreters, ABIs, and platforms. Enable javascript to be able to filter the list of wheel files.

tensortorrent-0.3.2-cp313-cp313-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.6 MB view details)

Uploaded Aug 10, 2026 CPython 3.13manylinux: glibc 2.17+ x86-64

tensortorrent-0.3.2-cp313-cp313-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (1.5 MB view details)

Uploaded Aug 10, 2026 CPython 3.13manylinux: glibc 2.17+ ARM64

tensortorrent-0.3.2-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.6 MB view details)

Uploaded Aug 10, 2026 CPython 3.12manylinux: glibc 2.17+ x86-64

tensortorrent-0.3.2-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (1.5 MB view details)

Uploaded Aug 10, 2026 CPython 3.12manylinux: glibc 2.17+ ARM64

tensortorrent-0.3.2-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.6 MB view details)

Uploaded Aug 10, 2026 CPython 3.11manylinux: glibc 2.17+ x86-64

tensortorrent-0.3.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.6 MB view details)

Uploaded Aug 10, 2026 CPython 3.10manylinux: glibc 2.17+ x86-64

File details

Details for the file tensortorrent-0.3.2.tar.gz.

File metadata

Download URL: tensortorrent-0.3.2.tar.gz
Upload date: Aug 10, 2026
Size: 409.4 kB
Tags: Source
Uploaded using Trusted Publishing? Yes
Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for tensortorrent-0.3.2.tar.gz
Algorithm	Hash digest
SHA256	`464b21ea4b5c379ac4f86fde18bc8326245eeae4adf2bb9e51325584a7e74194`
MD5	`a0496e8604ade5c86fe03078fbc0a3e9`
BLAKE2b-256	`4f78e807239d9790e68da41b313cba3cef63c556620ed21a8064d0a45fab2202`

See more details on using hashes here.

Provenance

The following attestation bundles were made for tensortorrent-0.3.2.tar.gz:

Publisher: release.yml on alhussein-jamil/TensorTorrent

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Statement:
- Statement type: https://in-toto.io/Statement/v1
- Predicate type: https://docs.pypi.org/attestations/publish/v1
- Subject name: tensortorrent-0.3.2.tar.gz
- Subject digest: 464b21ea4b5c379ac4f86fde18bc8326245eeae4adf2bb9e51325584a7e74194
- Sigstore transparency entry: 2413665290
- Sigstore integration time: Aug 10, 2026
Source repository:
- Permalink: alhussein-jamil/TensorTorrent@e142fbe960e9e607adaf9931c2ee651cb195aea7
- Branch / Tag: refs/tags/v0.3.2
- Owner: https://github.com/alhussein-jamil
- Access: public
Publication detail:
- Token Issuer: https://token.actions.githubusercontent.com
- Runner Environment: github-hosted
- Publication workflow: release.yml@e142fbe960e9e607adaf9931c2ee651cb195aea7
- Trigger Event: push

File details

Details for the file tensortorrent-0.3.2-cp313-cp313-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

Download URL: tensortorrent-0.3.2-cp313-cp313-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Upload date: Aug 10, 2026
Size: 1.6 MB
Tags: CPython 3.13, manylinux: glibc 2.17+ x86-64
Uploaded using Trusted Publishing? Yes
Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for tensortorrent-0.3.2-cp313-cp313-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm	Hash digest
SHA256	`2cbf4bb1ed22b607757e652731e7ed2f7e88f1a1de6d7e373921453e168c61a2`
MD5	`5e67ea8983ee3efe7c991976e1e9fce1`
BLAKE2b-256	`2b18d8c874308fd9632c75170a45bb2ca97096ef4bbbafaba0c4b67acb09642a`

See more details on using hashes here.

Provenance

The following attestation bundles were made for tensortorrent-0.3.2-cp313-cp313-manylinux_2_17_x86_64.manylinux2014_x86_64.whl:

Publisher: release.yml on alhussein-jamil/TensorTorrent

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Statement:
- Statement type: https://in-toto.io/Statement/v1
- Predicate type: https://docs.pypi.org/attestations/publish/v1
- Subject name: tensortorrent-0.3.2-cp313-cp313-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
- Subject digest: 2cbf4bb1ed22b607757e652731e7ed2f7e88f1a1de6d7e373921453e168c61a2
- Sigstore transparency entry: 2413666944
- Sigstore integration time: Aug 10, 2026
Source repository:
- Permalink: alhussein-jamil/TensorTorrent@e142fbe960e9e607adaf9931c2ee651cb195aea7
- Branch / Tag: refs/tags/v0.3.2
- Owner: https://github.com/alhussein-jamil
- Access: public
Publication detail:
- Token Issuer: https://token.actions.githubusercontent.com
- Runner Environment: github-hosted
- Publication workflow: release.yml@e142fbe960e9e607adaf9931c2ee651cb195aea7
- Trigger Event: push

File details

Details for the file tensortorrent-0.3.2-cp313-cp313-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

Download URL: tensortorrent-0.3.2-cp313-cp313-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Upload date: Aug 10, 2026
Size: 1.5 MB
Tags: CPython 3.13, manylinux: glibc 2.17+ ARM64
Uploaded using Trusted Publishing? Yes
Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for tensortorrent-0.3.2-cp313-cp313-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm	Hash digest
SHA256	`f20e92692a0328bb1a44cfd4c7a9aec00e698bb668000628bbc740d9f9d4cf1f`
MD5	`5ee2434c13eb3b6b13ccd69259669855`
BLAKE2b-256	`4d096499fca6da0edc77a14ae0fad051eb2d79181c30225859dffaf41797e54f`

See more details on using hashes here.

Provenance

The following attestation bundles were made for tensortorrent-0.3.2-cp313-cp313-manylinux_2_17_aarch64.manylinux2014_aarch64.whl:

Publisher: release.yml on alhussein-jamil/TensorTorrent

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Statement:
- Statement type: https://in-toto.io/Statement/v1
- Predicate type: https://docs.pypi.org/attestations/publish/v1
- Subject name: tensortorrent-0.3.2-cp313-cp313-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
- Subject digest: f20e92692a0328bb1a44cfd4c7a9aec00e698bb668000628bbc740d9f9d4cf1f
- Sigstore transparency entry: 2413666686
- Sigstore integration time: Aug 10, 2026
Source repository:
- Permalink: alhussein-jamil/TensorTorrent@e142fbe960e9e607adaf9931c2ee651cb195aea7
- Branch / Tag: refs/tags/v0.3.2
- Owner: https://github.com/alhussein-jamil
- Access: public
Publication detail:
- Token Issuer: https://token.actions.githubusercontent.com
- Runner Environment: github-hosted
- Publication workflow: release.yml@e142fbe960e9e607adaf9931c2ee651cb195aea7
- Trigger Event: push

File details

Details for the file tensortorrent-0.3.2-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

Download URL: tensortorrent-0.3.2-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Upload date: Aug 10, 2026
Size: 1.6 MB
Tags: CPython 3.12, manylinux: glibc 2.17+ x86-64
Uploaded using Trusted Publishing? Yes
Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for tensortorrent-0.3.2-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm	Hash digest
SHA256	`e0228d92843c7ece76a8c08897b7e5842ed2a67171e9fab319e4637bc7ebc946`
MD5	`b7282fd2de3640258e76260675df5021`
BLAKE2b-256	`f50b80c96163be3f6bdbb07d190152af705aa0fd9fa8a664346f52a69d050e51`

See more details on using hashes here.

Provenance

The following attestation bundles were made for tensortorrent-0.3.2-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl:

Publisher: release.yml on alhussein-jamil/TensorTorrent

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Statement:
- Statement type: https://in-toto.io/Statement/v1
- Predicate type: https://docs.pypi.org/attestations/publish/v1
- Subject name: tensortorrent-0.3.2-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
- Subject digest: e0228d92843c7ece76a8c08897b7e5842ed2a67171e9fab319e4637bc7ebc946
- Sigstore transparency entry: 2413665987
- Sigstore integration time: Aug 10, 2026
Source repository:
- Permalink: alhussein-jamil/TensorTorrent@e142fbe960e9e607adaf9931c2ee651cb195aea7
- Branch / Tag: refs/tags/v0.3.2
- Owner: https://github.com/alhussein-jamil
- Access: public
Publication detail:
- Token Issuer: https://token.actions.githubusercontent.com
- Runner Environment: github-hosted
- Publication workflow: release.yml@e142fbe960e9e607adaf9931c2ee651cb195aea7
- Trigger Event: push

File details

Details for the file tensortorrent-0.3.2-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

Download URL: tensortorrent-0.3.2-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Upload date: Aug 10, 2026
Size: 1.5 MB
Tags: CPython 3.12, manylinux: glibc 2.17+ ARM64
Uploaded using Trusted Publishing? Yes
Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for tensortorrent-0.3.2-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm	Hash digest
SHA256	`cc75463e58e9de61c913ab7befb52b1fea53293506af3812d691b84d14dc76a0`
MD5	`272533f5dee4859dd6dab35e03bf6409`
BLAKE2b-256	`769661e52445e44d8c0a12d7fab1ffa52a8729c6e108edf44c50f10f5f062c1d`

See more details on using hashes here.

Provenance

The following attestation bundles were made for tensortorrent-0.3.2-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl:

Publisher: release.yml on alhussein-jamil/TensorTorrent

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Statement:
- Statement type: https://in-toto.io/Statement/v1
- Predicate type: https://docs.pypi.org/attestations/publish/v1
- Subject name: tensortorrent-0.3.2-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
- Subject digest: cc75463e58e9de61c913ab7befb52b1fea53293506af3812d691b84d14dc76a0
- Sigstore transparency entry: 2413665749
- Sigstore integration time: Aug 10, 2026
Source repository:
- Permalink: alhussein-jamil/TensorTorrent@e142fbe960e9e607adaf9931c2ee651cb195aea7
- Branch / Tag: refs/tags/v0.3.2
- Owner: https://github.com/alhussein-jamil
- Access: public
Publication detail:
- Token Issuer: https://token.actions.githubusercontent.com
- Runner Environment: github-hosted
- Publication workflow: release.yml@e142fbe960e9e607adaf9931c2ee651cb195aea7
- Trigger Event: push

File details

Details for the file tensortorrent-0.3.2-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

Download URL: tensortorrent-0.3.2-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Upload date: Aug 10, 2026
Size: 1.6 MB
Tags: CPython 3.11, manylinux: glibc 2.17+ x86-64
Uploaded using Trusted Publishing? Yes
Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for tensortorrent-0.3.2-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm	Hash digest
SHA256	`3bfe72822d3f3ba4d683feff14e680cd8d3a37380e6573532f0e4c5d1cff14a5`
MD5	`950d8eda44ec1a6b6762938ff7eb2ef0`
BLAKE2b-256	`83ada48ac8d57bfa8843179dfd44489d184796a92ff6158a19c64dcbcaffc3c1`

See more details on using hashes here.

Provenance

The following attestation bundles were made for tensortorrent-0.3.2-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl:

Publisher: release.yml on alhussein-jamil/TensorTorrent

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Statement:
- Statement type: https://in-toto.io/Statement/v1
- Predicate type: https://docs.pypi.org/attestations/publish/v1
- Subject name: tensortorrent-0.3.2-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
- Subject digest: 3bfe72822d3f3ba4d683feff14e680cd8d3a37380e6573532f0e4c5d1cff14a5
- Sigstore transparency entry: 2413667202
- Sigstore integration time: Aug 10, 2026
Source repository:
- Permalink: alhussein-jamil/TensorTorrent@e142fbe960e9e607adaf9931c2ee651cb195aea7
- Branch / Tag: refs/tags/v0.3.2
- Owner: https://github.com/alhussein-jamil
- Access: public
Publication detail:
- Token Issuer: https://token.actions.githubusercontent.com
- Runner Environment: github-hosted
- Publication workflow: release.yml@e142fbe960e9e607adaf9931c2ee651cb195aea7
- Trigger Event: push

File details

Details for the file tensortorrent-0.3.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

Download URL: tensortorrent-0.3.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Upload date: Aug 10, 2026
Size: 1.6 MB
Tags: CPython 3.10, manylinux: glibc 2.17+ x86-64
Uploaded using Trusted Publishing? Yes
Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for tensortorrent-0.3.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm	Hash digest
SHA256	`7fff982b8404903eeb898b32fb90003a5c1bdcd5f31ad0a9b84fa3bc61960ac9`
MD5	`8afd4b7d1761fdfb8203f1a7c5cd5b83`
BLAKE2b-256	`15832b92e2245ce4fbbbee6194e97131ea30c1fa1c460cc2f9ed5436fefb8781`

See more details on using hashes here.

Provenance

The following attestation bundles were made for tensortorrent-0.3.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl:

Publisher: release.yml on alhussein-jamil/TensorTorrent

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Statement:
- Statement type: https://in-toto.io/Statement/v1
- Predicate type: https://docs.pypi.org/attestations/publish/v1
- Subject name: tensortorrent-0.3.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
- Subject digest: 7fff982b8404903eeb898b32fb90003a5c1bdcd5f31ad0a9b84fa3bc61960ac9
- Sigstore transparency entry: 2413666348
- Sigstore integration time: Aug 10, 2026
Source repository:
- Permalink: alhussein-jamil/TensorTorrent@e142fbe960e9e607adaf9931c2ee651cb195aea7
- Branch / Tag: refs/tags/v0.3.2
- Owner: https://github.com/alhussein-jamil
- Access: public
Publication detail:
- Token Issuer: https://token.actions.githubusercontent.com
- Runner Environment: github-hosted
- Publication workflow: release.yml@e142fbe960e9e607adaf9931c2ee651cb195aea7
- Trigger Event: push

Release history Release notifications | RSS feed

0.3.4

Aug 13, 2026

This release

0.3.2

Aug 10, 2026

0.3.1

Aug 9, 2026

0.3.0

Aug 9, 2026

0.2.8

Aug 7, 2026

0.2.7

Aug 7, 2026

0.2.6

Aug 7, 2026

0.2.5

Aug 7, 2026

0.2.4

Aug 6, 2026

0.2.3

Aug 6, 2026

0.2.2

Aug 6, 2026

0.2.1

Aug 6, 2026

0.2.0

Aug 6, 2026

tensortorrent 0.3.2

Benchmarks

Qwen3-8B BF16 logits forward (seq_len=16, 16.38 GB params)

DeepMLP 1.5× VRAM (12.35 GB params)

Fit-in-VRAM (native PyTorch wins)

Install

Quick start

When to use

Docs

License

Project links

Key dates

1 maintainer

Credits

GitHub Statistics

License expression

Requires

Provides Extra

Tags

Classifiers

Download files

Source Distribution

Built Distributions

File details

File metadata

File hashes

Provenance

File details

File metadata

File hashes

Provenance

File details

File metadata

File hashes

Provenance

File details

File metadata

File hashes

Provenance

File details

File metadata

File hashes

Provenance

File details

File metadata

File hashes

Provenance

File details

File metadata

File hashes

Provenance

Release history Release notifications | RSS feed

Qwen3-8B BF16 logits forward (`seq_len=16`, 16.38 GB params)

DeepMLP 1.5× VRAM (12.35 GB params)