English | 한국어

torchnative replaces PyTorch's compiled core, torch._C, with a native Rust extension. The genuine torch and transformers packages then run on a phone the way they run on a workstation. Models are not ported, converted or re-expressed. They are imported.

from transformers import AutoModelForCausalLM     # the real one
model = AutoModelForCausalLM.from_pretrained("...")
model.generate(...)                                # on the device

💡 Why not a reimplementation

Every other route to on-device inference re-expresses the model somewhere else.

Nobody runs the real thing because torch._C cannot be built for mobile: PyTorch's build sets INTERN_BUILD_MOBILE for any Android or iOS toolchain, and that path forces BUILD_PYTHON off. torchnative supplies that one module. Everything above it is upstream source, unmodified.

✨ Features

• 🤖 LLM inference with real transformers: from_pretrained loads bit-identical weights, generate emits upstream's tokens in float32, and streaming works across threads.
• 🎯 Agreement, not just reachability: 302 ATen operators compared against upstream; 297 of 297 swept architectures forward and 284 of 285 judgeable ones agree numerically, at a tolerance derived from upstream's own float32 error.
• 🔁 Test-time adaptation: adapt.wrap(model, method=adapt.Tent()) adapts in place; a weight delta over the base reverts bit-identically.
• 🌐 Federated learning on torch.distributed: FedAvg across processes equals the same average computed centrally, element for element.
• ⚡ Accelerators without silent fallback: Metal and Vulkan compute on the real GPU, a CoreML graph runs on the Apple Neural Engine, and anything that would fall back to the host refuses by name instead.
• 🧩 NPUs by module replacement: model.to(torchnative.device.npu) lowers leaves for the host's NPU and hands back the same nn.Module, so it still trains and still generates.
• 📦 One wheel per platform: cp313-abi3, so one binary loads on CPython 3.13 and later.

🚀 Quick start

uv add --prerelease allow torchnative
# or, with pypackpack
ppp core add "torchnative==0.1.0b4"

Stream tokens from a real Hugging Face model:

import torch
from threading import Thread
from transformers import AutoModelForCausalLM, AutoTokenizer, TextIteratorStreamer

name  = "HuggingFaceTB/SmolLM2-135M"
model = AutoModelForCausalLM.from_pretrained(name, dtype=torch.float32)
tok   = AutoTokenizer.from_pretrained(name)

streamer = TextIteratorStreamer(tok, skip_prompt=True, skip_special_tokens=True)
inputs   = tok("On-device inference is", return_tensors="pt")

Thread(target=model.generate, kwargs=dict(**inputs, max_new_tokens=32, streamer=streamer)).start()

for piece in streamer:          # yields as the model decodes
    print(piece, end="", flush=True)

On an M-series desktop the first token arrives in 34 ms, then 47 tokens/s. In float32 the tokens are the ones upstream emits; in bfloat16 they diverge, because upstream disagrees with itself under a change of accumulation order.

Adapt at test time, then put the base weights back:

from torchnative import adapt

model = adapt.wrap(model, method=adapt.Tent(), lr=1e-3)
model.online()                                   # adapt as it serves
model.revert()                                   # the base weights are back, byte for byte

Send it to the NPU without giving up the module:

from torchnative import device

model.to(device.npu)   # resolves the host's NPU, lowers what it can, returns the same nn.Module;
                       # a UserWarning names anything that stayed on the CPU

More task pages are in the guide.

🧩 How it works

flowchart TB
    A["your code · transformers · torch/*.py<br/><i>upstream Python, unmodified</i>"]
    B["torch._C, replaced<br/><b>_aten_dispatch</b>: the single door every operator passes"]
    C["kernels, Rust, backed by candle"]
    D["CPU · Metal · Vulkan"]
    E["CoreML · OpenVINO · QNN<br/><i>whole subgraphs, behind a module</i>"]
    A --> B --> C --> D
    B -- "graph capture" --> E

• One door. Every operator reaches its kernel through _aten_dispatch, and nothing bypasses it. An unimplemented operator names itself instead of failing downstream, and graph capture has exactly one place to attach.
• Fixed front end, swapped back end. The user keeps the real transformers model; an NPU back end replaces leaves or submodules in place, and a vendor blob is never something you hold.
• Refuse by name. What is not supported says so, with the reason. A silent CPU fallback is made unrepresentable rather than merely avoided.
• Stable ABI. Built against CPython's limited API (abi3-py313): one binary per platform.

📊 Status

Every number below has a run behind it. The long form, with the measurement and caveat for every row, is docs/platform/STATUS.md.

🖥️ Platform support

Legend: ✅ measured working · ❌ measured refusing · ⚠️ built, never executed · 🔲 not built · n/a not applicable to that platform

The dtype and quantisation matrices, and the reasoning behind every cell, are in docs/platform/STATUS.md and docs/platform/WHEELMATRIX.md.

🗺️ Roadmap

This table records what has been measured, not what is planned. No row promises more than a run shows. Last re-measured 2026-09-07 (docs/verification/REMEASURE2.md); where it disagrees with a docs/ file, the file is the measurement and this is the summary.

Planned, not measured (#18): kernel bundles that satisfy the Hugging Face kernels contract and resolve at build time (DESIGN.md §8). Not on the roadmap: torch.compile, which is refused by name permanently. torch.export is the direction. The long-form roadmap and the API it is heading for are in docs/platform/STATUS.md.

📦 Install

uv add --prerelease allow torchnative
# or, with pypackpack
ppp core add "torchnative==0.1.0b4"

Nine platform wheels, all cp313-abi3. Each carries the _C extension and the vendored upstream tree, so import torch resolves to this build. It cannot coexist with an installed PyTorch. 0.0.1a0 on PyPI is a broken py3-none-any skeleton; ask for 0.1.0b0 or later. There is no sdist.

Build from source (Rust toolchain, CPython 3.13+):

bash scripts/vendor/vendor_torch.sh     # assemble the vendored torch tree
bash scripts/vendor/install_shim.sh     # build the extension and install it
python scripts/wheel/build.py                            # -> dist/*.whl
python scripts/wheel/verify.py dist/torchnative-*.whl    # clean venv, real import

Cross-compilation: docs/platform/RUST_CROSSBUILD.md.

🧪 Verification

Correctness here means agreeing with upstream PyTorch, so the strategy is comparison, not assertion.

• Golden comparison: every operator runs on upstream torch and on this shim, compared on value, shape and dtype.
• The harness tests itself: --self-test injects plausible misimplementations into each comparator and fails if any is accepted.
• Tokens are not enough: a wrong gelu produced identical tokens with logits off by 5.9e-04, so end-to-end tests compare logits too.
• Documentation is checked: DOCWATCH markers tie the numbers on this page to live runs.

bash tests/run.sh                # the gate
python tests/golden/compare.py                  # golden comparison against upstream

📖 Documentation

• 🌐 Guide: getting started, concepts, task guides, English and Korean (source in docs/guide/)
• 📊 Status in detail: every measurement and caveat behind the tables above
• 🏛️ Design: why only torch._C, and the decisions that constrain the rest
• 🇰🇷 한국어 README

docs/ records what was measured, what was assumed, and where an earlier conclusion turned out to be wrong. Corrections stay visible so that a reader can see what changed.

🤝 Contributing

Work starts from an issue and lands through a pull request into develop; tests come first. The contributing page of the guide explains the workflow and the verification standard.

🔗 Ecosystem

• PythonMultiplatform: embeds CPython 3.13 in Kotlin Multiplatform, the deployment target for this library
• pypackpack: the build and bundling tool
• Hugging Face kernels: the fused-kernel contract this adopts, with resolution moved from runtime download to build time

📜 License

Apache-2.0: see LICENSE.

The repository and the wheel carry different licences. This repository contains only this project's code; the vendored PyTorch tree is assembled at build time and not redistributed here. A platform wheel carries upstream PyTorch's entire Python tree with torch._C replaced, so most of its files are upstream's, under upstream's terms, which is why pyproject.toml's license is torch 2.13.0's own License-Expression, verbatim:

Apache-2.0 AND Apache-2.0 WITH LLVM-exception AND BSD-2-Clause
AND BSD-3-Clause AND BSL-1.0 AND MIT

scripts/wheel/build.py ships torch's dist-info, third-party notices included.