toktier

English | 简体中文

Tokenize the conversation once — after that, only what's new.

TokTier is a stateful tokenization system for agentic LLM serving. It keeps per-session token state, repairs appended text with a certified CPU path, and offers a certified GPU path for fresh or large requests. Both fast paths return token IDs bit-identical to a full Hugging Face (HF) tokenizers encode from scratch.

• Exact, at scale. The release campaign records 57.0 billion checks across 15 tokenizer artifacts and 3.8 billion real documents (12.33 trillion characters), with zero observed divergence.
• Fast on both paths. On the recorded benchmark battery, the GPU path encodes a fresh 4-million-character request (~786K tokens) in 3.88 ms, and the bounded CPU repair for a 256-character append to a 4.19M-character session takes 1.68 ms. The benchmark protocol excludes engine construction, so the 3.88 ms result assumes an already constructed and prepared engine; it is not a cold first-call figure. The repair reading measures the repair operation itself; it excludes materializing the full historical token sequence as a Python tuple.
• Certified before acceleration. Fast paths are admitted only for the exact tokenizer artifact, oracle version, kernel delivery, and architecture covered by recorded evidence. explain() reports the route and its reasons.

Every bar is a measured median. The 1.68 ms reading above is the toktier repair (HF tokenizers window) lane, which is the repair window that cell measured; the corrected-Gigatoken window is a separate lane in the same figure (2.39 ms on the 65,536-character append). Both are bounded repairs under the routing table below, and the figure data names the lane of each bar. A native Rust serving integration can avoid the full-sequence materialization by retaining session state and consuming only the repaired suffix. Exact values, workload sizes, and sample counts are in hero_session_vs_reencode.data.json; the complete sweeps are in docs/benchmarks.md.

News

• 2026.08.15 🚀 toktier 0.2.5 released — the Rust crate's network feature is now opt-in, which takes sixteen packages and the whole TLS stack out of a default build that may never fetch; add features = ["network"] to keep acquiring artifacts over the network, and cargo install --locked --features network toktier for a CLI that fetches. The Python package acquires artifacts exactly as before; the feature change does not reach it. What does reach it is Session.revision, which is durable from this release: a conversation resumed in a later process reports the revision its record carries, and Tokenizer.store_session_revision() returns that number where it returned None. Diagnostics gain an execution reason and a network_compiled build fact. Served IDs, the store format, and the kernel ABI are unchanged. See the v0.2.5 release notes.
• 2026.08.14 🚀 toktier 0.2.4 released — the Han family (kimi_k3) joins the certified roster with the product's own end-to-end GPU engine; Rust certification now judges the packages a build actually compiles, not only its sources, and its build flags claim only what a build script can observe; the Rust crate follows TOKTIER_HOME/XDG; the Python facade gains the session() context manager, --json on every command, and doctor --family; a directory root that cannot hold private state, artifacts check-conversion, and the last-execution diagnostic all answer inside their contracts. Served IDs, the store format, and the kernel ABI are unchanged. See the v0.2.4 release notes.
• 2026.08.11 🚀 toktier 0.2.1 released — a maintenance update: richer diagnostics (doctor now reports JIT toolchain eligibility, and explain() summaries state the time window each field covers) plus documentation fixes. Served IDs, the store format, and the kernel ABI are unchanged. See the v0.2.1 release notes.
• 2026.08.10 🚀 toktier 0.2.0 released — the first public release: certified exact-ID sessions with bounded CPU repair, a prebuilt GPU path, and the Rust serving API, shipped as a Python wheel on PyPI and six Rust crates on crates.io.
• 2026.07.31 📄 Our paper TokTier: Exact Stateful CPU+GPU Tokenization for Agentic LLM Serving is available on arXiv.

Quick start

Install with pip install toktier; GPU options are described in Install.

import toktier

tok = toktier.load("qwen3_8b")          # family id from the support matrix
enc = tok.encode("hello world")         # token IDs
print(enc.ids)
print(tok.decode(enc.ids))
print(tok.explain(summary=True))        # concise route and verdict

encode followed by decode does not necessarily reproduce the original text. For example, a tokenizer whose pipeline applies NFC normalization returns normalized text; that is the tokenizer's own behavior, not a TokTier divergence. TokTier's guarantee concerns token IDs: they match a from-scratch HF encode of the same input, and both decoders return the same text.

If application code starts from a Hugging Face model repository instead of a TokTier family id, resolve it by content:

tok = toktier.from_pretrained("Qwen/Qwen3-0.6B")

from_pretrained() downloads the audited immutable revision for a recorded sibling or canonical repository, hashes the exact file, and consults the sibling registry -- 210 audited repositories plus one canonical self-row -- which is itself covered by a root digest. For an unknown repository, from_pretrained() resolves main unless revision= is passed. Byte-identical, canonicalization-equivalent, and serialization-equivalent records run on the already certified canonical artifact through the same CPU/GPU router. A known repository whose bytes have changed — and any unregistered content — stays on HF under policies that permit the reference fallback; REQUIRE_ACCELERATED raises an error instead. See explain()["model_resolution"] for both the source identity and the canonical identity actually executed. load(family) remains the direct family API and the air-gap-friendly path.

Sessions

Use session= to name a growing transcript and persist its token state across calls and processes:

tok = toktier.load("qwen3_8b", store="./toktier-store")

transcript = "user: hello\nassistant: hello! how can I help?\n"
enc = tok.encode(transcript, session="chat-42")

transcript += "user: what changed since my last call?\n"
enc = tok.encode(transcript, session="chat-42")

# Store-backed and from-scratch paths return the same IDs.
assert enc.ids == tok.encode(transcript, lookup="off").ids

Without session=, the store can find a byte-verified stored prefix by content. Use lookup="off" to skip that lookup. A failed byte check is a miss, never a trusted hit; cache eviction changes latency, not output. Long sessions whose stable prefix has been sealed remain reusable after restart: TokTier binds the record to the caller-presented historical prefix before restoring it, and a missing or corrupt binding becomes a cold encode.

Routing and policies

Routing policy is selectable and inspectable:

from toktier import RoutingPolicy

tok = toktier.load("qwen3_8b", policy=RoutingPolicy.CERTIFIED)

The install profile and input shape then determine the automatic route:

The two GPU rows describe what happens when the GPU route is admitted, and admission is narrower than "a GPU is present": the certified policy opens only the architectures the shipped evidence covers -- sm_89 and sm_120 for the prebuilt delivery -- so on any other architecture those rows fall to the row above them and explain() records the reason. The evidence scale per architecture is in docs/support-matrix.md.

explain(summary=True) reports:

• the headline route and certification verdict;
• what the last request actually did (last_execution_backend / _path / _source), plus last_execution_fallback when a request finished somewhere other than where it started; and
• whether any fallback has occurred over the process lifetime (fallback_ever_occurred, which also counts the ordinary below-threshold crossover).

The full no-argument explain() report adds the fixed chain, the crossover decision (gpu_min_bytes, 64 KiB by default), detailed probe and certification data, and every fallback counter.

Rust serving API

The workspace includes a Python-free Rust serving facade for frontends that retain token state directly. It exposes:

• pinned artifact fetch, mirror, and air-gap operations
• reference, corrected-CPU, and prebuilt-or-direct-JIT GPU routing
• continuous token buffers
• bounded, executor-neutral batching
• persistent named sessions
• delta-native TokenPatch results

use toktier::{Device, Runtime};

let runtime = Runtime::builder().device(Device::Auto).build()?;
let tokenizer = runtime.load("qwen3_8b")?;
let mut session = tokenizer.open_session("agent-42")?;
let seed = session.seed("user: hello\n")?;
let patch = session.append("assistant: hi\n")?;

patch.keep_tokens() says where a retained downstream ID buffer should be truncated; patch.replacement_ids() is the exact repaired suffix. The append does not allocate the complete historical ID sequence unless the caller asks for snapshot(). The crate is published on crates.io from 0.2.0 onward and tracks the package version, so cargo add toktier resolves it from the registry. See docs/rust-api.md for the serving surface and docs/rust-lifecycle.md for acquisition, JIT, concurrency, and reproducible offline distribution.

Since 0.1.1, the UTF-8 crossover and no-hit added-token prefilter execute in one allocation-free Rust selector call. On the recorded RTX 5090 host, its 4M-byte control-plane microprofile fell from 2.97 ms to 0.052 ms (57.5x); this is a routing-only measurement, separate from tokenization and Python result materialization. See docs/native-routing.md.

Install

pip install toktier                 # complete certified CPU product
pip install "toktier[gpu]"          # CPU product + automatic prebuilt GPU route
pip install "toktier[gpu-jit]"      # same routing, with local JIT delivery
cargo add toktier                   # Python-free Rust serving API
cargo add toktier --features network # ... plus artifact acquisition over TLS

The Python package is unaffected by the Rust crate's features: it fetches artifacts through huggingface-hub as it always has. On the Rust side, network is opt-in from 0.2.5; without it the crate still verifies, mirrors, imports, exports, and runs from a verified cache.

Both GPU extras pull in torch and its CUDA wheels, so budget for it: a fresh [gpu] or [gpu-jit] virtual environment measures around 5 GiB, and an uncached install downloads several wheels in the hundreds-of-megabytes range (plus a comparable pip cache). That is the Torch ecosystem's footprint, not TokTier's — the base toktier wheel needs none of it.

JIT toolchain certification

JIT is fail-closed at the toolchain boundary. Certification checks the actual nvcc selected by PyTorch's extension builder, torch.version.cuda, and the PyTorch distribution version as independent axes. If that exact triple is not recorded in the registry, automatic routing emits a prominent warning and keeps using the corrected Gigatoken → HF fallback chain; an explicit CUDA request fails with the observed compiler/runtime triple, certified constraint, and a copyable remedy. For example, torch CUDA 13.0 with NVCC 13.2 is not treated as the judged NVCC 13.0 combination. A judged combination can be compiled ahead of first use with:

toktier gpu compile qwen3_8b

For evaluation only, an unjudged toolchain combination can be compiled with explicit risk acceptance:

toktier gpu compile qwen3_8b --accept-uncertified-jit

This does not certify the resulting kernel. The command runs under EXPERIMENTAL, prints an UNCERTIFIED JIT OPT-IN warning, and records every waived premise. Application code must also opt in explicitly with policy="experimental", gpu_delivery="jit"; the acceptance is deliberately not persisted or inherited by later certified processes. Inspect explain()["experimental_waivers"] before using those results.

CPU engine provenance and build identity

The corrected, data-version-pinned Gigatoken implementation is linked directly into the core toktier._native extension. TokTier does not install or trust a top-level package named gigatoken, and the wheel carries no second CPU native module. The base wheel also pins the HF loader and oracle versions needed to open this certified route; there is no separate CPU-fast installation step.

For provenance, a source checkout can independently recompute the active source identity and build the same release profile:

python tools/fast_cpu_source_identity.py
python tools/compute_identity_v2.py
python tools/compute_identity_v2.py --show-diff
maturin build --locked --release

The three established identity scripts (fast_cpu_source_identity.py, native_host_source_identity.py, rust_api_source_identity.py) remain the byte-exact v1 view used by current build facts. compute_identity_v2.py hashes those same fast-CPU, native-host, and Rust-API coverage sets under new domains after normalizing only the enumerated workspace version fields; --show-diff prints every normalized line for review. tools/dev.py check also rejects package-version reads in covered Rust or Python code outside the explicitly enumerated build-fact reporting sites, so tolerated metadata changes cannot select runtime behavior.

The provenance and build record pins the upstream commit, patch, Unicode inputs, compiler, and release flags. The executing extension reports its domain-separated source digest, exact Rust toolchain, and build flags; the registry verifies all of them together with the repair configuration, oracle, and tokenizer artifact before opening the route. The core wheel carries Gigatoken's MIT license, TokTier's modification notice, the dependency SBOM, and the dependency-license bundle.

TokTier is currently published as an ABI3 Linux x86-64 wheel, not an sdist. An arbitrary install-time rebuild would have a different toolchain/build identity and therefore fail closed until separately certified. The tagged repository contains the complete source and pinned build record; sdist publication remains a separate release decision.

GPU delivery

The prebuilt fatbin contains sm_75/80/86/89/90/100/120 images and a compute_75 PTX fallback. Its binary-digest-bound certificate covers sm_89 and sm_120; the other embedded architectures are marked experimental. With the default facade, toktier[gpu] selects this prebuilt delivery and toktier[gpu-jit] selects JIT from the detected profile; an explicit gpu_delivery= argument can override that detection. Under prebuilt delivery the GPU engine opens lazily, on the first request that routes to the GPU at or above the crossover, so explain()["gpu_backend"]["loaded"] stays false while only short requests have run; the crossover decides per input which backend executes. JIT delivery keeps the Python host, whose GPU backend opens the same way at the first input that routes to the GPU. The JIT delivery is certified_source on sm_89 and sm_120, meaning its certificate binds source, class tables, flags, and toolchain constraints rather than a machine-local binary. See docs/gpu-jit.md for the automatic facade, explicit engine API, and delivery diagnostics.

Tokenizer artifacts, mirrors, and air-gapped hosts

Tokenizer artifacts are fetched from pinned upstream revisions and verified by SHA-256; they are not bundled in the wheel. The CLI supports connected, mirrored, and air-gapped environments:

toktier artifacts fetch qwen3_8b
toktier artifacts export qwen3_8b --out qwen3_8b.tar
toktier artifacts import qwen3_8b.tar
toktier artifacts verify qwen3_8b
toktier inspect qwen3_8b
toktier doctor --json

This recipe transports tokenizer artifacts, and only those. A genuinely disconnected host also needs the TokTier wheel and every dependency wheel staged separately (a wheelhouse or a local index); the bundle format carries no Python distributions.

Doctor: what will actually run here

toktier doctor is probe-only and never loads a CUDA kernel. Its devices entries report index, name, and architecture; driver_version reports the driver observed by the shared host probe; and automatic_gpu_delivery_certification maps each observed architecture to the status of the selected installation-profile delivery. cuda_available reports whether TokTier's CUDA runtime binding is installed; cuda_hardware_present reports whether the device probe found at least one usable CUDA device.

It also answers "what will actually run here?" without constructing a tokenizer or compiling anything:

Those fields describe the installation. toktier doctor --family FAMILY adds a family section that answers for one family on it — its certification identity, its fast_cpu and GPU statuses, and two effective backends, one at or above the crossover and one below it. The second is where families differ: one whose CPU lane is the reference engine reads hf below the crossover while the installation-level field, correctly, reads fast_cpu.

So a toktier[gpu-jit] install on an unjudged compiler reports automatic_gpu_candidate: true beside jit_toolchain_satisfied: false, automatic_gpu_eligible: false, and automatic_effective_backend: fast_cpu — the same conclusion toktier gpu compile reaches, before you run it.

Caches, state, and directory layout

The core package has no torch dependency and can be imported without CUDA, network access, or hardware probing. Artifact caches, compiled-kernel caches, and persistent session state use separate directories: the two caches follow XDG_CACHE_HOME and the session store follows XDG_STATE_HOME, because state is not a cache. Relocating everything at once is what TOKTIER_HOME is for (docs/contracts/config.md Section 5).

Since 0.2.4 the Rust crate reads the same variables with the same precedence, so one environment places both layers:

The leaf names stay the crate's own — artifacts land beside the Python product's, JIT products in jit-rust, deliberately apart from the Python kernels directory, because the two hold different things. Persistent sessions with no resolvable home are refused rather than placed by guesswork: state is not a cache. See docs/rust-lifecycle.md.

Experimental: Fastokens comparison

Fastokens 0.3.1 is available only as an explicit experimental comparison:

tok = toktier.load(
    "qwen3_8b", policy="experimental", repair_backend="fastokens"
)

This adapter re-encodes the full session and reports exact_id_guarantee: false; it is never selected by the certified policy and is not covered by the 12.4 TB (12.33 trillion characters) corrected-Gigatoken claim.

Correctness and evidence

The certified reference is Hugging Face tokenizers 0.22.2 with default settings. Certification is bound to exact artifact bytes and that oracle version. If the installed HF version is outside the certified set, accelerated routing is disabled and the request remains on the installed reference path.

Four different counts appear in this document and each answers a different question: 15 packaged artifacts (what toktier inspect lists), 15+3 model families (byte-level BPE plus WordPiece, since families can share an artifact), 11 artifacts with a certified CPU fast path (12 families by exact-artifact inheritance), and 211 registry rows (210 audited sibling repositories plus one canonical self-row). A number that looks inconsistent with another usually belongs to a different one of those axes.

The machine-readable records are evidence/evidence_manifest.json, evidence/evidence_manifest_added_families.json, evidence/evidence_manifest_kimi_band.json, and tables/support_registry.json. Shipped per-artifact readings account for 53,720,215,504 checks; an archived earlier phase accounts for the remaining 3,280,031,861. Together they produce the headline total above. A focused end-to-end rerun through the historical public session API is kept in readings/fast_cpu_focused_parity.json. The executing one-call Rust front end is separately checked across all 11 CPU-fast artifacts in readings/fast_cpu_native_frontend_parity.json.

Three status values keep evidence and runtime behavior distinct:

These are empirical differential results, not a proof over all possible inputs. Per-request checks and the reference fallback remain part of the contract.

Repository self-checks:

pip install pytest==9.1.1 jsonschema==4.26.0    # or: pip install --group test
python tools/generate_evidence.py --check
python tools/verify_carryover.py --check
python tools/generate_native_legal.py --check    # needs cargo
python tools/validate_registry.py tables/support_registry.json
python tools/generate_registry.py --release-check
python tools/generate_sibling_aliases.py --check
python tools/dev.py test-packaging

Five of these run from the published Rust source archive as well, which carries evidence/, data/, and this README's translation alongside the sources. Two are repository-only. There they decline rather than fail mysteriously: each prints a line beginning declined:, says that nothing was checked, and exits 3 — neither a pass nor a finding. generate_registry.py --release-check reads the repository's own sources and its built extension, and the archive has neither; the copy of the registry it carries is verified by validate_registry.py, which does run there. dev.py test-packaging runs the test suite, which the archive deliberately does not carry. The two trees are told apart by the SOURCE-MANIFEST.json the archive builder writes at its root and nowhere else, so a repository checkout runs all seven exactly as before.

The prerequisites are stated here so that a failure means a real problem rather than a missing tool. The schema checks need jsonschema (the test dependency group in pyproject.toml); without it they refuse with error: the jsonschema package is required .... The last command runs the packaging test suite and also needs pytest, which the same group carries — the first line above installs both pins directly and covers every command in the block. The pip install --group alternative reads that group from pyproject.toml and needs pip 25.1 or newer (PEP 735).

generate_native_legal.py reads the workspace's locked resolve graph and needs cargo (the toolchain pinned by rust-toolchain.toml). On a fresh checkout, populate the local Cargo cache once with cargo fetch --locked (network required; a plain cargo build is not enough, since the legal closure covers every target) — the check itself then runs offline. generate_registry.py reads the native host's compile-time identity out of the built extension: it uses this tree's src/toktier/_native when one has been built, and otherwise the extension of an installed toktier wheel, saying on stderr which it read; the identity must equal the current source set either way. That second reading is a convenience of a repository checkout only — in the published source archive the command declines instead, rather than answering about an extension that belongs to some other installation on the machine. pytest tests/gpu follows the same rule: the two tests that assert on that identity read whichever extension is available and skip with a stated reason only when neither is.

Performance

The top figure compares the automatic GPU and repair routes with full re-encoding on identical text. The released batched GPU path also has a same-host throughput measurement:

This pair uses 2.2 GB of RAM-resident real web text and reports UTF-8 bytes over wall clock with host ID arrays materialized. Full protocols, all cells, and provenance are in docs/benchmarks.md.

The primary study used an RTX PRO 6000 Blackwell, but a same-protocol consumer RTX 5090 sweep was 11–17% faster (4.24–5.50 GB/s across the reported families). Consumer hardware is therefore a practical target, not a reduced mode: the RTX 4090 also passed the sm_89 correctness and prebuilt-delivery battery. These observations do not promise the same speedup for every GPU; architecture, workload, and host delivery still matter.

The figures name Hugging Face (HF) tokenizers explicitly and link to their machine-readable docs/figures/*.data.json files. The benchmark document also shows the regimes where direct use of another engine is faster.

Support matrix

docs/support-matrix.md lists every anchor artifact, SHA-256, backend status, and 210 verified model repositories that share an identical or serialization-equivalent tokenizer. Coverage follows tokenizer content, not repository naming. toktier.from_pretrained(repo_id) enforces that rule at runtime: it hashes the resolved file, maps registered content to the canonical artifact, and otherwise remains on HF.

The shipped registry holds 211 rows: those 210 siblings plus moonshotai/Kimi-K3 itself, so resolving the canonical repository by name reports itself as the evidence repository rather than a byte-identical sibling. 204 of the rows map to canonical artifacts present in this wheel. The other 7 are WordPiece rows, whose canonical artifacts are not packaged and which therefore run through HF. The 13 source-level kimi_k3 rows are among the 204: their canonical artifact is derived on your machine from pinned upstream bytes, so the comparison stays at tiktoken.model level while the loaded object is the certified conversion. toktier inspect is the authoritative packaged-family list.

Relation to existing work

Incremental BPE work studies how the merge stage can be extended as bytes arrive. TokTier operates one layer above it: session state contains token IDs and spans for the complete tokenizer pipeline — normalization, pre-tokenization, merges, and added-token handling — and an append is accepted only after its boundary check passes.

Serving projects such as llm-tokenizer and NVIDIA Dynamo's dynamo-tokenizers also cache encodings. The main interface differences are:

See docs/integration/dynamo.md for using the two layers together.

Documentation

• docs/releases/v0.2.5.md — release notes for this version.
• ARCHITECTURE.md — layers, routing, and store format.
• ROADMAP.md — release scope and planned integration.
• docs/support-matrix.md — artifacts and covered repositories.
• docs/gpu-jit.md — prebuilt and JIT GPU deliveries.
• docs/rust-api.md — Python-free Rust serving API.
• docs/rust-lifecycle.md — native artifacts, direct JIT, concurrency, and offline distribution.
• docs/integration/dynamo.md — Dynamo integration.
• docs/paper/toktier-preprint.pdf — current preprint.

Acknowledgements

TokTier's CPU Fast Pass and Fast Repair build on the excellent Gigatoken and Fastokens projects. We thank their authors and contributors for making this work openly available.

Corrected Gigatoken is the default certified repair-window engine for 11 unique tokenizer artifacts, covering 12 model families because NVIDIA Nemotron-Terminal ships a tokenizer that is byte-for-byte identical to the qwen3_8b tokenizer. TokTier's compatibility patch aligns Gigatoken's Unicode data and UTF-8 handling with the frozen Hugging Face tokenizers reference; the resulting path recorded 41.8 billion checks over 12.33 trillion characters with zero observed token-ID divergence.

Fastokens 0.3.1 remains an explicit experimental alternative. TokTier does not claim exact-ID equivalence for that adapter and never chooses it automatically. Fastokens is Apache-2.0 and Gigatoken is MIT; exact revisions, license copies, the Gigatoken patch, and modification notices are in THIRD_PARTY_NOTICES and packaging/.

License and citation

toktier is licensed under the Apache License 2.0; see NOTICE for attribution information.

Paper: TokTier: Exact Stateful CPU+GPU Tokenization for Agentic LLM Serving · PDF

If you use toktier in your research, please cite:

@misc{zhang2026toktier,
  title         = {TokTier: Exact Stateful CPU+GPU Tokenization for Agentic LLM Serving},
  author        = {Zhenyu Zhang and Zhichao Cao},
  year          = {2026},
  eprint        = {2607.29678},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CL},
  doi           = {10.48550/arXiv.2607.29678},
  url           = {https://arxiv.org/abs/2607.29678}
}

Machine-readable citation metadata is available in CITATION.cff.