toktier

English | 简体中文

Tokenize the conversation once — after that, only what's new.

TokTier is a stateful tokenization system for agentic LLM serving. It keeps per-session token state, repairs appended text with a certified CPU path, and offers a certified GPU path for fresh or large requests. Both fast paths return token IDs bit-identical to a full Hugging Face (HF) tokenizers encode from scratch.

• Exact, at scale. The release campaign records 57.0 billion checks across 15 tokenizer artifacts and 3.8 billion real documents (12.33 trillion characters), with zero observed divergence.
• Fast on both paths. On the recorded benchmark battery, the GPU path encodes a fresh 4-million-character request (~786K tokens) in 3.88 ms, and the bounded CPU repair for a 256-character append to a 4.19M-character session takes 1.68 ms. The benchmark protocol excludes engine construction, so the 3.88 ms result assumes an already constructed and prepared engine; it is not a cold first-call figure. The repair reading measures the repair operation itself; it excludes materializing the full historical token sequence as a Python tuple.
• Certified before acceleration. Fast paths run only for the exact tokenizer artifacts, under the reference versions, that recorded evidence covers. Under the default SUPPORTED policy a device architecture or compiler no campaign has judged still runs when every constraint the registry binds verifies, labelled supported_untested rather than certified. explain() reports the route and its reasons.

Every bar is a measured median. The 1.68 ms reading above comes from the toktier repair (HF tokenizers window) lane, which is the repair window measured in that cell; the corrected-Gigatoken window is a separate lane in the same figure (2.39 ms on the 65,536-character append). Both are bounded repairs under the routing table below, and the figure data names the lane of each bar. A native Rust serving integration can avoid the full-sequence materialization by retaining session state and consuming only the repaired suffix. Exact values, workload sizes, and sample counts are in hero_session_vs_reencode.data.json; the complete sweeps are in docs/benchmarks.md.

News

• 2026.10.15 🚀 toktier 0.3.0 released — one long request can now use more than one core on the CPU route without leaving the exact-ID contract. toktier.chunked.encode cuts the input at certified stable boundaries, encodes the chunks in parallel and concatenates the ids, which is the serial encoding by construction; it runs as a Python helper over the batch path or inside a native chunk encoder, and the Rust crate reaches the same encoder through Tokenizer::chunked(). Which artifacts may be chunked is read from a table that ships with the package and is generated from evidence on the chunked path itself, and three levels of evidence are kept apart: the corrected CPU engine's whole-document campaign (41,800,181,401 checks, zero divergence) is about the engine that encodes each chunk and never cut a document, so it is not chunked-path evidence; the chunked path holds an adversarial battery (460,000 strings over the eleven roster artifacts, both paths, 2026-10-15) and a web-corpus cut comparison (102,400,000 documents each for four artifacts, Python path, 2026-09-17), neither with a difference from the reference. The four artifacts that hold both levels, qwen3_8b, llama_3_1_8b, deepseek_v3 and gpt_oss_120b, are admitted, and on them the native encoder runs by default on both faces, labelled native_evidence:adversarial_only until the corpus comparison covers the native path too. The other seven roster artifacts hold the battery only and are encoded serially until the corpus campaign covers them (R_CHUNK_FAMILY_UNSUPPORTED, refusal chunked_evidence_incomplete:adversarial_only): a measurement not yet taken, not a measured problem, and their whole-document fast CPU routes are unchanged. An added token with an lstrip, rstrip or single_word flag is not admitted (added_token_strip_or_single_word), and an installed tokenizers outside the certified set keeps the call serial (oracle_differs_from_chunked_evidence:<version>). In cold readings on one 16-core desktop (2026-10-15) the native encoder ran 1.35 to 5.25 times as fast as the serial encode with lookup="off" on four to eight cores, for texts of 262,144 to 4,194,304 characters, least on English text for Llama-3.1-8B. The release also adds Session.append_patch, which returns the splice without a copy of the stream, and an SGLang adapter (toktier.integrations.sglang) that serves SGLang's encode calls on the certified CPU route once it has checked that the tokenizer it wraps is the one toktier loaded (TOKTIER_SGLANG_IDENTITY accepts only require), with an optional pool of tokenizer worker processes, chunked dispatch for long stateless requests and a chunk-encoder warm-up when it wraps an admitted tokenizer; the TOKTIER_SGLANG_* settings are read by the adapter alone. It was checked on SGLang 0.5.19, 0.5.20 and 0.5.21 on 2026-10-13, and on 2026-10-15 against a real SGLang 0.5.21 model server, where the ids of all 138 requests equaled the Hugging Face ids. The exact tokenizers and transformers pins become ranges (tokenizers>=0.22.2,<0.24, transformers>=4.57.6,<6), so pip install toktier beside SGLang resolves the ordinary way (a new environment also gets platformdirs), and pip check reported no broken requirements beside SGLang 0.5.21 and beside SGLang main's pins (checked on 2026-10-14 and 2026-10-15). The certificates speak for the certified reference versions tokenizers 0.22.2, 0.23.2 and 0.23.3, the two 0.23 releases install beside a transformers loader of the 5.19 line, and an installed tokenizers outside that set runs on the reference route. On the Rust crate every third-party dependency takes a caret edge (most were exact before; Cargo.lock is unchanged); the fast CPU engine is admitted by its source identity under every policy but Reference, so a build on another rustc or feature recipe runs it labelled supported_untested, with rust-version at 1.89 and the judged toolchain still 1.93.1, and the Python facade follows the same rule; a reference library outside its certified versions is a label on the CPU route, while the GPU route keeps its certified-build requirement; and Runtime::family_for_sha256 names the shipped family of a tokenizer.json digest. The prebuilt GPU delivery is certified on sm_80, sm_89 and sm_120; sm_90 is not certified in this release (the fatbin keeps its image as experimental, and certification is planned for 0.3.1). Because the native chunk encoder is compiled into the corrected CPU engine, all three source identities move, exact and version-normalized alike, and the roster is recertified against them; artifacts and capability ids do not move. A persistent session store written by 0.2.9 is refused at open with SESSION_STATE_MISMATCH, and 0.2.9 refuses one 0.3.0 wrote: point store= at a new directory and replay the transcript. The store format and the artifact fetch manifest do not change, so a 0.2.9 artifact cache serves 0.3.0, offline included. With the same tokenizers and transformers installed, served ids are unchanged on every packaged artifact (qwen3_5_08b beside a transformers 5.x loader follows that loader and reports R_LOADER_FACE_MISMATCH). doctor --json gains three top-level keys and one family key, the reason-code namespace gains R_CHUNK_NO_CUT, R_CHUNK_FAMILY_UNSUPPORTED, R_CHUNK_ROUTE_NOT_FAST_CPU and R_LOADER_FACE_MISMATCH, and explain() gains a chunked block and the loader-face digest. Same-host A/B runs against 0.2.9 kept the encode, session, serving and GPU timings of both faces within a few percent, so the paths this release did not add are not faster; Runtime::doctor() takes about 0.6 to 0.8 microseconds more to report its new facts. The chunk-parallel path, the CPU pool and the SGLang adapter are described in the Delta Tokenizer paper (arXiv:2610.19801); TokTier (arXiv:2607.29678) remains the citation for the system. See the v0.3.0 release notes.
• 2026.10.14 📄 Our paper Delta Tokenizer: Reference-Identical Tokenization of Session Deltas and Segment-Parallelism for Long Prompts on Multicore CPUs is available on arXiv.
• 2026.08.31 🚀 toktier 0.2.9 released — both capability fingerprints are now computed on the loader face the pinned loader materializes rather than on the artifact file, so a capability id names the document the certification readings were taken against. Six cells of the shipped registry move — five pipeline.* ids and the added-frontend.* id of qwen3_5_08b — and the release notes map old to new; no served id changes on any input over any packaged artifact, artifact_sha256 still keys every certification lookup, and neither runtime recomputes a fingerprint. On that shared face Qwen/Qwen3.8-27B and Qwen/Qwen3.8-Flash-Next are admitted by content through from_pretrained() onto the qwen3_5_08b anchor, recorded on a fifth sibling basis equivalent_loader_face (215 registry rows, 214 siblings, 208 packaged); no sixteenth family id is added, so load("qwen3_8_27b") still reports ARTIFACT_NOT_FOUND. This release also fixes a defect the published 0.2.8 carries: the shipped artifact manifest named only tokenizer.json, so a cache built from nothing by artifacts fetch lacked the sidecar that declares the seven added tokens of qwen3_5_08b (ids 248070-248076, <tts_pad> at 248072), and toktier.load("qwen3_5_08b") failed closed with ARTIFACT_HASH_MISMATCH, reason config_added_tokens_mismatch. The manifest now pins the loader face's whole input closure, 15 file lines to 46, measured rather than assumed and proved by rematerializing from the recorded group alone. An offline deployment wants that wider set staged before it upgrades: qwen3_8b, for one, goes from one file to three. A connected host repairs itself on the first load(); an offline host raises ARTIFACT_NOT_FOUND, and toktier artifacts fetch <family> --force needs the network. Its route is toktier artifacts export <family> --out <family>.tar on a connected 0.2.9 host, then toktier artifacts import on the offline one — move the existing alias directory aside first, or the import answers ALIAS_CONFLICT over the older, smaller tree already there. A persistent session store written by 0.2.8 is refused at open with SESSION_STATE_MISMATCH, as 0.2.8 was for 0.2.7 and for the same reason: point store= at a new directory and replay the transcript. Smaller movements: an import clears the verified marker's own leftover temporary instead of refusing over it; a directory holding neither config.json nor tokenizer_config.json is materialized file-only rather than through a loader class guessed from the directory path, so resolution no longer depends on where the cache sits; doctor reads the engine binding for automatic_effective_backend and names the CPU lane it refuses; the support-registry schema drops two optional carry-over fields no document ever carried; and chacha20 moves to 0.10.2, the version crates.io has not withdrawn. The registry schema version stays 1 and the fingerprint preimages and domain tags are unchanged, though the two JSON shapes just named — the sibling basis vocabulary and that schema — do move under it. The ALIAS_CONFLICT code and its details, the store format, the shipped fatbin, and the kernel ABI are unchanged. See the v0.2.9 release notes.
• 2026.08.30 🚀 toktier 0.2.8 released — on the Python face the certified subject is now the loader face: tokenizer.json plus the configuration-only added tokens that the pinned transformers.AutoTokenizer materializes. The seven configuration-only literals of qwen3_5_08b, at ids 248070-248076, now return their declared single ids under every Python policy, where REFERENCE used to spell them as ordinary text; the Rust crate still executes the artifact file, so its answers for those literals do not change. The declared subset is verified fail-closed, and a drifted sidecar raises ARTIFACT_HASH_MISMATCH with reason config_added_tokens_mismatch. Re-importing an unchanged air-gap bundle is now idempotent, and a conflicting alias raises the new stable ALIAS_CONFLICT code. Diagnostics add automatic_routing_policy, guarantee_basis.family_ids, and, on Rust verify-local --json, route_admitted and unserved_paths; under REFERENCE, doctor now answers false/hf instead of true/gpu. pip install "toktier[fastokens]" resolves toktier-fastokens==0.3.1.2, and 0.3.1.1 stays recognized. A persistent store written by 0.2.7 is refused with SESSION_STATE_MISMATCH because the semantic fingerprint moved: point store= at a new directory and replay the transcript. The store format, the shipped fatbin, and the kernel ABI are unchanged, and existing Python facade calls remain source-compatible. See the v0.2.8 release notes.
• 2026.08.27 🚀 toktier 0.2.7 released — pip install "toktier[fastokens]" now installs toktier-fastokens, a pinned build of fastokens 0.3.1 with five patches from this project. The project publishes the build, so the extra installs the same bytes described by the adapter's readings. The adapter resolves the installed engine by its import package and reports engine_assurance, while its admission remains experimental. For the published wheel, the report shows certified_pinned with a guarded exact_id_guarantee: true; otherwise, it identifies the premise that does not hold. The 154-code-point Unicode guard moved from the judge into the adapter. Served IDs, the store format, and the kernel ABI are unchanged. See the v0.2.7 release notes.
• 2026.08.27 🚀 toktier 0.2.6 released — Rust certification now speaks for the certified core (TokTier's own crates, the packages they call directly, and the text-semantics libraries beneath them). Drift elsewhere in the compiled closure is reported as an advisory with alignment commands and no longer withholds acceleration. The whole-closure reading stays available as dependency_closure. The libraries whose Unicode tables decide where text is cut are compared by the version of those tables rather than by their package version, and doctor reports each one. The property data the fast CPU pre-tokenizer reads is pinned to the Unicode version the reference engine carries, with an exhaustive gate that keeps the two equal. On the GPU side, driver and CUDA versions are reported as environment facts, and sm_80 and sm_90 join the certified prebuilt list on a bounded spot check. A device architecture or compiler toolchain unjudged by any campaign now runs under the new default supported policy and is labelled supported_untested. verify-local (the same command on both faces) compares such a route with the reference engine on your own text and records the answer as locally_verified. Served IDs, the store format, and the kernel ABI are unchanged. See the v0.2.6 release notes.
• 2026.08.15 🚀 toktier 0.2.5 released — the Rust crate's network feature is now opt-in, which takes sixteen packages and the whole TLS stack out of a default build that may never fetch; add features = ["network"] to keep acquiring artifacts over the network, and cargo install --locked --features network toktier for a CLI that fetches. The Python package acquires artifacts exactly as before; the feature change does not reach it. What does reach it is Session.revision, which is durable from this release: a conversation resumed in a later process reports the revision its record carries, and Tokenizer.store_session_revision() returns that number instead of None. Diagnostics gain an execution reason and a network_compiled build fact. Served IDs, the store format, and the kernel ABI are unchanged. See the v0.2.5 release notes.
• 2026.08.14 🚀 toktier 0.2.4 released — the Han family (kimi_k3) joins the certified roster with the product's own end-to-end GPU engine; Rust certification now judges the packages a build actually compiles, not only its sources, and its build flags claim only what a build script can observe; the Rust crate follows TOKTIER_HOME/XDG; the Python facade gains the session() context manager, --json on every command, and doctor --family; a directory root that cannot hold private state, artifacts check-conversion, and the last-execution diagnostic all answer inside their contracts. Served IDs, the store format, and the kernel ABI are unchanged. See the v0.2.4 release notes.
• 2026.08.11 🚀 toktier 0.2.1 released — a maintenance update: richer diagnostics (doctor now reports JIT toolchain eligibility, and explain() summaries state the time window each field covers) plus documentation fixes. Served IDs, the store format, and the kernel ABI are unchanged. See the v0.2.1 release notes.
• 2026.08.10 🚀 toktier 0.2.0 released — the first public release: certified exact-ID sessions with bounded CPU repair, a prebuilt GPU path, and the Rust serving API, shipped as a Python wheel on PyPI and six Rust crates on crates.io.
• 2026.07.31 📄 Our paper TokTier: Exact Stateful CPU+GPU Tokenization for Agentic LLM Serving is available on arXiv.

Quick start

Install with pip install toktier; GPU options are described in Install. On a GPU host where ninja is importable, the default gpu_delivery="auto" selects the JIT delivery, so the first long request compiles the kernel for tens of seconds; pass gpu_delivery="prebuilt" to use the shipped kernel or device="cpu" to stay on the CPU route (GPU delivery).

import toktier

tok = toktier.load("qwen3_8b")          # family id from the support matrix
enc = tok.encode("hello world")         # token IDs
print(enc.ids)
print(tok.decode(enc.ids))
print(tok.explain(summary=True))        # concise route and verdict

encode followed by decode does not necessarily reproduce the original text. For example, a tokenizer whose pipeline applies NFC normalization returns normalized text; that is the tokenizer's own behavior, not a TokTier divergence. TokTier's guarantee concerns token IDs: they match a from-scratch HF encode of the same input, and both decoders return the same text under the same decoding options. The options themselves differ by default: decode here skips special tokens (skip_special_tokens=True) where the HF decoder keeps them, so ids carrying special tokens decode to different text until one call is given the other's setting. Pass the same value to both and the two results are identical byte for byte.

If application code starts from a Hugging Face model repository instead of a TokTier family id, resolve it by content:

tok = toktier.from_pretrained("Qwen/Qwen3-0.6B")

from_pretrained() downloads the audited immutable revision for a recorded sibling or canonical repository, hashes the exact file, and consults the sibling registry, which contains 214 audited repositories plus one canonical self-row and is itself covered by a root digest. For an unknown repository, from_pretrained() resolves main unless revision= is passed. Byte-identical, source-level identical, canonicalization-equivalent, serialization-equivalent, and loader-face-equivalent records run on the already certified canonical artifact through the same CPU/GPU router. A known repository whose bytes have changed — and any unregistered content — stays on HF under policies that permit the reference fallback; REQUIRE_ACCELERATED raises an error instead. See explain()["model_resolution"] for both the source identity and the canonical identity used for execution. load(family) remains the direct family API and the air-gap-friendly path.

Sessions

Use session= to name a growing transcript and persist its token state across calls and processes:

tok = toktier.load("qwen3_8b", store="./toktier-store")

transcript = "user: hello\nassistant: hello! how can I help?\n"
enc = tok.encode(transcript, session="chat-42")

transcript += "user: what changed since my last call?\n"
enc = tok.encode(transcript, session="chat-42")

# Store-backed and from-scratch paths return the same IDs.
assert enc.ids == tok.encode(transcript, lookup="off").ids

Without session=, the store can find a byte-verified stored prefix by content. Use lookup="off" to skip that lookup. A failed byte check is a miss, never a trusted hit; cache eviction changes latency, not output. Long sessions whose stable prefix has been sealed remain reusable after restart: TokTier binds the record to the caller-presented historical prefix before restoring it, and a missing or corrupt binding results in a cold encode.

Routing and policies

Routing policy is selectable and inspectable:

from toktier import RoutingPolicy

tok = toktier.load("qwen3_8b", policy=RoutingPolicy.CERTIFIED)

The install profile and input shape then determine the automatic route:

The two GPU rows describe what happens when the GPU route is admitted, and admission is narrower than "a GPU is present". The shipped evidence covers sm_80, sm_89 and sm_120 for the prebuilt delivery. On any other architecture the default SUPPORTED policy still runs the shipped kernel when it loads and every constraint the registry does bind verifies, and labels that route supported_untested rather than certified; toktier verify-local compares it with the reference engine on your own text. The strict CERTIFIED policy refuses it instead, so those rows fall to the row above them. Either way explain() records the reason. The evidence scale per architecture is in docs/support-matrix.md.

explain(summary=True) reports:

• the headline route and certification verdict;
• what the last request actually did (last_execution_backend / _path / _source), plus last_execution_fallback when a request finished somewhere other than where it started; and
• whether any fallback has occurred over the process lifetime (fallback_ever_occurred, which also counts the ordinary below-threshold crossover).

The full no-argument explain() report adds the fixed chain, the crossover decision (gpu_min_bytes, 64 KiB by default), detailed probe and certification data, and every fallback counter.

Rust serving API

The workspace includes a Python-free Rust serving facade for frontends that retain token state directly. It exposes:

• pinned artifact fetch, mirror, and air-gap operations
• reference, corrected-CPU, and prebuilt-or-direct-JIT GPU routing
• continuous token buffers
• bounded, executor-neutral batching
• persistent named sessions
• delta-native TokenPatch results

use toktier::{Device, Runtime};

let runtime = Runtime::builder().device(Device::Auto).build()?;
let tokenizer = runtime.load("qwen3_8b")?;
let mut session = tokenizer.open_session("agent-42")?;
let seed = session.seed("user: hello\n")?;
let patch = session.append("assistant: hi\n")?;

patch.keep_tokens() says where a retained downstream ID buffer should be truncated; patch.replacement_ids() is the exact repaired suffix. The append does not allocate the complete historical ID sequence unless the caller asks for snapshot(). The crate is published on crates.io from 0.2.0 onward and tracks the package version, so cargo add toktier resolves it from the registry. See docs/rust-api.md for the serving surface and docs/rust-lifecycle.md for acquisition, JIT, concurrency, and reproducible offline distribution.

Since 0.1.1, the UTF-8 crossover and no-hit added-token prefilter execute in one allocation-free Rust selector call. On the recorded RTX 5090 host, its 4M-byte control-plane microprofile fell from 2.97 ms to 0.052 ms (57.5x); this is a routing-only measurement, separate from tokenization and Python result materialization. See docs/native-routing.md.

Install

pip install toktier                 # complete certified CPU product
pip install "toktier[gpu]"          # CPU product + automatic prebuilt GPU route
pip install "toktier[gpu-jit]"      # same routing, with local JIT delivery
cargo add toktier                   # Python-free Rust serving API
cargo add toktier --features network # ... plus artifact acquisition over TLS

The Python package is unaffected by the Rust crate's features: it fetches artifacts through huggingface-hub as it always has. On the Rust side, network is opt-in from 0.2.5; without it the crate still verifies, mirrors, imports, exports, and runs from a verified cache.

Both GPU extras pull in torch and its CUDA wheels, so budget for it: a fresh [gpu] or [gpu-jit] virtual environment measures around 5 GiB, and an uncached install downloads several wheels in the hundreds-of-megabytes range (plus a comparable pip cache). That is the Torch ecosystem's footprint, not TokTier's — the base toktier wheel needs none of it.

JIT toolchain certification

JIT is fail-closed on every binding the registry records — sources, class tables, build flags. Certification additionally checks the actual nvcc selected by PyTorch's extension builder, torch.version.cuda, and the PyTorch distribution version as independent axes; for example, torch CUDA 13.0 with NVCC 13.2 is not treated as the judged NVCC 13.0 combination.

That triple is the one premise where a miss is a coverage gap rather than a failed check: the sources and flags are the judged ones and no campaign has compiled them with this pair. The default SUPPORTED policy therefore compiles and runs it and labels the route supported_untested, with the gap reported rather than warned about: toktier doctor and explain() both show the observed triple beside jit_toolchain_satisfied: false. CERTIFIED refuses it as it always has — automatic routing keeps to the corrected Gigatoken → HF fallback chain and records the reason, and an explicit CUDA request fails with the observed compiler/runtime triple, the certified constraint, and a copyable remedy.

Under either policy, a combination can be compiled ahead of first use with:

toktier gpu compile qwen3_8b

Under the stricter policies, an unjudged combination can still be compiled for evaluation with explicit risk acceptance:

toktier gpu compile qwen3_8b --accept-uncertified-jit

This does not certify the resulting kernel. That form runs under EXPERIMENTAL, prints an UNCERTIFIED JIT OPT-IN warning, and records every waived premise; under the default policy it is not needed, and the command waives nothing. Application code reaches the same treatment by opting in explicitly with policy="experimental", gpu_delivery="jit"; the acceptance is deliberately not persisted or inherited by later certified processes. Inspect explain()["experimental_waivers"] before using those results.

CPU engine provenance and build identity

The corrected, data-version-pinned Gigatoken implementation is linked directly into the core toktier._native extension. TokTier does not install or trust a top-level package named gigatoken, and the wheel carries no second CPU native module. The base wheel also requires the HF loader and oracle packages as ranges (tokenizers>=0.22.2,<0.24, transformers>=4.57.6,<6); the route is certified under the tokenizers releases the registry lists as certified reference versions (0.22.2, 0.23.2 and 0.23.3 in this release). There is no separate CPU-fast installation step.

For provenance, a source checkout can independently recompute the active source identity and build the same release profile:

python3 tools/fast_cpu_source_identity.py
python3 tools/compute_identity_v2.py
python3 tools/compute_identity_v2.py --show-diff
maturin build --locked --release

The three established identity scripts (fast_cpu_source_identity.py, native_host_source_identity.py, rust_api_source_identity.py) remain the byte-exact v1 view used by current build facts. compute_identity_v2.py hashes those same fast-CPU, native-host, and Rust-API coverage sets under new domains after normalizing only the enumerated workspace version fields; --show-diff prints every normalized line for review. python3 tools/dev.py check also rejects package-version reads in covered Rust or Python code outside the explicitly enumerated build-fact reporting sites, so tolerated metadata changes cannot select runtime behavior.

The provenance and build record pins the upstream commit, patch, Unicode inputs, compiler, and release flags. The executing extension reports its domain-separated source digest, Rust toolchain, and build flags; the registry verifies the source digest together with the repair configuration, oracle, and tokenizer artifact before opening the route, and reports whether the toolchain and flags are the judged recipe. The core wheel carries Gigatoken's MIT license, TokTier's modification notice, the dependency SBOM, and the dependency-license bundle.

TokTier is currently published as an ABI3 Linux x86-64 wheel, not an sdist. An install-time rebuild from the tagged source with another toolchain carries the same source identity and runs the fast CPU engine; what it does not carry is the judged recipe, which doctor reports as gigatoken_recipe_judged: false and explain() as a supported_untested entry (since 0.3.0; earlier releases failed closed on the toolchain). The tagged repository contains the complete source and pinned build record; sdist publication remains a separate release decision.

GPU delivery

The prebuilt fatbin contains sm_75/80/86/89/90/100/120 images and a compute_75 PTX fallback. Its binary-digest-bound certificate covers sm_80, sm_89 and sm_120; the other embedded architectures, sm_90 among them in this release, are marked experimental. With the default facade, toktier[gpu] selects this prebuilt delivery and toktier[gpu-jit] selects JIT from the detected profile; an explicit gpu_delivery= argument can override that detection. Under prebuilt delivery, the GPU engine opens lazily on the first request that routes to the GPU at or above the crossover, so explain()["gpu_backend"]["loaded"] stays false while only short requests have run; the crossover decides per input which backend executes. JIT delivery keeps the Python host, whose GPU backend opens the same way on the first input that routes to the GPU. The JIT delivery is certified_source on sm_89 and sm_120, meaning its certificate binds source, class tables, flags, and toolchain constraints rather than a machine-local binary. See docs/gpu-jit.md for the automatic facade, explicit engine API, and delivery diagnostics.

Tokenizer artifacts, mirrors, and air-gapped hosts

Tokenizer artifacts are fetched from pinned upstream revisions and verified by SHA-256; they are not bundled in the wheel. The CLI supports connected, mirrored, and air-gapped environments.

On the connected host, fetch the pinned artifact and pack it:

toktier artifacts fetch qwen3_8b
toktier artifacts export qwen3_8b --out qwen3_8b.tar

Copy qwen3_8b.tar across, then on the air-gapped host unpack and check it:

toktier artifacts import qwen3_8b.tar
toktier artifacts verify qwen3_8b
toktier inspect qwen3_8b
toktier doctor --json

The two halves belong on two machines, or at least on two caches: import installs the alias into the cache it resolves. A cache that already holds that alias is re-read rather than overwritten. If the installed tree still authenticates as exactly this bundle — every declared path, byte count and SHA-256, and nothing undeclared beyond the verification marker toktier writes there itself — the import is idempotent and returns the directory that is already there, which is what running the recipe twice, or importing on the connected host where fetch just placed the same bytes, reaches. If it holds anything else the import stops and names one file, rather than replacing bytes someone else may be using: an entry the bundle does not declare, at any depth, if the tree holds one, and otherwise the first declared file that is missing or does not match. The bundle itself verified either way.

This recipe transports tokenizer artifacts, and only those. A genuinely disconnected host also needs the TokTier wheel and every dependency wheel staged separately (a wheelhouse or a local index); the bundle format carries no Python distributions.

Doctor: what will actually run here

toktier doctor is probe-only and never loads a CUDA kernel. Its devices entries report index, name, and architecture; driver_version reports the driver observed by the shared host probe; and automatic_gpu_delivery_certification maps each observed architecture to the status of the selected installation-profile delivery. cuda_available reports whether TokTier's CUDA runtime binding is installed; cuda_hardware_present reports whether the device probe found at least one usable CUDA device.

It also answers "what will actually run here?" without constructing a tokenizer or compiling anything:

Those fields describe the installation. toktier doctor --family FAMILY adds a family section that answers for one family on that installation — its certification identity, its fast_cpu and GPU statuses, and two effective backends, one at or above the crossover and one below it. The second is where families differ: one whose CPU lane is the reference engine reads hf below the crossover while the installation-level field, correctly, reads fast_cpu.

So a toktier[gpu-jit] install on an unjudged compiler reports automatic_gpu_candidate: true beside jit_toolchain_satisfied: false, and then answers for the policy in effect: under the default SUPPORTED policy automatic_gpu_eligible: true and automatic_effective_backend: gpu, because that policy runs the unjudged combination and labels it supported_untested; under CERTIFIED the same machine reads false and fast_cpu. Either way it is the conclusion toktier gpu compile and the next request reach, before you run them.

certified_cpu_profile_ready says whether the installed engine reported its facts and the installed tokenizers is one of the certified reference versions, not whether a route is open. certified_reference_versions lists that set as the shipped registry records it, and reference_certification_reason is null when the installed release is in it and uncertified_reference when it is not; an installed release outside the set still runs, on the reference route, and family.plan_reasons names the same comparison as R_ORACLE_MISMATCH. The transformers release is not a premise of this key: the loader is admitted as a range and judged per artifact by the recorded loader-face digest when a tokenizer is loaded, which is what a loaded tokenizer's explain() shows (Installing beside SGLang).

Caches, state, and directory layout

The core package has no torch dependency and can be imported without CUDA, network access, or hardware probing. Artifact caches, compiled-kernel caches, and persistent session state use separate directories: the two caches follow XDG_CACHE_HOME and the session store follows XDG_STATE_HOME, because state is not a cache. Relocating everything at once is what TOKTIER_HOME is for (docs/contracts/config.md Section 5).

Since 0.2.4 the Rust crate reads the same variables with the same precedence, so one environment places both layers:

The leaf names stay the crate's own — artifacts land beside the Python product's, JIT products in jit-rust, deliberately apart from the Python kernels directory, because the two hold different things. Persistent sessions with no resolvable home are refused rather than placed by guesswork: state is not a cache. See docs/rust-lifecycle.md.

Experimental: the pinned Fastokens adapter

pip install "toktier[fastokens]" installs toktier-fastokens, a pinned build of fastokens 0.3.1 with six patches from the toktier project: five close defects, and one records in each modified file that it was modified. The project publishes this build on PyPI. The adapter still requires explicit selection:

tok = toktier.load(
    "qwen3_8b", policy="experimental", repair_backend="fastokens"
)

Two things are reported separately. certification: experimental says how it is admitted (explicit opt-in, never automatic); engine_assurance says what is known about the installed engine. When the installed engine's bytes match those in the wheel toktier published, engine_assurance is certified_pinned and exact_id_guarantee is true in the guarded sense: the IDs equal those from the pinned reference, or the adapter's Unicode guard routes the request to it. The comparison uses the engine digest, not the wheel builder's identity. A build whose digest is not among the published ones reports false. This includes the upstream wheel and, usually, a build from the sdist produced on another host or with another toolchain. A build with an identical digest reads exactly as the published one does. The pinned distribution keeps the upstream import name, so having both installed is not recommended: they share the files that import fastokens runs, and whichever was installed last owns them (toktier doctor reports which one that is, and names the other as co-installed). If the upstream distribution is already installed, reinstall rather than remove one of the two, because uninstalling either removes the files they share:

pip uninstall -y fastokens toktier-fastokens && pip install "toktier[fastokens]"

If other code needs the upstream distribution, use a separate environment. The two share one import name, so only one of them can own the bytes that import fastokens runs: installing both leaves the files belonging to whichever was installed last, and uninstalling either removes them for both. toktier doctor reports the other as co-installed rather than pretending it is not there.

The readings behind certified_pinned were taken on the published wheel (engine digest 2c4f48450a866e3f...). Across 15 tokenizer artifacts, 998,857,881 documents per artifact were checked against tokenizers==0.22.2, using eight visible CPUs, with zero guarded mismatches and zero engine errors. The same wheel was also used for a stateful-replay gate, a six-topology gate, and a splice/edit gate. The five patches close five defects we observed in the upstream 0.3.1 code. Of these defects, one raises an error on a rare character, and four are silent ID divergences. Reports on these defects have been submitted to the upstream project. The guard covers 154 combining marks that the frozen reference does not reorder; a request containing one is answered by the reference and counted as routed. docs/support-matrix.md carries the full digests, the states the adapter reports and what each one means.

Correctness and evidence

The certified reference is Hugging Face tokenizers 0.22.2 with default settings: the reference engine compiled into the native extension is that release exactly, and the roster certification readings use it as the reference; 0.23.2 and 0.23.3 have their own shipped reference-compatibility readings (readings/oracle_compat/). Certification is bound to exact artifact bytes and to the certified reference versions, the exact tokenizers releases the support registry's oracle record lists; a later release joins that set only on evidence that it agrees with the embedded reference on the certified surface, and the package metadata admits a range of releases so that it can be installed beside other pins. If the installed tokenizers is outside the certified set, accelerated routing is disabled and the request remains on the installed reference path.

One related definition, settled in 0.2.8: the reference is the loader face. An artifact's identity key is its tokenizer.json, and the certified subject covers that file plus the added tokens its tokenizer_config.json declares beyond it -- the object transformers.AutoTokenizer materializes, which is what loader-based serving stacks compare against. Accelerated and reference routes read that one subject and return the same IDs for inputs holding such a literal; the existing campaign readings carry over unchanged because a dedicated scan found zero occurrences of every affected literal in the certified corpora. docs/support-matrix.md records the one artifact whose two faces differ as documents and the carry-over evidence.

Four different counts appear in this document, and each answers a different question: 15 packaged artifacts (what toktier inspect lists), 15+3 model families (byte-level BPE plus WordPiece, since families can share an artifact), 11 artifacts with a certified CPU fast path (12 families by exact-artifact inheritance), and 215 registry rows (214 audited sibling repositories plus one canonical self-row). Numbers that appear inconsistent usually belong to different axes.

The machine-readable records are evidence/evidence_manifest.json, evidence/evidence_manifest_added_families.json, evidence/evidence_manifest_kimi_band.json, and tables/support_registry.json. Shipped per-artifact readings account for 53,720,215,504 checks; an archived earlier phase accounts for the remaining 3,280,031,861. Together they produce the headline total above. What ships is that per-artifact summary: the document-by-document ledger the campaigns were reduced from lives in the audit records behind this document and does not ship inside the package. A focused end-to-end rerun through the historical public session API is kept in readings/fast_cpu_focused_parity.json. The executing one-call Rust front end is separately checked across all 11 CPU-fast artifacts in readings/fast_cpu_native_frontend_parity.json.

Three status values keep evidence and runtime behavior distinct:

These are empirical differential results, not a proof over all possible inputs. Per-request checks and the reference fallback remain part of the contract.

The chunk-parallel encode added in 0.3.0 has evidence of its own, kept apart from the table above, whose campaigns encode whole documents and never cut one: an adversarial battery and a corpus cut comparison on the chunked path, from which a shipped admission table decides, artifact by artifact, what may be chunked (docs/support-matrix.md).

Repository self-checks:

pip install pytest==9.1.1 jsonschema==4.26.0    # or: pip install --group test
python3 tools/generate_evidence.py --check
python3 tools/verify_carryover.py --check
python3 tools/generate_native_legal.py --check    # needs cargo
python3 tools/validate_registry.py tables/support_registry.json
python3 tools/generate_registry.py --release-check
python3 tools/generate_sibling_aliases.py --check
python3 tools/dev.py test-packaging

Five of these commands also run from the published Rust source archive, which carries evidence/, data/, and this README's translation alongside the sources. Two are repository-only. In the archive, they decline rather than fail without explanation: each prints a line beginning declined:, says that nothing was checked, and exits 3 — neither a pass nor a finding. generate_registry.py --release-check reads the repository's own sources and its built extension, and the archive has neither; the copy of the registry it carries is verified by validate_registry.py, which does run there. dev.py test-packaging runs the test suite, which the archive deliberately does not carry. The two trees are told apart by the SOURCE-MANIFEST.json the archive builder writes at its root and nowhere else, so a repository checkout runs all seven exactly as before.

The prerequisites are stated here so that a failure means a real problem rather than a missing tool. The schema checks need jsonschema (the test dependency group in pyproject.toml); without it they refuse with error: the jsonschema package is required .... The last command runs the packaging test suite and also needs pytest, which the same group carries — the first line above installs both pins directly and covers every command in the block. The pip install --group alternative reads that group from pyproject.toml and needs pip 25.1 or newer (PEP 735).

generate_native_legal.py reads the workspace's locked resolve graph and needs cargo (the toolchain pinned by rust-toolchain.toml). On a fresh checkout, populate the local Cargo cache once with cargo fetch --locked (network required; a plain cargo build is not enough, since the legal closure covers every target) — the check itself then runs offline. generate_registry.py reads the native host's compile-time identity from the built extension. It uses this tree's src/toktier/_native when one has been built; otherwise, it uses the extension of an installed toktier wheel and says on stderr which one it read. The identity must equal the current source set either way. That second reading is available only as a convenience in a repository checkout — in the published source archive the command declines instead, rather than answering about an extension that belongs to some other installation on the machine. pytest tests/gpu follows the same rule: the two tests that assert on that identity read whichever extension is available and skip with a stated reason only when neither is.

Performance

The top figure compares the automatic GPU and repair routes with full re-encoding on identical text. The released batched GPU path also has a same-host throughput measurement:

This pair uses 2.2 GB of RAM-resident real web text and reports UTF-8 bytes over wall clock with host ID arrays materialized. Full protocols, all cells, and provenance are in docs/benchmarks.md.

The primary study used an RTX PRO 6000 Blackwell, but a consumer RTX 5090 sweep using the same protocol was 11–17% faster (4.24–5.50 GB/s across the reported families); that sweep measures the kernel's batched throughput rather than the end-to-end figure in the table above, so the two are not comparable cell by cell. Consumer hardware is therefore a practical target, not a reduced mode: the RTX 4090 also passed the sm_89 correctness and prebuilt-delivery battery. These observations do not promise the same speedup for every GPU; architecture, workload, and host delivery still matter.

The figures name Hugging Face (HF) tokenizers explicitly and link to the corresponding machine-readable docs/figures/*.data.json files. The benchmark document also shows the regimes where direct use of another engine is faster.

Support matrix

docs/support-matrix.md lists every anchor artifact, SHA-256, backend status, and 214 verified model repositories whose tokenizer is byte-identical to a certified one, identical at the upstream source file it is converted from, or equivalent after canonicalization, in serialization form, or on the loader face. Coverage follows tokenizer content, not repository naming. toktier.from_pretrained(repo_id) enforces that rule at runtime: it hashes the resolved file, maps registered content to the canonical artifact, and otherwise remains on HF.

The shipped registry holds 215 rows: those 214 siblings plus moonshotai/Kimi-K3 itself, so resolving the canonical repository by name reports itself as the evidence repository rather than a byte-identical sibling. 208 of the rows map to canonical artifacts present in this wheel. The other 7 are WordPiece rows, whose canonical artifacts are not packaged and which therefore run through HF. The 13 source-level kimi_k3 rows are among the 208: their canonical artifact is derived on your machine from pinned upstream bytes, so the comparison stays at tiktoken.model level while the loaded object is the certified conversion. toktier inspect is the authoritative packaged-family list.

Relation to existing work

Incremental BPE work studies how the merge stage can be extended as bytes arrive. TokTier operates one layer above it: session state contains token IDs and spans for the complete tokenizer pipeline — normalization, pre-tokenization, merges, and added-token handling — and an append is accepted only after its boundary check passes.

Serving projects such as llm-tokenizer and NVIDIA Dynamo's dynamo-tokenizers also cache encodings. The main interface differences are:

See docs/integration/dynamo.md for using the two layers together.

Documentation

• docs/releases/v0.3.0.md — release notes for this version.
• ARCHITECTURE.md — layers, routing, and store format.
• docs/serving.md — which encode call for which request shape, patch-only appends, and chunk-parallel encode for long requests.
• docs/integration/sglang.md — using toktier as an SGLang tokenizer backend.
• ROADMAP.md — release scope and planned integration.
• docs/support-matrix.md — artifacts and covered repositories.
• docs/gpu-jit.md — prebuilt and JIT GPU deliveries.
• docs/rust-api.md — Python-free Rust serving API.
• docs/rust-lifecycle.md — native artifacts, direct JIT, concurrency, and offline distribution.
• docs/integration/dynamo.md — Dynamo integration.
• docs/paper/toktier-preprint.pdf — current preprint.

Acknowledgements

TokTier's CPU Fast Pass and Fast Repair build on the excellent Gigatoken and Fastokens projects. We thank their authors and contributors for making this work openly available.

Corrected Gigatoken is the default certified repair-window engine for 11 unique tokenizer artifacts, covering 12 model families because NVIDIA Nemotron-Terminal ships a tokenizer that is byte-for-byte identical to the qwen3_8b tokenizer. TokTier's compatibility patch aligns Gigatoken's Unicode data and UTF-8 handling with the frozen Hugging Face tokenizers reference; the resulting path recorded 41.8 billion checks over 12.33 trillion characters with zero observed token-ID divergence.

The toktier project publishes toktier-fastokens, a pinned build of Fastokens 0.3.1 with six patches, for use with its explicitly selected experimental adapter. The upstream project is a separate implementation and does not endorse this build. Fastokens is Apache-2.0 and Gigatoken is MIT; exact revisions, license copies, patch series, and modification notices are in THIRD_PARTY_NOTICES and packaging/.

License and citation

toktier is licensed under the Apache License 2.0; see NOTICE for attribution information.

Paper: TokTier: Exact Stateful CPU+GPU Tokenization for Agentic LLM Serving · PDF

If you use toktier in your research, please cite:

@misc{zhang2026toktier,
  title         = {TokTier: Exact Stateful CPU+GPU Tokenization for Agentic LLM Serving},
  author        = {Zhenyu Zhang and Zhichao Cao},
  year          = {2026},
  eprint        = {2607.29678},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CL},
  doi           = {10.48550/arXiv.2607.29678},
  url           = {https://arxiv.org/abs/2607.29678}
}

For the chunk-parallel CPU path and the serving adapter, cite:

@misc{zhang2026deltatokenizer,
  title         = {Delta Tokenizer: Reference-Identical Tokenization of Session Deltas and Segment-Parallelism for Long Prompts on Multicore CPUs},
  author        = {Zhenyu Zhang and Zhichao Cao},
  year          = {2026},
  eprint        = {2610.19801},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CL},
  doi           = {10.48550/arXiv.2610.19801},
  url           = {https://arxiv.org/abs/2610.19801}
}

Machine-readable citation metadata is available in CITATION.cff.