snaptokens

Fast BPE tokenization in Rust. The fastest in the current generic BPE matrix, in fact.

snaptokens is an optimized fork of fastokens that is significantly quicker and simplified to work in sinter. This repo will be archived once ported to sinter.

Give snaptokens a local Hugging Face tokenizer.json and get token IDs back. Opt in to a .tkz copy when faster future loads matter.

The current twelve-tokenizer, thirteen-host comparison is 2.19× faster than Gigatoken, 14.96× faster than upstream fastokens, and 49.39× faster than Hugging Face by geometric mean of paired medians. Snaptokens wins 777/780 Gigatoken cells; the full report keeps all three losses and the no-Gemma sensitivity result visible.

Optimizations

• Hand-written scanners replace general regex machinery for recognized production tokenizer patterns, with a table-driven ASCII path and exact Unicode fallback.
• Split, ByteLevel transformation, and BPE are fused so raw bytes map directly to initial token IDs and cache hits skip ByteLevel work entirely.
• BPE uses precomputed flat lookups: byte-to-token IDs, all 65,536 initial byte pairs, open-addressed merge ranks, and CSR merge adjacency.
• Pre-tokenized text stays in one contiguous buffer with byte ranges instead of allocating a string for every piece.
• Thread-local and shared caches are amortized across whole chunks, while a fixed Rayon pool keeps worker caches warm and balances uneven BPE work.
• Non-ByteLevel tokenizers split only at vocabulary-proven unbridgeable byte pairs; byte-fallback models stay on the normal exact merge path.
• Opt-in .tkz sidecars cache validated native construction data. Across twelve models and thirteen hosts, direct .tkz loads are 1.55× faster than Snaptokens JSON and artifacts are 24.9% smaller by geometric mean.

Benchmarks

The 2026-07-31 suite uses DeepSeek R1, Gemma 3, GLM 4.7, GPT-2, GPT-OSS, Llama 3, MiniMax M2.1, Mistral Nemo, Nemotron 3, Phi-4 mini, Qwen 2.5, and Qwen 3. It runs five batch and input shapes on Apple M2 plus twelve four-core Modal placements across AWS, GCP, and OCI.

Every timed engine first passes Hugging Face token-ID parity. Gigatoken and Snaptokens cover all twelve models; upstream fastokens covers nine. Flat-ragged engines are compared only with the same flat-ragged output contract, nested engines only with nested output.

tokenizer.json and .tkz produce the same steady-state Snaptokens encoder, so the format comparison measures fresh load, first encode, disk size, and memory rather than drawing two redundant throughput bars. TKZ loads 1.55× faster than Snaptokens JSON, but is still 1.34× slower than Hugging Face JSON in this suite.

The full benchmark report contains the per-host, per-model, per-shape, load, footprint, variability, correctness, and limitation analysis. The complete accepted evidence includes every raw round, hardware capture, hash, summary, and an empty failure ledger.

Install

Install the Python package:

pip install snaptokens

Or add the Rust crate:

[dependencies]
snaptokens = "0.2"

Use

Python:

from snaptokens import Tokenizer

tokenizer = Tokenizer.from_file("/path/to/tokenizer.json")
ids = tokenizer.encode("Tokenization should not be the bottleneck.").ids

Pass tkz_cache=True to create and reuse the adjacent .tkz file. A .tkz path loads directly without that flag.

To replace a Transformers v4 or v5 fast-tokenizer backend:

import snaptokens

snaptokens.patch_transformers()

Call snaptokens.unpatch_transformers() to restore the original backend.

Rust:

use std::{error::Error, path::Path};

use snaptokens::Tokenizer;

/// Loads one local tokenizer and encodes a prompt.
fn main() -> Result<(), Box<dyn Error>> {
    let tokenizer = Tokenizer::from_file(Path::new("/path/to/tokenizer.json"))?;
    let ids = tokenizer.encode("Tokenization should not be the bottleneck.")?;

    println!("{ids:?}");
    Ok(())
}

Download the model yourself and pass the local JSON path. snaptokens never tries to infer or download a Hugging Face model.

Optional .tkz cache

let tokenizer = Tokenizer::from_file_with_tkz_cache(
    Path::new("/path/to/tokenizer.json"),
    true,
)?;

The first enabled load atomically writes tokenizer.tkz; later loads validate and reuse it. Passing that .tkz path loads it directly without creating another copy. Tokenizer::from_file remains JSON-only and never writes a cache.

Scope

Snaptokens is only built for inference on BPE tokenizers, and to do just that, well.

Real-tokenizer validation covers GPT-2, GPT-OSS, DeepSeek, MiniMax, Qwen and Qwen3.5, GLM, Nemotron, Mistral, Gemma 4, Phi-4, and Kimi K2.5 families, including added-token flags and byte fallback where present.

Credits

Licensed under Apache-2.0.