toktok

A fast, exact BPE tokenizer for OpenAI encodings — Rust core, Python bindings. Token ids are byte-identical to tiktoken.

The Pile, 25 MB, cl100k_base, single thread, GitHub runners. Ids verified identical before timing.

Python 3.11

Python 3.14

Python 3.14t · free-threaded

Rust

_{Each table is one machine; the Python ones ran on separate runners, so compare within a table. Method and full results: docs/BENCHMARKS.md.}

Encodings

Pass an encoding name — not a model name. The three bundled encodings are compiled into the binary, so there are no data files, downloads or runtime dependencies.

An unknown name raises KeyError listing the valid ones.

Python

uv add toktok-rs        # or: pip install toktok-rs

One function. Give it your texts and an encoding, get a token count per text:

import toktok

toktok.batch_count(["hello world", "how many tokens is this?"])
# [2, 6]

toktok.batch_count(docs, "o200k_base")                # GPT-4o and friends
toktok.batch_count(docs, "cl100k_base", threads=4)    # 0 = every core (default)
toktok.batch_count(["a<|endoftext|>b"], with_special=True)   # [3]

batch_count(texts, encoding="cl100k_base", threads=0, with_special=False)

Returns list[int] — one count per text, in the order given.

Ids are never built: one scratch buffer per thread is reused across texts, so a batch of any size allocates O(threads), not O(tokens).

Wheels cover CPython 3.11–3.14 plus free-threaded 3.14t, on Linux (x86_64, aarch64), macOS and Windows.

truncate(text, max_tokens, encoding="cl100k_base")

Cut text to a token budget. Returns (text, total_tokens), where total_tokens counts the whole input — truncated or not — so you can report how much was dropped.

toktok.truncate("hello world", 1)          # ('hello', 2)
toktok.truncate("hello world", 50)         # ('hello world', 2) — same object back

toktok.batch_truncate(docs, 8192, "o200k_base", threads=8)   # [(text, total), ...]

One pass, no ids built, nothing decoded — the cut is a byte offset into the string you passed in. About 3× faster than decode(encode(text)[:n]), and ~6× with the batch form.

It also cuts on a character boundary. Byte-level BPE can split one character across tokens ("🧠" is three), so a token-boundary cut can leave a partial character that decodes to U+FFFD — decode(enc.encode("hello 🧠")[:2]) gives 'hello \ufffd'. truncate gives 'hello '.

Need ids, decoding or offsets? toktok._encoding(name) returns the full tokenizer — see docs/PYTHON.md.

Rust

cargo add toktok-rs     # the crate is toktok-rs, the library is toktok

let tok = toktok::Tokenizer::builtin("cl100k_base")?;   // nothing to download

let ids = tok.encode(b"hello world");
let text = tok.decode(&ids);                 // lossless, even on invalid UTF-8
let n = tok.count(b"how many tokens is this?");

// batch APIs are parallel; 0 threads means every core
let docs: Vec<&[u8]> = vec![b"first", b"second"];
let counts = tok.count_batch(&docs, 0, false);   // no ids materialized
let batches = tok.encode_batch(&docs, 0, false);

let (ids, spans) = tok.encode_with_offsets(b"per-token byte spans");

// truncation without building ids or decoding: a byte offset back
let t = tok.truncate(text, 8192);
let kept = &text[..t.bytes];        // always a character boundary
t.total_tokens;                     // the whole input, truncated or not

Every batch method takes the same two knobs as the Python API:

Tokenizer is Send + Sync — load it once and share it. Full API on docs.rs/toktok-rs.

Cargo features

More

• docs/BENCHMARKS.md — full results, memory and latency profile, how to reproduce
• docs/PYTHON.md — the tokenizer behind batch_count
• docs/DEVELOPMENT.md — layout, tests, how it's fast
• RELEASING.md — scripts/release.sh patch|minor|major|X.Y.Z, and the registry setup

License

MIT — see LICENSE. Ported from quicktok (MIT); vocabulary data derives from tiktoken (MIT). See NOTICE.