puretiktoken 0.2.0

tiktoken in pure Python: byte-for-byte compatible BPE tokenizer with no Rust extension

puretiktoken

tiktoken in pure Python: byte-for-byte compatible BPE tokenizer for OpenAI models — no Rust extension, no binary wheel, zero dependencies.

OpenAI's tiktoken is a Rust extension. That's fast, but it means a compiled wheel has to exist (and load) for your platform — which it doesn't on Pyodide/WASM, often fails on fresh ARM or musl/Alpine boxes, bloats AWS Lambda layers, and won't build in locked-down sandboxes with no Rust toolchain. puretiktoken is the same tokenizer written in plain Python, so it runs anywhere Python runs and produces the exact same token ids.

pip install puretiktoken

import puretiktoken as tiktoken      # drop-in: same names, same ids

enc = tiktoken.get_encoding("cl100k_base")
enc.encode("hello world")             # -> [15339, 1917]   (identical to tiktoken)
enc.decode([15339, 1917])             # -> "hello world"

tiktoken.encoding_for_model("gpt-4o").encode("tokens!")   # o200k_base
len(tiktoken.get_encoding("o200k_base").encode(prompt))    # count before an API call

The API mirrors the slice of tiktoken people actually use — get_encoding, encoding_for_model, encode, encode_ordinary, decode, decode_bytes, n_vocab, plus allowed_special / disallowed_special handling — so for token counting and offline tokenization it is a drop-in replacement.

Why pure Python

tiktoken (Rust), Hugging Face tokenizers (Rust) and sentencepiece (C++) are all native extensions — there was no pure-Python tokenizer that matches tiktoken's output. The hard part isn't the BPE merge (that's ~30 lines); it's tiktoken's pre-tokenizer, a fancy-regex pattern using \p{L}/\p{N} Unicode classes, possessive quantifiers and a lookahead — features the stdlib re can't even compile. puretiktoken reproduces that splitter exactly: it translates each character to a single-byte Unicode class code (the only Unicode-aware step, done with str.translate at C speed), then runs an equivalent pattern — now free of \p{...} — in the C re engine, and feeds the pieces to the same rank-based merge. The result is verified byte-for-byte against the real tiktoken (see below).

Zero dependencies, zero binaries: the vocabularies ship gzipped inside the wheel, so it works fully offline — no download of *.tiktoken files at runtime, which is the other thing that breaks tiktoken in sandboxed or air-gapped environments.

Speed, stated honestly

A Rust extension should win, but on real-world text the gap is small — and sometimes the other way. puretiktoken translates each character to a one-byte Unicode class and lets the C re engine do the splitting, then caches the ids of each pre-token (real text repeats tokens heavily, by Zipf's law). On prose, code, JSON or markdown it runs at ~3.5–5.5 M tokens/sec (CPython 3.12), within ~1.2–2.6× of tiktoken — occasionally faster, because it stays in-process while tiktoken pays Python↔Rust overhead on every call.

corpus (CPython 3.12)	cl100k_base	o200k_base
code / prose / markdown	~1.2–1.5× tiktoken	~2.0–2.6× tiktoken
heavily repetitive	~1.0× (can beat tiktoken)	~1.5× tiktoken
adversarial high-entropy¹	~10–11× tiktoken	~13–14× tiktoken

¹ Random unique strings where no token ever repeats, so the cache can't help and it falls back to the pure-Python BPE merge. Real text never looks like this; run python tools/bench.py to measure on your own.

The point was never to beat Rust — it's getting tiktoken's exact ids where tiktoken won't install or run. That it's now also fast on real text is a bonus.

Verified against the real tiktoken

Conformance is differential, the same way purere2 checks itself against the real RE2 and purefzf against the fzf binary: text is tokenized by both puretiktoken and tiktoken and the token ids are compared. Across a curated multilingual/code/emoji corpus plus tens of thousands of randomized inputs over every assigned Unicode codepoint, the output is byte-for-byte identical for both cl100k_base and o200k_base — including the tricky parts: contractions with Unicode case folding ('ſ folds to 's), o200k's camelCase splitting (iPhone → i+Phone), combining marks, and whitespace-run edges. Decoding round-trips losslessly for arbitrary input. The test suite locks this, so any regression fails CI.

The one place puretiktoken and tiktoken can differ is unassigned codepoints. tiktoken's Rust regex bundles its own Unicode database; puretiktoken uses your Python's unicodedata. When those versions disagree about whether a brand-new codepoint is a letter, the split can differ — but only for text that contains codepoints unassigned in your Python, which real-world text never does. The conformance suite asserts this explicitly: every divergence must be explained by an unassigned codepoint; a mismatch on assigned text is treated as a bug.

Encodings

cl100k_base (GPT-4, GPT-3.5, text-embedding-3-*, ada-002) and o200k_base (GPT-4o, GPT-4o-mini, o1, o3). encoding_for_model() maps model names to these, matching tiktoken's mapping for the current model families.

License

MIT. The bundled vocabularies are OpenAI's, distributed under the MIT terms of tiktoken.