puretiktoken 0.1.0

tiktoken in pure Python: byte-for-byte compatible BPE tokenizer with no Rust extension

puretiktoken

tiktoken in pure Python: byte-for-byte compatible BPE tokenizer for OpenAI models — no Rust extension, no binary wheel, zero dependencies.

OpenAI's tiktoken is a Rust extension. That's fast, but it means a compiled wheel has to exist (and load) for your platform — which it doesn't on Pyodide/WASM, often fails on fresh ARM or musl/Alpine boxes, bloats AWS Lambda layers, and won't build in locked-down sandboxes with no Rust toolchain. puretiktoken is the same tokenizer written in plain Python, so it runs anywhere Python runs and produces the exact same token ids.

pip install puretiktoken

import puretiktoken as tiktoken      # drop-in: same names, same ids

enc = tiktoken.get_encoding("cl100k_base")
enc.encode("hello world")             # -> [15339, 1917]   (identical to tiktoken)
enc.decode([15339, 1917])             # -> "hello world"

tiktoken.encoding_for_model("gpt-4o").encode("tokens!")   # o200k_base
len(tiktoken.get_encoding("o200k_base").encode(prompt))    # count before an API call

The API mirrors the slice of tiktoken people actually use — get_encoding, encoding_for_model, encode, encode_ordinary, decode, decode_bytes, n_vocab, plus allowed_special / disallowed_special handling — so for token counting and offline tokenization it is a drop-in replacement.

Why pure Python

tiktoken (Rust), Hugging Face tokenizers (Rust) and sentencepiece (C++) are all native extensions — there was no pure-Python tokenizer that matches tiktoken's output. The hard part isn't the BPE merge (that's ~30 lines); it's tiktoken's pre-tokenizer, a fancy-regex pattern using \p{L}/\p{N} Unicode classes, possessive quantifiers and a lookahead — features the stdlib re can't even compile. puretiktoken reproduces that splitter exactly with a hand-written scanner over unicodedata categories, then runs the same rank-based merge. The result is verified byte-for-byte against the real tiktoken (see below).

Zero dependencies, zero binaries: the vocabularies ship gzipped inside the wheel, so it works fully offline — no download of *.tiktoken files at runtime, which is the other thing that breaks tiktoken in sandboxed or air-gapped environments.

Speed, stated honestly

Pure Python can't beat a Rust extension, and this doesn't try to. It encodes at roughly 0.7–1.3 M tokens/sec (CPython 3.12, single core), which is ~7× slower than tiktoken on cl100k_base and ~15× on o200k_base — and still fast enough that token counting is never your bottleneck.

	puretiktoken (CPython 3.12)	tiktoken (Rust)
cl100k_base	~1.25 M tok/s	~5.7 M tok/s
o200k_base	~0.71 M tok/s	~8.1 M tok/s

The point is not speed — it's getting tiktoken's exact ids where tiktoken won't install or run. If you need both, use tiktoken when its wheel loads and fall back to puretiktoken when it doesn't; the ids are identical.

Verified against the real tiktoken

Conformance is differential, the same way purere2 checks itself against the real RE2 and purefzf against the fzf binary: text is tokenized by both puretiktoken and tiktoken and the token ids are compared. Across a curated multilingual/code/emoji corpus plus tens of thousands of randomized inputs over every assigned Unicode codepoint, the output is byte-for-byte identical for both cl100k_base and o200k_base — including the tricky parts: contractions with Unicode case folding ('ſ folds to 's), o200k's camelCase splitting (iPhone → i+Phone), combining marks, and whitespace-run edges. Decoding round-trips losslessly for arbitrary input. The test suite locks this, so any regression fails CI.

The one place puretiktoken and tiktoken can differ is unassigned codepoints. tiktoken's Rust regex bundles its own Unicode database; puretiktoken uses your Python's unicodedata. When those versions disagree about whether a brand-new codepoint is a letter, the split can differ — but only for text that contains codepoints unassigned in your Python, which real-world text never does. The conformance suite asserts this explicitly: every divergence must be explained by an unassigned codepoint; a mismatch on assigned text is treated as a bug.

Encodings

cl100k_base (GPT-4, GPT-3.5, text-embedding-3-*, ada-002) and o200k_base (GPT-4o, GPT-4o-mini, o1, o3). encoding_for_model() maps model names to these, matching tiktoken's mapping for the current model families.

License

MIT. The bundled vocabularies are OpenAI's, distributed under the MIT terms of tiktoken.