quicktok

A fast, exact BPE tokenizer for OpenAI and open-model encodings, written in C++. Token ids are byte-identical to tiktoken; encoding runs 2–3.4× faster than the fastest exact tokenizer we know of (bpe-openai) and 3.5–11× faster than tiktoken itself. See benchmarks.

• Exact — ids match each encoding's reference (tiktoken / Hugging Face / Meta) byte-for-byte; every benchmark is exactness-checked before timing.
• Drop-in — Python wheels with a tiktoken-style API, a stable C ABI, CMake support.
• Self-contained — C++20, no external dependencies; cl100k_base, o200k_base, o200k_harmony, Llama-3, and Qwen2.5/3 ship in the repo.
• Thread-safe — load once, call encode() from as many threads as you like.

Install

Python (the PyPI package is quicktok-v1; you still import quicktok):

pip install quicktok-v1

Runs on Linux, Mac and Windows. Python ≥ 3.9 required.

C++ — via CMake (find_package or FetchContent), or make install and pkg-config. There's also a stable C ABI (quicktok.h) for FFI from any language.

find_package(quicktok REQUIRED)
target_link_libraries(app PRIVATE quicktok::quicktok)

Python

import quicktok
enc = quicktok.get_encoding("cl100k_base")
ids = enc.encode("hello world")                   # tiktoken semantics (raises on a stray special)
text = enc.decode(ids)

quicktok.encoding_for_model("meta-llama/Llama-3.1-8B").count("...")   # model-name lookup

# other byte-level-BPE tokenizers: import once (exactness-verified), then use by name
quicktok.import_tokenizer("mistralai/Mistral-Nemo-Instruct-2407", "tekken")  # HF repo id, URL, or local file
quicktok.get_encoding("tekken")

All encoding names — bundled, gated, imported — are in Encodings.

For bulk work (dataset prep, corpus token counting), encode_batch tokenizes documents in parallel and returns one flat uint32 token array plus int64 offsets:

tokens, offsets = enc.encode_batch(docs)    # doc i = tokens[offsets[i]:offsets[i+1]]
tokens.tofile("corpus.tokens.bin")          # flat binary, ready for training
counts = quicktok.count_batch(enc, docs)    # per-doc token counts for budgeting

enc.encode_batch(chats, with_special=True)  # chat-templated data: "<|im_start|>..." -> special ids

• Drop-in for tiktoken — same method names and semantics, so a tiktoken Encoding swaps for quicktok.get_encoding(...); like tiktoken, encode raises on a stray special token unless you pass allowed_special (or call encode_ordinary).
• Imported encodings are stored in $QUICKTOK_DATA (default ~/.cache/quicktok); get_encoding finds them automatically.
• import_tokenizer verifies against the model's own tokenizer, so the reference must be installed: pip install tokenizers (HF models) or mistral-common (Tekken) — plus huggingface_hub to fetch from a repo id (and its auth for gated repos).

C++

git clone https://github.com/dmatth1/quicktok
cd quicktok
make            # builds build/libquicktok.{a, dylib/so}
make test       # verifies exact ids vs the references (all encodings) + C ABI

#include <quicktok.hpp>

auto tok = quicktok::Tokenizer::load_dir("data");              // cl100k_base
auto gpt4o = quicktok::Tokenizer::load_dir("data", "o200k_base");

auto ids = tok.encode("Hello, quicktok! 日本語 🚀");  // std::vector<uint32_t>
std::string text = tok.decode(ids);                   // lossless round-trip
size_t n = tok.count("how many tokens is this?");
auto with_sp = tok.encode_with_special("a<|endoftext|>b");  // specials -> ids
auto batch = tok.encode_batch(texts);                       // parallel

namespace quicktok {
class Tokenizer {
    // encoding: a built-in name (see Encodings) or any imported one
    static Tokenizer load_dir(const std::string& dir, const std::string& encoding = "cl100k_base");

    std::vector<uint32_t> encode(std::string_view text) const;          // encode_ordinary semantics
    std::vector<uint32_t> encode_with_special(std::string_view) const;  // allowed_special="all"
    void encode(const uint8_t* text, size_t len, std::vector<uint32_t>& out) const;
    std::vector<std::vector<uint32_t>> encode_batch(const std::vector<std::string_view>&,
                                                    unsigned threads = 0, bool with_special = false) const;
    size_t count(std::string_view text) const;

    std::string decode(const std::vector<uint32_t>& ids) const;         // handles special ids too
    void decode(const uint32_t* ids, size_t n, std::string& out) const;

    size_t vocab_size() const;     // base vocab; n_vocab() = max id + 1 incl. specials
    size_t n_vocab() const;
    const std::vector<std::pair<std::string, uint32_t>>& special_tokens() const;
    const std::string& encoding() const;
};
}

• load_dir takes the directory holding the data files (data/ in this repo; make install puts them in share/quicktok). It throws std::runtime_error on missing or corrupt files; nothing throws on the encode hot path (one exception: inputs over 4 GiB per call are rejected).
• A loaded Tokenizer is safe to share across threads — concurrent encode()/decode() is supported and tested.

Benchmarks

Five encoders, same machine (Apple M1), single thread, every output verified token-for-token identical before timing, every reference called through the same raw API its own benchmark uses. Three 25 MB corpora streamed from their real sources — The Pile (diverse), GitHub code, Common Crawl (multilingual) — across both common OpenAI encodings (throughput in MB/s):

cl100k_base (GPT-3.5 / GPT-4)

o200k_base (GPT-4o)

Reproduce these tables: make bench-compare — see bench/README.md.

How it's fast

Same algorithm as bpe-openai (exact backtracking BPE) — the speed is data-structure engineering:

• 2-byte trie — the longest-match walk reads 2 input bytes per single 8-byte slot load, with a zero-lookup direct table for CJK characters.
• Dense validity memos — merge-validity checks hit exactly-keyed caches (2 MB for 17-bit token ids, a second wide one for 200k-vocab ids; a bijective mixer means no aliasing, ever).
• Specialized pretokenizers — the fixed cl100k/o200k-family regexes are compiled by hand into SIMD scanners; no general regex engine anywhere.
• Single-pass product machines — for ASCII text (most of code and English), one loop owns both the pretokenizer's boundary rules and token emission: contractions, prefix-words, digit triples, punct runs, and the whitespace cascade are handled inline with no per-piece scanner dispatch; only Unicode contact falls back to the general scanner, one piece at a time.

Encodings

Five encodings ship in the repo; Llama-4's code path ships too, but its vocab is gated. Each is byte-exact vs its reference:

• qwen3 reproduces the HF pipeline including NFC normalization: clean input pays one cheap scan, only spans with non-NFC codepoints are normalized, and — like HF — decode returns the normalized text for such input.
• llama3 matches Meta's original tiktoken-rank tokenizer. Hugging Face and llama.cpp infer the same vocab from a merge list and agree on ~99.9998% of tokens; the rare differences are non-Latin+symbol sequences where rank order and merge order pick different splits.
• llama4 shares o200k_base's pretokenizer, but Meta gates the vocab. With repo access, import it like any other tokenizer — quicktok.import_tokenizer("meta-llama/Llama-4-Scout-17B-16E-Instruct", "llama4") (the import verifies, so a wrong guess can't ship; this checks against HF's merge-list tokenizer, the same rank-vs-merges nuance as llama3). To match Meta's original rank file instead, export from a checkout: python tools/export_llama4.py <tokenizer.model> data.

Vocabs regenerate from their references with tools/export_*.py; the Unicode and NFC tables are pinned, version-stamped, and exhaustively re-derivable (tools/export_unicode.py verify, tools/export_nfc.py verify). Third-party vocab licenses: NOTICE.

Importing other tokenizers

Any byte-level-BPE tokenizer whose pretokenizer matches one of quicktok's hand-compiled grammars (cl100k / o200k / llama3 / qwen / tekken) can be imported — quicktok.import_tokenizer() from Python (shown above), or:

python -m quicktok.importer path/to/tokenizer.json myenc --corpus big.txt

The import checks the normalizer (none/NFC), classifies the pretokenizer regex, writes the data files, then encodes a stress suite plus any --corpus files with both the reference tokenizer and quicktok and compares token-for-token. A mismatch fails the import; an unrecognized pattern is refused with the pattern printed. There is no fallback regex engine and no approximate mode — each grammar is compiled by hand, which is where the speed comes from.

• Mistral Tekken v3 imports and verifies: the o200k grammar minus contractions, single-digit numbers, ids offset by its 1,000 reserved special slots — exact vs mistral-common, at full o200k-class speed.
• DeepSeek V3/R1 is refused: a pipeline of three sequential Split regexes, a different grammar shape. Supporting it would be a new scanner, not an import.
• SentencePiece models (Gemma, T5, Llama-2) are a different algorithm entirely, not a missing grammar — out of scope.

Notes

• Builds tune to the host CPU by default (-march=native); set CXXFLAGS_ARCH for portable binaries.

• To regenerate the data files from the references:

  pip install tiktoken regex tokenizers
  python tools/export_fixtures.py        # cl100k/o200k/o200k_harmony from tiktoken
  python tools/export_qwen.py --download  # data/qwen3.vocab (Apache-2.0)
  python tools/export_unicode.py         # data/uniclass.bin + version stamp
  python tools/export_nfc.py             # data/nfc.bin (NFC tables) + version stamp
  python tools/gen_vectors.py            # test vectors (tiktoken encodings)
  

License

MIT — see LICENSE.