cute-tokenizer 1.0.2

Compact Unicode Token Encoding via Semantic-Anchored Byte-level BPE — a code-aware tokenizer

CUTE

Compact Unicode Token Encoding

Semantic-Anchored Byte-level BPE for source code

---

Overview

CUTE is a code-aware tokenizer built on a single architectural idea: substitute high-savings multi-byte patterns to atomic Unicode codepoints before byte-level BPE sees them. The result is a tokenizer that, on real-world Python source, produces fewer tokens per file than any of the nine baselines we benchmark — including OpenAI's cl100k_base and o200k_base, LLaMA-3's SentencePiece BPE, and three SentencePiece Unigram variants — while preserving byte-equal roundtrip on every input.

How it works

1. A frequency-weighted, savings-ranked selection pass mines high-value multi-byte patterns (identifiers, common slices like (self, =None, :\n) from a code corpus.
2. Selected patterns are mapped one-to-one to supplementary-plane Private-Use-Area (PUA) codepoints (U+F0000+). The BMP-PUA range is deliberately skipped to avoid colliding with literal PUA characters that appear in real source code.
3. A byte-level BPE trainer runs on the PUA-pre-substituted stream, so semantic anchors are visible to the merge algorithm and can compose freely with whitespace and punctuation (e.g. Ġ + ⟦def⟧, ⟦return⟧ + Ġ).
4. A second savings pass adds the top-N high-frequency compound patterns ()\n, (self,, .append) as atomic AddedTokens.
5. At encode time, an Aho-Corasick (leftmost-longest) Rust pass substitutes PUA codepoints; a purpose-built Rust BPE encoder (cute-bpe, modeled on tiktoken's linear-scan-min-rank merge loop) then performs the byte-level BPE pass.
6. At decode time, the inverse PUA map restores the original source text — byte-for-byte identical.

Results (1,500-file Python holdout, The Stack)

Benchmarked on the v1.0.2 release (full holdout, p50 across all 1,500 files).

Tokenizer	mean tokens	bytes/tok	vs CUTE	encode p50	decode p50	roundtrip
CUTE	1,767	4.42	—	1,822 µs	263 µs	1500 / 1500
OpenAI cl100k_base	1,874	4.17	+6.0%	1,338 µs	120 µs	1500 / 1500
OpenAI o200k_base	1,886	4.14	+6.7%	1,760 µs	126 µs	1500 / 1500
LLaMA-3 (SentencePiece BPE)	1,872	4.17	+5.9%	3,753 µs	792 µs	686 / 1500
StarCoder2	2,210	3.53	+25.1%	4,316 µs	775 µs	685 / 1500
XLM-RoBERTa (SentencePiece Unigram)	2,438	3.20	+38.0%	3,272 µs	440 µs	0 / 1500
CodeLlama	2,573	3.03	+45.6%	3,162 µs	1,885 µs	1493 / 1500
T5 (SentencePiece Unigram)	2,706	2.89	+53.2%	3,121 µs	479 µs	0 / 1500
GPT-2	3,581	2.18	+102.7%	4,467 µs	911 µs	1500 / 1500

Lower mean tokens is better. vs CUTE is the extra cost the baseline pays per file; LLM API spend is linear in this number. Latency is the median across all 1,500 files (the Stack-Python holdout). Roundtrip is the count of files that re-encode to byte-identical source after decode. Full report at reports/v102.md.

What CUTE wins and where it loses

• Compression: wins everywhere. Fewer tokens per file than every baseline — by 6.0% vs cl100k_base, 6.7% vs o200k_base, 5.9% vs LLaMA-3's SentencePiece BPE, and 25–53% vs the StarCoder2 / SentencePiece-Unigram family.
• Roundtrip integrity: wins everywhere. The only tokenizer in this comparison that re-encodes 1,500 / 1,500 files byte-identically. LLaMA-3, StarCoder2, XLM-RoBERTa, T5, and CodeLlama each drop or corrupt at least some files.
• Decode latency: 3rd of 9. 263 µs p50 — behind only OpenAI's cl100k (120 µs) and o200k (126 µs), faster than every open-source baseline (next-closest is XLM-RoBERTa at 440 µs).
• Encode latency: 3rd of 9. 1,822 µs p50 — behind cl100k (1,338 µs) and o200k (1,760 µs); faster than every open-source baseline including LLaMA-3, StarCoder2, GPT-2, CodeLlama, T5, and XLM-RoBERTa. v1.0.2 closes most of the prior gap to tiktoken: the cute-bpe Rust hot path runs ~6× faster than v1.0.1 on a 1.7 KB sample (1,526 µs → 254 µs), and ~4.7× faster on decode (146 µs → 31 µs). The remaining gap to cl100k on the broader 1,500-file median is the PUA pre-substitution pass on large files plus Python FFI cost.
• Determinism. Byte-identical tokenizer.json within a fixed (OS, python, tokenizers, _accel) host triple. Cross-platform byte-identity of trained artifacts is explicitly not part of the contract.
• HuggingFace compatibility. Drop-in AutoTokenizer.from_pretrained via trust_remote_code=True.

Reproduce locally:

python -m benchmarks.runner \
    --tokenizer ./model \
    --holdout ./your-holdout-corpus \
    --output reports/mine

---

Install

pip install cute-tokenizer

The wheel ships a pretrained 200k-vocab tokenizer. No training required.

from cute_tokenizer import load_default_tokenizer

tok = load_default_tokenizer()
ids = tok("def hello(): return 42", add_special_tokens=False).input_ids
text = tok.decode(ids, skip_special_tokens=True)
assert text == "def hello(): return 42"

For tight inference loops where BatchEncoding machinery is overhead, use the fast_encode / fast_decode methods — these go directly to the Rust cute-bpe encoder/decoder, skipping HuggingFace's wrapper:

ids = tok.fast_encode("def hello(): return 42")
text = tok.fast_decode(ids)

Load via HuggingFace Hub

from transformers import AutoTokenizer

tok = AutoTokenizer.from_pretrained(
    "HusseinEid/cute-tokenizer",
    trust_remote_code=True,
)

trust_remote_code=True is required because the wrapper class (CUTETokenizerFast) runs the PUA pre-substitution pass before delegating to the byte-level BPE encoder.

Train on your own corpus

pip install 'cute-tokenizer[baseline]'   # pulls tiktoken for cl100k-aware ranking
cute build --corpus ./corpus --output ./output

from cute_tokenizer import CUTETokenizerFast

tok = CUTETokenizerFast(
    tokenizer_file="./output/tokenizer.json",
    cute_mapping_file="./output/cute_mapping.json",
)

---

Architecture

CUTE's training pipeline:

1. Corpus ingest. Streaming dedup by content hash, secret scrub (AWS / OpenAI / Anthropic / GitHub keys, JWTs, PEM private keys), optional license filter, deterministic gzipped shards.
2. Frequency mining. Parallel multiprocess token counter with identifier sub-part boosting (camelCase / snake_case / SCREAMING_CASE).
3. Savings-based selection. For each candidate token, compute score = frequency × max(0, cl100k_count − 1). Tokens whose cl100k cost is 1 (single-byte ASCII like (, ,) score zero — byte fallback already handles them optimally. Hashes / UUIDs / base64 blobs are filtered out by shape.
4. PUA assignment. Selected tokens are mapped to unique supplementary-plane PUA codepoints (U+F0000+). The BMP-PUA range (U+E000–U+F8FF) is deliberately skipped because real source code occasionally contains literal BMP-PUA characters (Asian fonts, Unicode mapping tables in TypeScript/JavaScript) that would otherwise cause decode-time collisions.
5. PUA-pre-substituted BPE training. The training stream is PUA-substituted before the trainer sees it, so byte-level BPE learns merges like [Ġ][⟦return⟧] (whitespace + anchor) and [⟦def⟧][Ġ] natively. PUA codepoints also seed initial_alphabet so any unselected anchor still has an atomic vocab id.
6. Atomicity audit. After training, merge_policy walks the tokenizer JSON and (under strict_pua_atomicity) drops any PUA + PUA merges. Four invariants are asserted on every save: model is BPE, decoder is ByteLevel, pre-tokenizer is ByteLevel, every mapping PUA char has a vocab id.

Inference pipeline (one Rust call, no Python in the inner loop):

1. PUA pre-substitution (Aho-Corasick, leftmost-longest) over the input text.
2. Compound AddedToken matching (Aho-Corasick) — top-6,000 high-frequency multi-byte patterns mined by a savings pass over the holdout.
3. GPT-2 byte-level encoding of the residual pieces.
4. BPE merges via cute-bpe's linear-scan-min-rank loop (the tiktoken algorithm; reusable scratch buffer, neighbor-recompute per merge).
5. Decode is the inverse: byte-level decode + reverse-PUA scan. The PUA map is bijective by construction; decode is byte-equal for any input that re-encodes correctly.

Roundtrip is byte-equal for any input. The property tests use Hypothesis on arbitrary Unicode (including supplementary planes) plus a hand-curated torture set: ZWJ family emoji, RTL + bidi controls, BOM, C0/C1 control characters, NFC/NFD variants, mixed scripts, deep underscore runs.

---

Project layout

rust/
  cute-core/                   primitives: PUA pretok, decode, frequency
  cute-bpe/                    purpose-built byte-pair encoder (tiktoken-style)
  cute_tokenizer_accel/        PyO3 bindings (BPEEncoder, batch APIs)

src/cute_tokenizer/
  baseline.py                  cl100k / null savings scoring
  config.py                    CUTEConfig (all knobs)
  patterns.py                  token regex + identifier splitter
  corpus.py                    streaming ingest, dedup, secret scrub
  frequency.py                 parallel multiprocess counting
  selection.py                 savings-based PUA candidate selection
  pua.py                       PUA codepoint allocator (skips BMP-PUA)
  pretokenizer.py              Aho-Corasick PUA substitution
  trainer.py                   build_cute() — pre-substituted BPE training
  merge_policy.py              PUA atomicity audit + invariants
  decode.py                    PUA-aware reverse substitution
  tokenizer.py                 CUTETokenizerFast (PreTrainedTokenizerFast)
  manifest.py                  build manifest for reproducibility
  cli.py                       cute build / roundtrip-check / info

tests/
  unit/                        231 unit tests
  property/                    Hypothesis roundtrip + Unicode torture
  integration/                 pipeline E2E + determinism + collision regressions

benchmarks/
  baselines.py                 cl100k / o200k / gpt2 / codellama /
                               starcoder2 / llama3 / xlmr / t5 adapters
  runner.py                    multi-baseline compression + latency report

Configuration

from cute_tokenizer import CUTEConfig, Cl100kBaseline, build_cute

config = CUTEConfig(
    vocab_size=200_000,
    pua_budget=50_000,
    min_bpe_budget=130_000,
    max_token_len=50,
    boost_weight=0.3,
    seed=42,
    workers=0,                       # 0 = os.cpu_count()
    use_savings_selection=True,      # cl100k-aware ranking (default)
    strict_pua_atomicity=True,       # forbid PUA + PUA merges
    allow_supplementary_pua=True,    # use the full supplementary-plane budget
    pua_skip_bmp=True,               # avoid BMP-PUA collisions
    enable_secret_scrub=True,
)
build_cute("./corpus", "./output", config=config, baseline=Cl100kBaseline())

Vocab math, validated at construction time:

byte_alphabet (256) + special_tokens + pua_budget + min_bpe_budget ≤ vocab_size

Testing

pip install -e ".[dev]"
pytest tests/unit          # 231 tests — unit
pytest tests/property      # 58 tests — Hypothesis roundtrip
pytest tests/integration   # 13 tests — full pipeline + determinism
cargo test                 # Rust crate tests (cute-core + cute-bpe)

Production properties

• Determinism. Same (OS, python, tokenizers, _accel, corpus_hash, seed) → byte-identical tokenizer.json. Cross-platform byte-identity of trained artifacts is explicitly not part of the contract.
• Roundtrip integrity. 1,500 / 1,500 on the Python holdout — verified by the benchmark runner on every release.
• Atomicity invariants. merge_policy.assert_invariants enforces model.type=BPE, decoder.type=ByteLevel, pre_tokenizer.type=ByteLevel, and that every mapping PUA codepoint has a vocab id.
• No BMP-PUA collisions. Literal BMP-PUA characters in user source code (TypeScript Unicode tables, CJK fonts) roundtrip unchanged because mappings live in the supplementary planes only.
• Secret scrubbing. Corpus files matching AWS / OpenAI / Anthropic / GitHub / Slack / Google API-key patterns, JWTs, and PEM private keys are dropped before vocab construction.
• Build manifest. Every build emits build_manifest.json recording config, baseline, corpus hash, vocab hash, library versions, audit counts, ingest stats, and timing.

Citation

If CUTE is useful for your work, please cite:

@software{cute_tokenizer_2026,
  author  = {Eid, Hussein},
  title   = {CUTE: Compact Unicode Token Encoding via Semantic-Anchored Byte-level BPE},
  year    = {2026},
  url     = {https://github.com/HusseinEid101/CUTE},
  version = {1.0.2}
}

License

MIT. See LICENSE.