TinyBPE

An ultra-fast, lightweight BPE tokenizer and trainer with a pure-C core.

Ever wished you could load a GPT-4 compatible tokenizer in one line without network calls? TinyBPE ships 8 pre-built ByteLevel BPE models directly in the package. The CPython C core runs BPE encoding/decoding at native speed — typically 10-50× faster than pure-Python implementations while depending only on regex.

Why TinyBPE?

Installation

pip install tinybpe

Optional extras:

pip install tinybpe[dev]       # Development tools (pytest, ruff, mypy)
pip install tinybpe[tiktoken]  # For tiktoken comparison testing
pip install tinybpe[hf]        # For HuggingFace model conversion
pip install tinybpe[all]       # Everything

Quick Start

One-Line Model Loading

from tinybpe import Tokenizer

# Load any built-in model in one line — no network, no download
tok = Tokenizer.from_pretrained("cl100k_base")

ids = tok.encode("hello world")
tok.decode(ids)  # → 'hello world'

List Available Models

import tinybpe

tinybpe.list_models()
# ['cl100k_base', 'deepseek-v4', 'llama4', 'minicpm5', 'o200k_base',
#  'p50k_base', 'qwen35', 'r50k_base']

Built-in Model Catalog

Training

from tinybpe import Trainer

trainer = Trainer("hello world " * 500)
trainer.train(100)          # learn 100 merges
trainer.save("my_model")    # → my_model.tbm

Streaming Decode

parts = []
decoder = tok.stream_decode(lambda s: parts.append(s))
for token_id in ids:
    decoder(tid)
assert "".join(parts) == "hello world"

With Regex Pre-tokenization

PAT = r"""'(?i:[sdmt]|ll|ve|re)|[^\r\n\p{L}\p{N}]?+\p{L}+|\p{N}{1,3}| ?[^\s\p{L}\p{N}]++[\r\n]*|\s*[\r\n]|\s+(?!\S)|\s+"""

tok = Tokenizer.from_file("my_model.tbm", pat_str=PAT)

With Special Tokens

special_tokens = {"<eot>": 1000, "<fim_prefix>": 1001, "<fim_suffix>": 1002}
tok = Tokenizer(merges, special_tokens=special_tokens)
ids = tok.encode("<fim_prefix> hello world <eot>")

With Byte Remapping (TikToken Compat)

from tinybpe import load_model

merges, bytes_maps = load_model("cl100k_base.tbm")
tok = Tokenizer(merges, bytes_maps=bytes_maps)

API Reference

Tokenizer

class Tokenizer:
    def __init__(self, merges, *, bytes_maps=None, pat_str=None, special_tokens=None)
    def encode(self, text: str) -> list[int]
    def encode_ordinary(self, text: str) -> list[int]
    def decode(self, ids: list[int]) -> str
    def stream_decode(self, callback: Callable[[str], None]) -> Callable[[int], None]
    def stream_decode_reset(self) -> None
    def save(self, path: str) -> None
    def save_vocab(self, path: str) -> None

    @classmethod
    def from_file(cls, path: str, *, pat_str=None, special_tokens=None) -> Tokenizer
    @classmethod
    def from_pretrained(cls, name: str) -> Tokenizer

    @property
    def merges(self) -> list[tuple[int, int]]
    @property
    def vocab(self) -> dict[int, bytes]
    @property
    def n_vocab(self) -> int

Trainer

class Trainer(bpe.Trainer):
    def __init__(self, text, *, preprocess=None, callback=None)
    def step(self) -> tuple | None
    def train(self, n: int) -> int
    def save(self, path: str) -> None

    @property
    def merges(self) -> list[tuple[int, int]]
    @property
    def n_merges(self) -> int

Model Discovery

def list_models() -> list[str]

File I/O

def load_model(path: str) -> tuple[list[tuple[int, int]], list[int] | None]
def save_model(path: str, merges, bytes_maps=None) -> None
def load_vocab(path: str) -> dict[int, bytes]
def save_vocab(path: str, vocab: dict[int, bytes]) -> None

Model Format

.tbm (TinyBPE Model) is a human-readable text file:

TinyBPE Model v1
0               # 0 = no remap, 256 = has remap
104 101         # merge pairs, one per line
256 108
...

See docs/file-formats.md for the full specification.

Conversion Scripts

Convert existing tokenizers to TinyBPE format:

# TikToken
python scripts/convert_tiktoken.py cl100k_base -o models/cl100k_base.tbm

# HuggingFace
python scripts/convert_hf_tokenizer.py tokenizer.json -o output.tbm
python scripts/convert_hf_tokenizer.py Qwen/Qwen3.5-0.8B -o models/qwen35.tbm

See scripts/README.md for details.

Performance

The C core uses an AVL tree for O(log n) pair lookup during training and greedy lowest-rank-first merging during encoding. Typical throughput on a modern CPU:

Run benchmarks locally:

python benchmarks/bench_train.py
python benchmarks/bench_encode.py
python benchmarks/bench_decode.py

Development

git clone https://github.com/neluca/tinybpe.git
cd tinybpe
pip install -e ".[dev]"
make test && make lint && make typecheck

See CONTRIBUTING.md for full development setup and PR guidelines.

License

MIT — see LICENSE.