mbpe
A high-performance, trainable, tiktoken-compatible BPE tokenizer written in Mojo.
The compile-time PreTokenizer trait enables GPT-2, GPT-4, GPT-4o and custom tokenization pipelines without changing the core tokenizer.
import mbpe
tokenizer = mbpe.get_encoding("gpt2")
tokens = tokenizer.encode("hello world")
print(tokens) # [31373, 995]
print(tokenizer.decode(tokens)) # "hello world"
Why mbpe?
- Drop-in for
tiktoken— sameget_encoding(), sameencode()/allowed_special/disallowed_special, same.tiktokenfile format. Point existing code atmbpeand it works. - Fast — Mojo native beats
tiktoken-rs(Rust) on both encode and decode across all three encodings. See Benchmarks - Fast Python bindings - Substantially outperform Python
tiktoken, while still remaining competitive withtiktoken-rs. - Train your own —
tokenizer.train(["hello world"], vocab_size=300), then save directly to.tiktokenformat. - Extensible by design —
PreTokenizeris a Mojo trait, not a hardcoded implementation. Ships with r50k_base, cl100k_base, and o200k_base; write your own to match it. - Byte-level, lossless — all 256 bytes are base vocabulary. No UNK token. Any valid UTF-8 input round-trips exactly.
Installation
pip install mbpe
Linux x86_64 only (Mojo runtime bundled). Requires Python ≥ 3.9.
Mojo (conda):
pixi add mbpe --channel https://repo.prefix.dev/modular-community
Requires mojo-compiler >=1.0.0b2. The .tiktoken data files are auto-discovered when the environment is activated (via MBPE_DATA_DIR). Set this env var manually if using outside conda.
Usage
Load a pre-built encoding
import mbpe
# GPT-2 (r50k_base, ~50K vocab)
tokenizer = mbpe.get_encoding("gpt2")
print(tokenizer.n_vocab) # 50257
print(tokenizer.encode("hello world")) # [31373, 995]
# GPT-4 (cl100k_base, ~100K vocab)
tokenizer = mbpe.get_encoding("cl100k")
print(tokenizer.n_vocab) # 100277
# GPT-4o (o200k_base, ~200K vocab)
tokenizer = mbpe.get_encoding("o200k")
print(tokenizer.n_vocab) # 200019
Train from scratch
tokenizer = mbpe.GPreTokenizer()
corpus = ["the cat sat on the mat", "the dog sat on the log"]
tokenizer.train(corpus, vocab_size=300)
print(tokenizer.encode("the cat sat")) # [259, 270, 265]
tokenizer.save_tiktoken("my_tokenizer.tiktoken")
# Or train with a GPT-4-style `PreTokenizer`
tokenizer = mbpe.GPT4Tokenizer()
tokenizer.train(corpus, vocab_size=300)
Encode with special tokens
tokenizer = mbpe.get_encoding("gpt2")
# Encode handles special tokens (e.g. <|endoftext|>)
tokenizer.encode("<|endoftext|>hello world") # [50256, 31373, 995]
# encode_ordinary ignores them
tokenizer.encode_ordinary("<|endoftext|>hello") # [27, 91, 437, 1659, 5239, 91, 29, 31373]
# Control which specials are allowed
tokenizer.encode("<|endoftext|>hello world", allowed_special={"<|endoftext|>"})
tokenizer.encode("<|endoftext|>hello world", allowed_special="all") # default
Decode
tokenizer = mbpe.get_encoding("gpt2")
tokenizer.decode([31373, 995]) # "hello world"
tokenizer.decode_bytes([31373, 995]) # b"hello world"
tokenizer.decode_with_offsets([31373, 995]) # ("hello world", [(0, 5), (5, 11)])
tokenizer.decode_single_token_bytes(31373) # b"hello"
Save and load
tokenizer = mbpe.get_encoding("gpt2")
tokenizer.save_tiktoken("backup.tiktoken")
tokenizer2 = mbpe.GPT2Tokenizer()
tokenizer2.load_tiktoken("backup.tiktoken")
assert tokenizer2.n_vocab == tokenizer.n_vocab
Register custom special tokens
tokenizer = mbpe.get_encoding("gpt2")
tokenizer.register_special_tokens({"<|im_start|>": 50257, "<|im_end|>": 50258})
print(tokenizer.encode("<|im_start|> hello")) # [50257, 23748]
Mojo API:
Load an encoding
BPETokenizer[PT] / Tokenizers.get[PT] → returns comptime parameterized tokenizer. Loads from .tiktoken files located via MBPE_DATA_DIR env var, falling back to ./data/
from bpe.tokenizer import Tokenizers
var gpt2 = Tokenizers.get[Tokenizers.gpt2]()
var ids = gpt2.encode("hello world")
print(gpt2.decode(ids))
Train:
from bpe.tokenizer import BPETokenizer
from bpe.pretokenizer import GPT2Pretokenizer
var tok = BPETokenizer[GPT2Pretokenizer]()
tok.train((["hello world"]), 300)
Benchmarks
5 MB corpus (Alice in Wonderland), best-of-3 encode + decode, pre-trained 50K+ vocabularies.
Across all three OpenAI encodings, native Mojo is consistently the fastest implementation for both encoding and decoding, while the Python bindings substantially outperform Python tiktoken.
gpt2 (r50k_base)
| Implementation | Tokens | Encode (M tok/s) | Decode (M tok/s) |
|---|---|---|---|
| mbpe — Mojo native | 1.54M | 6.9 | 110.1 |
| mbpe — Python bindings | 1.54M | 6.2 | 61.6 |
| tiktoken (Python) | 1.54M | 2.8 | 27.2 |
| tiktoken-rs | 1.53M | 2.5 | 54.1 |
cl100k (cl100k_base)
| Implementation | Tokens | Encode (M tok/s) | Decode (M tok/s) |
|---|---|---|---|
| mbpe — Mojo native | 1.28M | 5.6 | 102.5 |
| mbpe — Python bindings | 1.28M | 4.7 | 51.6 |
| tiktoken (Python) | 1.28M | 2.3 | 28.7 |
| tiktoken-rs | 1.28M | 2.5 | 51.6 |
o200k (o200k_base)
| Implementation | Tokens | Encode (M tok/s) | Decode (M tok/s) |
|---|---|---|---|
| mbpe — Mojo native | 1.28M | 4.6 | 97.4 |
| mbpe — Python bindings | 1.28M | 4.3 | 54.7 |
| tiktoken (Python) | 1.28M | 3.6 | 26.3 |
| tiktoken-rs | 1.28M | 4.0 | 45.1 |
Training throughput (Mojo, self-trained, GPT4Pretokenizer (cl100k_base / o200k_base), 5 MB corpus):
| Vocab size | 500 | 1000 | 2000 | 4000 |
|---|---|---|---|---|
| Train time | 5271 ms | 12204 ms | 24230 ms | 45200 ms |
| Merges/s | 46 | 60 | 71 | 82 |
| Encode (M tok/s) | 10.8 | 8.4 | 6.8 | 5.9 |
Environment: Intel Xeon @ 3.10 GHz, 8 cores, 31 Gi RAM, Ubuntu 24.04. Mojo 1.0.0b2, Python 3.14.6, Rust 1.97.1, tiktoken 0.13.0.
Architecture
At the core of mbpe is BPETokenizer[PT], where PT is any implementation of the compile-time PreTokenizer trait.
Text
│
▼
PreTokenizer(trait)
│
▼
Symbol Sequence
│
▼
BPETokenizer[PT]
┌────────┴─────────┐
▼ ▼
Training Encode/Decode
A
PreTokenizerconverts input text into a sequence of symbols (e.g. regex chunks, Ġ-prefixed words, or future custom segmentations) before BPE merging begins.
comptime aliases:
Tokenizers.get[Tokenizers.gpt2]()→BPETokenizer[GPT2Pretokenizer]→ r50k_baseTokenizers.get[Tokenizers.cl100k]()→BPETokenizer[GPT4Pretokenizer[SEQUENTIAL]]→ cl100k_baseTokenizers.get[Tokenizers.o200k]()→BPETokenizer[GPT4Pretokenizer[SHUFFLED]]→ o200k_base
Implementation highlights
- Compile-time
PreTokenizertrait dispatch - Two-tier
MergeLookupcache (flat array + Dict) - Incremental pair statistics for training - (O(N) vs O(V×W))
- Flat memcpy-chain decoder
- Byte-level, lossless vocabulary
Development
git clone https://github.com/ratulb/mbpe
cd mbpe
# Install dependencies
pixi install
# Build Python shared library
pixi run mojo build python-binding/mbpe.mojo -I . \
--emit shared-lib -o python-binding/mbpe/_mbpe.so
# Run Mojo tests (78 total)
# pixi run mojo main.mojo # 36 tests
# pixi run mojo -I . tests/test_tokenizer.mojo # 9 tests
# pixi run mojo -I . tests/exhaustive_tokenizer.mojo # 33 tests
# Run Python tests
# pixi run --environment dev python -m pytest tests/python/ -v
API
| Python class | Mojo backend | Pre-tokenizer | name() |
|---|---|---|---|
GPreTokenizer |
BPETokenizer[GPreTokenizer] |
Ġ (space → U+0120 + split) | "gpre" |
GPT2Tokenizer |
BPETokenizer[GPT2Pretokenizer] |
r50k_base regex (7 patterns) | "gpt2" |
GPT4Tokenizer |
BPETokenizer[GPT4Pretokenizer[SEQUENTIAL]] |
cl100k_base regex (8 patterns) | "cl100k" |
GPT4oTokenizer |
BPETokenizer[GPT4Pretokenizer[SHUFFLED]] |
o200k_base regex (8 patterns, shuffled byte mapping) | "o200k" |
All four classes share the same method interface:
| Method | Description |
|---|---|
train(texts, vocab_size) |
Train BPE from scratch |
encode(text, **kwargs) |
Encode with special token handling |
encode_ordinary(text) |
Encode ignoring special tokens |
decode(ids) |
Decode token IDs to string |
decode_bytes(ids) |
Decode to raw bytes |
decode_single_token_bytes(id) |
Raw bytes for one token |
decode_with_offsets(ids) |
(text, [(start, end), ...]) — decoded text with byte offsets per token |
encode_single_token(text) |
Look up a token string's ID |
token_byte_values() |
Raw bytes for all token IDs |
name() |
Pre-tokenizer name |
n_vocab (property) |
Vocabulary size |
save_tiktoken(path) |
Save in .tiktoken format |
load_tiktoken(path) |
Load from .tiktoken file |
register_special_tokens(dict) |
Register special tokens |
Module-level functions:
| Function | Description |
|---|---|
get_encoding(name) |
Load pre-trained encoding (gpt2, cl100k, o200k) |
train(texts, vocab_size) |
Train with GPreTokenizer (default) |
_train_impl(texts, vocab_size, pt) |
Train with specific pre-tokenizer |
Roadmap
- Resume BPE training from existing vocabularies
- Unicode-native tokenization through custom
PreTokenizers - Tokenizers specialized for new languages and domains
Related projects
- tiktoken — OpenAI's BPE tokenizer (Rust + Python); the format and API mbpe is compatible with
- tiktoken-rs — Rust port of tiktoken
- minbpe — Karpathy's minimal, from-scratch BPE reference implementation in pure Python
Metadata
Release files for mbpe 0.1.2
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| mbpe-0.1.2-py3-none-manylinux_2_35_x86_64.whl | Python 3 | none | Linux glibc 2.35+ x86-64 | Details for mbpe-0.1.2-py3-none-manylinux_2_35_x86_64.whl |
Release files / mbpe-0.1.2-py3-none-manylinux_2_35_x86_64.whl
| Download URL | mbpe-0.1.2-py3-none-manylinux_2_35_x86_64.whl |
|---|---|
| Size | 22.3 MB |
| Tags | Linux glibc 2.35+ x86-64 Python 3 |
|
SHA-256 checksum How to use checksums |
ebc009ffedbe14fb509571fad186f014e63484331167abcd79cbe533d81a5eec
|
|
BLAKE2b-256 checksum How to use checksums |
712b895998acf7e5c844f05b4f69a1f10d0ee47f101f96a562e413fdfebca04c
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Security for mbpe 0.1.2
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.