mbpe
A trainable byte-pair encoding engine in Mojo, generic over a compile-time pre-tokenizer. The BPE engine knows only BPE; tokenizer-family differences live in the pre-tokenizer. Ships with GPT-2, GPT-4, and GPT-4o pre-tokenizers.
import mbpe
tokenizer = mbpe.get_encoding("gpt2")
tokens = tokenizer.encode("hello world")
print(tokens) # [31373, 995]
print(tokenizer.decode(tokens)) # "hello world"
Why mbpe?
Drop-in for tiktoken. Same get_encoding(), encode() / allowed_special / disallowed_special handling, same .tiktoken file format. Point existing code at mbpe and it works.
Train your own. tokenizer.train(["hello world"], vocab_size=300), then save directly to .tiktoken format. tiktoken ships vocabularies; mbpe builds them.
A BPE engine that doesn't know about encodings. The core is BPETokenizer[PT: PreTokenizer], generic over a compile-time Mojo trait. The engine knows only about BPE. Byte mappings, split rules, and special-token sets — the things that distinguish r50k_base, cl100k_base, and o200k_base — are PreTokenizer implementations. Adding a new encoding means providing a new PreTokenizer, not modifying the engine. The BPE core has no knowledge of .tiktoken files or specific encodings.
Byte-level and lossless. All 256 bytes are in the base vocabulary. No <UNK> token. Any valid UTF-8 input round-trips exactly.
Fast. Mojo-native encode and decode are competitive with existing tokenizers across shipped encodings. Python bindings substantially outperform Python tiktoken and remain competitive with tiktoken-rs.
Benchmarks
5 MB corpus (Alice in Wonderland), best-of-3 encode + decode, pre-trained 50K+ vocabularies.
Across all three OpenAI encodings, native Mojo is consistently the fastest implementation for both encoding and decoding, while the Python bindings substantially outperform Python tiktoken.
Environment: AMD EPYC 9B45, 4 cores, 15Gi RAM, Debian GNU/Linux 12 (bookworm). Mojo 1.1.0, Python 3.14.7, Rust 1.99.0, tiktoken 0.14.0.
Installation
pip install mbpe
Linux x86_64 only (Mojo runtime bundled). Requires Python ≥ 3.9.
Mojo (conda):
pixi add mbpe --channel https://repo.prefix.dev/modular-community
Requires mojo-compiler >=1.2.0. The .tiktoken data files are auto-discovered when the environment is activated (via MBPE_DATA_DIR). Set this env var manually if using outside conda.
Usage
Load a pre-built encoding
import mbpe
# GPT-2 (r50k_base, ~50K vocab)
tokenizer = mbpe.get_encoding("gpt2")
print(tokenizer.n_vocab) # 50257
print(tokenizer.encode("hello world")) # [31373, 995]
# GPT-4 (cl100k_base, ~100K vocab)
tokenizer = mbpe.get_encoding("cl100k")
print(tokenizer.n_vocab) # 100277
# GPT-4o (o200k_base, ~200K vocab)
tokenizer = mbpe.get_encoding("o200k")
print(tokenizer.n_vocab) # 200019
Train from scratch
tokenizer = mbpe.GPT2Tokenizer()
corpus = ["the cat sat on the mat", "the dog sat on the log"]
tokenizer.train(corpus, vocab_size=300)
print(tokenizer.encode("the cat sat")) # [258, 269, 263]
tokenizer.save_tiktoken("my_tokenizer.tiktoken")
# Or train with a GPT-4-style `PreTokenizer`
tokenizer = mbpe.GPT4Tokenizer()
tokenizer.train(corpus, vocab_size=300)
Encode with special tokens
tokenizer = mbpe.get_encoding("gpt2")
# Encode handles special tokens (e.g. <|endoftext|>)
tokenizer.encode("<|endoftext|>hello world") # [50256, 31373, 995]
# encode_ordinary ignores them
tokenizer.encode_ordinary("<|endoftext|>hello") # [27, 91, 437, 1659, 5239, 91, 29, 31373]
# Control which specials are allowed
tokenizer.encode("<|endoftext|>hello world", allowed_special={"<|endoftext|>"})
tokenizer.encode("<|endoftext|>hello world", allowed_special="all") # default
Decode
tokenizer = mbpe.get_encoding("gpt2")
tokenizer.decode([31373, 995]) # "hello world"
tokenizer.decode_bytes([31373, 995]) # b"hello world"
tokenizer.decode_with_offsets([31373, 995]) # ("hello world", [(0, 5), (5, 11)])
tokenizer.decode_single_token_bytes(31373) # b"hello"
Save and load
tokenizer = mbpe.get_encoding("gpt2")
tokenizer.save_tiktoken("backup.tiktoken")
tokenizer2 = mbpe.GPT2Tokenizer()
tokenizer2.load_tiktoken("backup.tiktoken")
assert tokenizer2.n_vocab == tokenizer.n_vocab
Register custom special tokens
tokenizer = mbpe.get_encoding("gpt2")
tokenizer.register_special_tokens({"<|im_start|>": 50257, "<|im_end|>": 50258})
print(tokenizer.encode("<|im_start|> hello")) # [50257, 23748]
Mojo API:
Load an encoding
BPETokenizer[PT] / Tokenizers.get[PT] → returns comptime parameterized tokenizer. Loads from .tiktoken files located via MBPE_DATA_DIR env var, falling back to ./data/
from bpe.tokenizer import Tokenizers
def main() raises:
var gpt2 = Tokenizers.get[Tokenizers.gpt2]()
var ids = gpt2.encode("hello world")
print(gpt2.decode(ids))
Train:
from bpe.tokenizer import BPETokenizer
from bpe.pretokenizer import GPT2Pretokenizer
def main() raises:
var tok = BPETokenizer[GPT2Pretokenizer]()
var corpus = List[String]()
corpus.append("hello world")
tok.train(corpus, 300)
Try it yourself
benchmarks/compare_mbpe_tiktoken.py — self-contained script that downloads a corpus from Project Gutenberg and compares mbpe vs tiktoken encode/decode throughput. Run it anywhere:
pip install mbpe tiktoken
python benchmarks/compare_mbpe_tiktoken.py
Paste the body into a Colab or Kaggle cell (prepend !pip install mbpe tiktoken) for the same comparison in a notebook.
Architecture
At the core of mbpe is BPETokenizer[PT], where PT is any implementation of the compile-time PreTokenizer trait.
Text
│
▼
PreTokenizer(trait)
│
▼
Symbol Sequence
│
▼
BPETokenizer[PT]
┌────────┴─────────┐
▼ ▼
Training Encode/Decode
A
PreTokenizerconverts input text into a sequence of symbols (e.g. regex chunks, Ġ-prefixed words, or future custom segmentations) before BPE merging begins.
comptime aliases:
Tokenizers.get[Tokenizers.gpt2]()→BPETokenizer[GPT2Pretokenizer]→ r50k_baseTokenizers.get[Tokenizers.cl100k]()→BPETokenizer[GPT4Pretokenizer[SEQUENTIAL]]→ cl100k_baseTokenizers.get[Tokenizers.o200k]()→BPETokenizer[GPT4Pretokenizer[SHUFFLED]]→ o200k_base
Implementation highlights
- Compile-time
PreTokenizertrait dispatch - Two-tier
MergeLookupcache (flat array + Dict) - Incremental pair statistics for training - (O(N) vs O(V×W))
- Flat memcpy-chain decoder
- Byte-level, lossless vocabulary
Development
git clone https://github.com/ratulb/mbpe
cd mbpe
# Install dependencies
pixi install
# Build Python shared library
pixi run mojo build python-binding/mbpe.mojo -I . \
--emit shared-lib -o python-binding/mbpe/_mbpe.so
# Run Mojo tests (77 total)
# pixi run mojo main.mojo # 33 tests
# pixi run mojo -I . tests/test_tokenizer.mojo # 10 tests
# pixi run mojo -I . tests/exhaustive_tokenizer.mojo # 34 tests
# Run Python tests
# pixi run --environment dev python -m pytest tests/python/ -v
Contributions are welcome — see CONTRIBUTING.md for how to get started, run the suite, and open a pull request.
API
| Python class | Mojo backend | Pre-tokenizer | name() |
|---|---|---|---|
GPT2Tokenizer |
BPETokenizer[GPT2Pretokenizer] |
r50k_base regex (7 patterns) | "gpt2" |
GPT4Tokenizer |
BPETokenizer[GPT4Pretokenizer[SEQUENTIAL]] |
cl100k_base regex (8 patterns) | "cl100k" |
GPT4oTokenizer |
BPETokenizer[GPT4Pretokenizer[SHUFFLED]] |
o200k_base regex (8 patterns, shuffled byte mapping) | "o200k" |
All three classes share the same method interface:
| Method | Description |
|---|---|
train(texts, vocab_size) |
Train BPE from scratch |
encode(text, **kwargs) |
Encode with special token handling |
encode_ordinary(text) |
Encode ignoring special tokens |
decode(ids) |
Decode token IDs to string |
decode_bytes(ids) |
Decode to raw bytes |
decode_single_token_bytes(id) |
Raw bytes for one token |
decode_with_offsets(ids) |
(text, [(start, end), ...]) — decoded text with byte offsets per token |
encode_single_token(text) |
Look up a token string's ID |
token_byte_values() |
Raw bytes for all token IDs |
name() |
Pre-tokenizer name |
n_vocab (property) |
Vocabulary size |
save_tiktoken(path) |
Save in .tiktoken format |
load_tiktoken(path) |
Load from .tiktoken file |
register_special_tokens(dict) |
Register special tokens |
Module-level functions:
| Function | Description |
|---|---|
get_encoding(name) |
Load pre-trained encoding (gpt2, cl100k, o200k) |
train(texts, vocab_size) |
Train with GPT2Pretokenizer (default) |
_train_impl(texts, vocab_size, pt) |
Train with specific pre-tokenizer |
Roadmap
- Resume BPE training from existing vocabularies
- Unicode-native tokenization through custom
PreTokenizers - Tokenizers specialized for new languages and domains
Related projects
- tiktoken — OpenAI's BPE tokenizer (Rust + Python); the format and API mbpe is compatible with
- tiktoken-rs — Rust port of tiktoken
- minbpe — Karpathy's minimal, from-scratch BPE reference implementation in pure Python
Metadata
Release files for mbpe 0.1.11
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| mbpe-0.1.11-py3-none-manylinux_2_35_x86_64.whl | Python 3 | none | Linux glibc 2.35+ x86-64 | Details for mbpe-0.1.11-py3-none-manylinux_2_35_x86_64.whl |
Release files / mbpe-0.1.11-py3-none-manylinux_2_35_x86_64.whl
| Download URL | mbpe-0.1.11-py3-none-manylinux_2_35_x86_64.whl |
|---|---|
| Size | 6.3 MB |
| Tags | Linux glibc 2.35+ x86-64 Python 3 |
|
SHA-256 checksum How to use checksums |
84ef46e44a718fdf58681897814978d8be22f49fa27735b84db014cc32cca866
|
|
BLAKE2b-256 checksum How to use checksums |
7847f62e79e29cd74c53bde202c0b75e977bed42aea0052a7f4f9926636bad54
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Security for mbpe 0.1.11
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.