Skip to main content

mbpe

Tests CodeQL

A high-performance, trainable, tiktoken-compatible BPE tokenizer written in Mojo. The compile-time PreTokenizer trait enables GPT-2, GPT-4, GPT-4o and custom tokenization pipelines without changing the core tokenizer.

import mbpe

tokenizer = mbpe.get_encoding("gpt2")
tokens = tokenizer.encode("hello world")
print(tokens)                          # [31373, 995]
print(tokenizer.decode(tokens))        # "hello world"

Why mbpe?

  • Drop-in for tiktoken — same get_encoding(), same encode()/allowed_special/disallowed_special, same .tiktoken file format. Point existing code at mbpe and it works.
  • Fast — Mojo native beats tiktoken-rs (Rust) on both encode and decode across all three encodings. See Benchmarks
  • Fast Python bindings - Substantially outperform Python tiktoken, while still remaining competitive with tiktoken-rs.
  • Train your own — tokenizer.train(["hello world"], vocab_size=300), then save directly to .tiktoken format.
  • Extensible by design — PreTokenizer is a Mojo trait, not a hardcoded implementation. Ships with r50k_base, cl100k_base, and o200k_base; write your own to match it.
  • Byte-level, lossless — all 256 bytes are base vocabulary. No UNK token. Any valid UTF-8 input round-trips exactly.

Note: the legacy GPreTokenizer (Ġ space convention) was removed. The default pre-tokenizer is now the GPT-2 GPT2Pretokenizer; mbpe.GPreTokenizer and the "gpre" encoding no longer exist.


Installation

pip install mbpe

Linux x86_64 only (Mojo runtime bundled). Requires Python ≥ 3.9.

Mojo (conda):

pixi add mbpe --channel https://repo.prefix.dev/modular-community

Requires mojo-compiler >=1.0.0b2. The .tiktoken data files are auto-discovered when the environment is activated (via MBPE_DATA_DIR). Set this env var manually if using outside conda.


Usage

Load a pre-built encoding

import mbpe

# GPT-2 (r50k_base, ~50K vocab)
tokenizer = mbpe.get_encoding("gpt2")
print(tokenizer.n_vocab)                          # 50257
print(tokenizer.encode("hello world"))            # [31373, 995]

# GPT-4 (cl100k_base, ~100K vocab)
tokenizer = mbpe.get_encoding("cl100k")
print(tokenizer.n_vocab)                          # 100277

# GPT-4o (o200k_base, ~200K vocab)
tokenizer = mbpe.get_encoding("o200k")
print(tokenizer.n_vocab)                          # 200019

Train from scratch

tokenizer = mbpe.GPT2Tokenizer()
corpus = ["the cat sat on the mat", "the dog sat on the log"]
tokenizer.train(corpus, vocab_size=300)
print(tokenizer.encode("the cat sat"))            # [258, 269, 263]
tokenizer.save_tiktoken("my_tokenizer.tiktoken")

# Or train with a GPT-4-style `PreTokenizer`
tokenizer = mbpe.GPT4Tokenizer()
tokenizer.train(corpus, vocab_size=300)

Encode with special tokens

tokenizer = mbpe.get_encoding("gpt2")

# Encode handles special tokens (e.g. <|endoftext|>)
tokenizer.encode("<|endoftext|>hello world")      # [50256, 31373, 995]

# encode_ordinary ignores them
tokenizer.encode_ordinary("<|endoftext|>hello")   # [27, 91, 437, 1659, 5239, 91, 29, 31373]

# Control which specials are allowed
tokenizer.encode("<|endoftext|>hello world", allowed_special={"<|endoftext|>"})
tokenizer.encode("<|endoftext|>hello world", allowed_special="all")   # default

Decode

tokenizer = mbpe.get_encoding("gpt2")

tokenizer.decode([31373, 995])                    # "hello world"
tokenizer.decode_bytes([31373, 995])              # b"hello world"
tokenizer.decode_with_offsets([31373, 995])       # ("hello world", [(0, 5), (5, 11)])
tokenizer.decode_single_token_bytes(31373)        # b"hello"

Save and load

tokenizer = mbpe.get_encoding("gpt2")
tokenizer.save_tiktoken("backup.tiktoken")

tokenizer2 = mbpe.GPT2Tokenizer()
tokenizer2.load_tiktoken("backup.tiktoken")
assert tokenizer2.n_vocab == tokenizer.n_vocab

Register custom special tokens

tokenizer = mbpe.get_encoding("gpt2")
tokenizer.register_special_tokens({"<|im_start|>": 50257, "<|im_end|>": 50258})
print(tokenizer.encode("<|im_start|> hello"))     # [50257, 23748]

Mojo API:

Load an encoding

BPETokenizer[PT] / Tokenizers.get[PT] → returns comptime parameterized tokenizer. Loads from .tiktoken files located via MBPE_DATA_DIR env var, falling back to ./data/

from bpe.tokenizer import Tokenizers


def main() raises:
    var gpt2 = Tokenizers.get[Tokenizers.gpt2]()
    var ids = gpt2.encode("hello world")
    print(gpt2.decode(ids))

Train:

from bpe.tokenizer import BPETokenizer
from bpe.pretokenizer import GPT2Pretokenizer


def main() raises:
    var tok = BPETokenizer[GPT2Pretokenizer]()
    var corpus = List[String]()
    corpus.append("hello world")
    tok.train(corpus, 300)

Benchmarks

5 MB corpus (Alice in Wonderland), best-of-3 encode + decode, pre-trained 50K+ vocabularies.

Across all three OpenAI encodings, native Mojo is consistently the fastest implementation for both encoding and decoding, while the Python bindings substantially outperform Python tiktoken.

gpt2 (r50k_base)

Implementation Tokens Encode (M tok/s) Decode (M tok/s)
mbpe — Mojo native 1.54M 13.0 161.6
mbpe — Python bindings 1.54M 10.8 83.0
tiktoken (Python) 1.54M 4.8 39.2
tiktoken-rs 1.53M 4.3 67.5

cl100k (cl100k_base)

Implementation Tokens Encode (M tok/s) Decode (M tok/s)
mbpe — Mojo native 1.28M 10.7 165.2
mbpe — Python bindings 1.28M 9.4 87.4
tiktoken (Python) 1.28M 4.2 34.6
tiktoken-rs 1.28M 4.1 66.3

o200k (o200k_base)

Implementation Tokens Encode (M tok/s) Decode (M tok/s)
mbpe — Mojo native 1.28M 8.8 163.2
mbpe — Python bindings 1.28M 7.6 78.7
tiktoken (Python) 1.28M 5.7 37.4
tiktoken-rs 1.28M 7.2 71.5

Training throughput (Mojo, self-trained, GPT4Pretokenizer (cl100k_base / o200k_base), 5 MB corpus):

Vocab size 500 1000 2000 4000
Train time 42 ms 46 ms 51 ms 71 ms
Merges/s 5799 16075 33643 52416
Encode (M tok/s) 23.3 18.6 15.1 12.9

Environment: INTEL(R) XEON(R) PLATINUM 8581C CPU @ 2.30GHz, 4 cores, 7.8Gi RAM, Debian GNU/Linux 13 (trixie). Mojo 1.0.0b2, Python 3.14.6, Rust 1.97.1, tiktoken 0.13.0.


Try it yourself

benchmarks/compare_mbpe_tiktoken.py — self-contained script that downloads a corpus from Project Gutenberg and compares mbpe vs tiktoken encode/decode throughput. Run it anywhere:

pip install mbpe tiktoken
python benchmarks/compare_mbpe_tiktoken.py

Paste the body into a Colab or Kaggle cell (prepend !pip install mbpe tiktoken) for the same comparison in a notebook.


Architecture

At the core of mbpe is BPETokenizer[PT], where PT is any implementation of the compile-time PreTokenizer trait.

                                         Text
                                          │
                                          ▼
                                  PreTokenizer(trait)
                                          │
                                          ▼
                                   Symbol Sequence
                                          │
                                          ▼
                                   BPETokenizer[PT]
                                 ┌────────┴─────────┐
                                 ▼                  ▼
                             Training          Encode/Decode

A PreTokenizer converts input text into a sequence of symbols (e.g. regex chunks, Ġ-prefixed words, or future custom segmentations) before BPE merging begins.

comptime aliases:

  • Tokenizers.get[Tokenizers.gpt2]() → BPETokenizer[GPT2Pretokenizer] → r50k_base
  • Tokenizers.get[Tokenizers.cl100k]() → BPETokenizer[GPT4Pretokenizer[SEQUENTIAL]] → cl100k_base
  • Tokenizers.get[Tokenizers.o200k]() → BPETokenizer[GPT4Pretokenizer[SHUFFLED]] → o200k_base

Implementation highlights

  • Compile-time PreTokenizer trait dispatch
  • Two-tier MergeLookup cache (flat array + Dict)
  • Incremental pair statistics for training - (O(N) vs O(V×W))
  • Flat memcpy-chain decoder
  • Byte-level, lossless vocabulary

Development

git clone https://github.com/ratulb/mbpe
cd mbpe

# Install dependencies
pixi install

# Build Python shared library
pixi run mojo build python-binding/mbpe.mojo -I . \
  --emit shared-lib -o python-binding/mbpe/_mbpe.so

# Run Mojo tests (77 total)
# pixi run mojo main.mojo            # 33 tests
# pixi run mojo -I . tests/test_tokenizer.mojo           # 10 tests
# pixi run mojo -I . tests/exhaustive_tokenizer.mojo     # 34 tests

# Run Python tests
# pixi run --environment dev python -m pytest tests/python/ -v

API

Python class Mojo backend Pre-tokenizer name()
GPT2Tokenizer BPETokenizer[GPT2Pretokenizer] r50k_base regex (7 patterns) "gpt2"
GPT4Tokenizer BPETokenizer[GPT4Pretokenizer[SEQUENTIAL]] cl100k_base regex (8 patterns) "cl100k"
GPT4oTokenizer BPETokenizer[GPT4Pretokenizer[SHUFFLED]] o200k_base regex (8 patterns, shuffled byte mapping) "o200k"

All three classes share the same method interface:

Method Description
train(texts, vocab_size) Train BPE from scratch
encode(text, **kwargs) Encode with special token handling
encode_ordinary(text) Encode ignoring special tokens
decode(ids) Decode token IDs to string
decode_bytes(ids) Decode to raw bytes
decode_single_token_bytes(id) Raw bytes for one token
decode_with_offsets(ids) (text, [(start, end), ...]) — decoded text with byte offsets per token
encode_single_token(text) Look up a token string's ID
token_byte_values() Raw bytes for all token IDs
name() Pre-tokenizer name
n_vocab (property) Vocabulary size
save_tiktoken(path) Save in .tiktoken format
load_tiktoken(path) Load from .tiktoken file
register_special_tokens(dict) Register special tokens

Module-level functions:

Function Description
get_encoding(name) Load pre-trained encoding (gpt2, cl100k, o200k)
train(texts, vocab_size) Train with GPT2Pretokenizer (default)
_train_impl(texts, vocab_size, pt) Train with specific pre-tokenizer

Roadmap

  • Resume BPE training from existing vocabularies
  • Unicode-native tokenization through custom PreTokenizers
  • Tokenizers specialized for new languages and domains

  • tiktoken — OpenAI's BPE tokenizer (Rust + Python); the format and API mbpe is compatible with
  • tiktoken-rs — Rust port of tiktoken
  • minbpe — Karpathy's minimal, from-scratch BPE reference implementation in pure Python

Metadata

Release files for mbpe 0.1.5

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Built distribution (wheel)

Table of built distributions (wheels) for mbpe 0.1.5
File Interpreter ABI Platform Actions
mbpe-0.1.5-py3-none-manylinux_2_35_x86_64.whl Python 3 none Linux glibc 2.35+ x86-64 Details for mbpe-0.1.5-py3-none-manylinux_2_35_x86_64.whl

Release files / mbpe-0.1.5-py3-none-manylinux_2_35_x86_64.whl

Download URL mbpe-0.1.5-py3-none-manylinux_2_35_x86_64.whl
Size 24.7 MB
Tags Linux glibc 2.35+ x86-64 Python 3
SHA-256 checksum
How to use checksums
5dc7c1d8ed076a318a354796919fc1fab21cbf56467f594580137239e6c70137
BLAKE2b-256 checksum
How to use checksums
297c3d8f31e8975e157fc3bf1e06b051095a9e03a1777c1042f121aa375e054b
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.13.14

Security for mbpe 0.1.5

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page