leximini 0.1.1

LexiMini: A fast, minimal Rust-based BPE tokenizer for Python

LexiMini

A fast, minimal Byte-Pair Encoding (BPE) tokenizer implemented in Rust, with seamless Python bindings via PyO3.

Highlights

• ⚡ Fast — Core tokenization logic written in Rust for maximum performance
• ** Minimal** — Clean, readable implementation — great for learning how BPE works
• ** Pythonic** — Drop-in Python module via native PyO3 extensions
• ** Trainable** — Train your own BPE vocabulary directly from Python

Quick Start

pip install leximini

import leximini

# Initialize
tokenizer = leximini.get_encoding("gpt2")

# Train on your corpus
tokenizer.train("The quick brown fox jumps over the lazy dog.", 270)

# Encode & Decode
tokens = tokenizer.encode("The quick brown fox")
decoded = tokenizer.decode(tokens)
assert decoded == "The quick brown fox"

Requirements

• Python 3.8+
• Rust and Cargo (for building from source): Install from rustup.rs

Building from Source

From the leximini directory (the folder containing pyproject.toml and Cargo.toml):

pip install maturin
pip install .

Behind the scenes, the maturin build system will automatically invoke Cargo to compile the Rust extension and install it into your active Python environment.

API Reference

Initialization

import leximini
tokenizer = leximini.get_encoding("gpt2")

Training (BPE)

Train byte-pair merges from a sample corpus. Target vocabulary size must be ≥ 256 (base ASCII bytes).

tokenizer.train("your training text here", 270)  # 256 base + 14 merges

Encoding & Decoding

tokens = tokenizer.encode("The quick brown fox")   # → list of ints
text = tokenizer.decode(tokens)                     # → original string

License

MIT