bpe-openai 0.1.0

Python bindings wrapping the bpe-openai Rust tokenizer crate with a tiktoken-compatible API.

bpe-openai

This directory contains the Python package that exposes the Rust bpe-openai tokenizer via PyO3 bindings. The package is a thin wrapper over the Rust crate; the compiled extension must be built before running tests or using the API.

Local development

python -m venv .venv
source .venv/bin/activate
pip install -r requirements-dev.txt  # optional, see below
maturin develop --release
pytest

Running maturin develop --release builds the Rust extension in-place so the package can be imported locally.

Packaging

./python/scripts/build_wheels.sh

Requirements:

• Docker daemon access (the script exits if it cannot talk to /var/run/docker.sock).
• Sufficient permissions to pull quay.io/pypa/manylinux2014_x86_64 (override with MANYLINUX_IMAGE if needed).
• Optional: set MANYLINUX_PYTHON to pick a different interpreter inside the container (defaults to /opt/python/cp39-cp39/bin/python3.9).

The container installs maturin on the fly and runs maturin build --release, producing wheels under python/target/wheels/ that meet manylinux requirements and therefore run on both old and new glibc releases.

Additional scripts are mirrored at the repo root (scripts/compare_with_tiktoken.py, scripts/benchmark_scaling.py) for convenience when working outside the python/ directory.

Parity Comparison

To compare this package’s behaviour with upstream tiktoken, install tiktoken in a virtual environment alongside the wheel and run:

./python/scripts/compare_with_tiktoken.py

The command prints per-model encode/decode parity summaries and highlights any differences. Pass --json to capture machine-readable output.

To inspect performance scaling, run:

./python/scripts/benchmark_scaling.py --batch-size 32 --output python/target/benchmark_scaling.png

The script now captures both single-document and batched (encode_batch) timings for tiktoken and bpe_openai across the requested input lengths and, when matplotlib is installed, plots the results (a thin wrapper lives at the repo root under scripts/benchmark_scaling.py).

tiktoken Compatibility Snapshot

Feature / API	Support Status	Notes
encoding_for_model	⚠️	Works for backend-exposed models (cl100k_base, o200k_base family, voyage3_base); legacy GPT-2 / p50k families are skipped
get_encoding	⚠️	Supports cl100k_base, o200k_base, voyage3_base; legacy GPT-2 / r50k / p50k encodings not yet implemented
Encoding.encode	✅	Calls the Rust bpe-openai tokenizer for exact parity
Encoding.decode	✅	Mirrors tiktoken behaviour (UTF-8 validation included)
Encoding.encode_batch / decode_batch	✅	Batch helpers mirror tiktoken signatures
Unsupported model errors	✅	Raises UnsupportedModelError with supported_models list
Custom special tokens	❌ Not yet	Registering extra special tokens is not supported
Chunk limit enforcement	✅	Uses tokenizer counts and raises TokenLimitError
Metrics hook (set_metrics_hook)	✅	Emits model, latency, token counts
Rust-backed performance parity	✅	Wheels/binaries use the upstream Rust implementation
Quickstart smoke test (drop-in replacement)	✅	Existing scripts run unmodified in tests

Legend: ✅ fully supported • ⚠️ requires follow-up (documented limitation) • ❌ not yet implemented |