splintr-rs 0.10.0

Fast Rust tokenizer (BPE + SentencePiece) with Python bindings

A high-performance tokenizer (BPE + SentencePiece + WordPiece) built with Rust with Python bindings, focused on speed, safety, and resource optimization.

The Problem

Tokenization is everywhere in modern AI. Whether you're building LLM applications, training models, or processing data pipelines, you're tokenizing text constantly. But existing tokenizers have a problem: they're slow.

When you need to tokenize batches of prompts, documents, or training data, you're stuck waiting. Python-based tokenizers can't fully leverage modern multi-core CPUs. You need something faster.

The Solution

Splintr brings Rust performance to Python. Built from the ground up for speed and efficiency:

Configuration	Splintr	Tiktoken	HuggingFace	TokenDagger
1,000 texts	111 MB/s	9 MB/s	28 MB/s	9 MB/s
500 texts	107 MB/s	10 MB/s	27 MB/s	8 MB/s
100 texts	69 MB/s	7 MB/s	20 MB/s	6 MB/s

10-12x faster than tiktoken. 4x faster than HuggingFace. Built in Rust, accessible from Python.

Quick Start

Python

pip install splintr-rs

from splintr import Tokenizer

# Load a pretrained vocabulary
tokenizer = Tokenizer.from_pretrained("cl100k_base")  # OpenAI GPT-4/3.5
# tokenizer = Tokenizer.from_pretrained("llama3")      # Meta Llama 3 family
# tokenizer = Tokenizer.from_pretrained("deepseek_v3") # DeepSeek V3/R1
# tokenizer = Tokenizer.from_pretrained("mistral_v1")  # Mistral 7B v0.1/v0.2
# tokenizer = Tokenizer.from_pretrained("mistral_v2")  # Mistral 7B v0.3, Codestral
# tokenizer = Tokenizer.from_pretrained("mistral_v3")  # Mistral NeMo, Large 2
# tokenizer = Tokenizer.from_pretrained("whisper_v3")  # OpenAI Whisper multilingual (v1/v2/v3)

# Encode and decode
tokens = tokenizer.encode("Hello, world!")
text = tokenizer.decode(tokens)

# Batch encode (10-12x faster)
texts = ["Hello, world!", "How are you?", "Machine learning is fun!"]
batch_tokens = tokenizer.encode_batch(texts)

See the API Guide for complete documentation and examples.

Rust

[dependencies]
splintr = "*"  # or pin to a specific version

use splintr::{Tokenizer, CL100K_BASE_PATTERN};

let tokenizer = Tokenizer::new(encoder, special_tokens, CL100K_BASE_PATTERN)?;
let tokens = tokenizer.encode("Hello, world!");
let batch_tokens = tokenizer.encode_batch(&texts);

See the API Guide and docs.rs for complete Rust documentation.

Key Features

Performance where it matters:

• 12x faster batch encoding - Parallel processing across multiple texts using Rayon
• 3-4x faster single text encoding - Optimized sequential algorithm for typical use cases
• Smart parallelization - Sequential for small texts (<1MB), parallel for large datasets
• LRU caching - Avoid redundant encoding of frequently seen text chunks

Built for production:

• Compatible vocabularies - Supports cl100k_base, o200k_base (OpenAI), Llama 3 family (Meta), DeepSeek V3 (DeepSeek), Mistral V1/V2/V3 (Mistral AI), and Whisper multilingual (OpenAI)
• Streaming decoders - Real-time LLM output display with proper UTF-8 handling (guide)
• 54 agent tokens - Built-in support for chat, CoT reasoning, ReAct agents, tool calling, RAG citations (docs)
• Battle-tested algorithms - Regexr with JIT (pure Rust), Aho-Corasick for special tokens, linked-list BPE, SentencePiece unigram, WordPiece for BERT-family models

Cross-platform:

• Python bindings via PyO3 (Linux, macOS, Windows)
• Native Rust library for maximum performance

Performance Deep Dive

All benchmarks performed on Linux (6.16.8-arch3-1) with 24 CPU cores, comparing against tiktoken (reference Python implementation), Hugging Face tokenizers, and TokenDagger.

Single Text Encoding

For single texts, splintr achieves 3-4x faster encoding across various text sizes:

Latency by content type:

Consistent low latency across Python code, JSON, English prose, and Chinese text makes splintr ideal for interactive applications and real-time processing.

Batch Encoding

The real magic happens with batches. Splintr parallelizes across texts to achieve 10-12x speedup:

Higher speedups on larger batches where parallelization overhead is amortized. Perfect for:

• Training data preprocessing
• Bulk document tokenization
• API batch processing
• Data pipeline throughput

Design Decision: Sequential by Default

Splintr uses sequential encoding for single texts and parallel encoding across batches based on empirical benchmarking:

Key findings:

• Sequential is faster for texts up to ~1MB (typical LLM prompts and documents)
• Rayon's parallelization overhead only pays off at ~1MB+ text sizes
• Most real-world inputs are well under 1MB
• encode() uses sequential processing for optimal single-text performance
• encode_batch() parallelizes across multiple texts for maximum throughput
• encode_rayon() available for the rare cases where you have >1MB single texts

This architecture ensures splintr is optimized for the most common tokenization patterns in LLM applications.

Running Benchmarks Yourself

# Clone and install
git clone https://github.com/ml-rust/splintr.git
cd splintr
pip install -e .
pip install tiktoken

# Run the benchmark suite
cd benchmarks
python benchmark.py --model cl100k_base --output results/my_benchmark.json

# View results
cat results/my_benchmark.md

The benchmark suite tests single text encoding, batch encoding, streaming decoder performance, and special token handling across various content types.

Regex Backends

Splintr uses a pure-Rust regex engine (regexr) by default, with optional PCRE2 support for compatibility.

Default Backend (regexr):

• Pure Rust implementation (no C dependencies)
• JIT compilation and SIMD acceleration
• Native UTF-8 and Unicode property support

Optional PCRE2 Backend:

from splintr import Tokenizer

# Default: regexr backend (pure Rust)
tokenizer = Tokenizer.from_pretrained("cl100k_base")

# Optional: switch to PCRE2 (requires --features pcre2)
tokenizer = Tokenizer.from_pretrained("cl100k_base").pcre2(True)

To enable PCRE2, build with the feature flag:

maturin develop --release --features pcre2

Benchmarking:

# Compare backends (requires PCRE2 feature)
python benchmarks/benchmark_regexr_comparison.py --model cl100k_base

# Visual comparison with charts
python benchmarks/benchmark_regexr_viz.py --model cl100k_base

Streaming Decoders

For real-time LLM applications where tokens arrive one at a time, Splintr provides streaming decoders that handle UTF-8 boundary alignment:

# Regular streaming decoder (cl100k_base, o200k_base, llama3)
decoder = tokenizer.streaming_decoder()

# ByteLevel streaming decoder (deepseek_v3, GPT-2)
decoder = tokenizer.byte_level_streaming_decoder()

# Process tokens as they arrive
for token_id in token_stream:
    if text := decoder.add_token(token_id):
        print(text, end="", flush=True)
print(decoder.flush())

Why streaming decoders? BPE tokens don't align with UTF-8 character boundaries. A multi-byte character like "世" might split across tokens. The streaming decoder buffers incomplete sequences and only outputs complete characters.

See the API Guide for detailed usage, examples, and best practices.

Supported Vocabularies

Vocabulary	Used By	Vocabulary Size	Special Tokens	Import Constant
cl100k_base	GPT-4, GPT-3.5-turbo	~100,000	5 + 54 agent	CL100K_BASE_PATTERN
o200k_base	GPT-4o	~200,000	2 + 54 agent	O200K_BASE_PATTERN
llama3	Llama 3, 3.1, 3.2, 3.3 (Meta)	~128,000	11 + 54 agent	LLAMA3_PATTERN
deepseek_v3	DeepSeek V3, DeepSeek R1	~128,000	17 + 54 agent	LLAMA3_PATTERN
mistral_v1	Mistral 7B v0.1/v0.2, Mixtral 8x7B	~32,000	3 + 54 agent	SENTENCEPIECE_PATTERN
mistral_v2	Mistral 7B v0.3, Codestral, 8x22B	~32,768	10 + 54 agent	SENTENCEPIECE_PATTERN
mistral_v3	Mistral NeMo, Large 2, Pixtral	~131,000	10 + 54 agent	MISTRAL_V3_PATTERN
whisper / whisper_v1 / whisper_v2 / whisper_v3	OpenAI Whisper multilingual (tiny..large-v3)	~51,000	1608 (no agent)	GPT2_PATTERN

Whisper is a speech model, so it carries no agent tokens — its special tokens are the standard Whisper set (<|startoftranscript|>, language tokens, <|transcribe|>/<|translate|>, 1501 timestamp tokens). Bare whisper resolves to v2. The English-only checkpoints (*.en) use a different base BPE and are not bundled; load those with from_json (below).

Loading any model from tokenizer.json

For models not bundled above, point splintr.from_json at a HuggingFace tokenizer.json. It dispatches on the tokenizer type and returns the matching object — Tokenizer (BPE), SentencePieceTokenizer (Unigram), or WordPieceTokenizer:

from splintr import from_json

tok = from_json("tokenizer.json")   # BERT, T5, Gemma, Qwen, Whisper.en, ...
ids = tok.encode("Hello, world!")                       # content tokens
ids = tok.encode_with_special_tokens("Hello, world!")   # + [CLS]/[SEP]/<s> etc. (post_processor)
text = tok.decode(ids)

encode returns content tokens (HF's add_special_tokens=False); encode_with_special_tokens additionally applies the model's post_processor template (HF's default encode). Honored end-to-end: the multi-stage pre-tokenizer pipeline (ByteLevel, Split incl. invert, Digits, Punctuation/Contiguous, Sequence, add_prefix_space/prepend_scheme), the full ordered normalizer (Replace, Strip, Prepend, NFC/NFD/NFKC/NFKD, Precompiled charsmap, …), BPE merge order, and added_tokens matching. Verified id-for-id (content and with special tokens) against GPT-2, RoBERTa, Qwen, Whisper, T5, Albert, XLNet, BERT, DistilBERT, Falcon, StarCoder2, DeepSeek-Coder, GPT-NeoX.

model.type	Returned type	Example models
BPE (byte-level)	Tokenizer	GPT-2, Whisper, Llama 3, Qwen, DeepSeek
Unigram	SentencePieceTokenizer	T5, Gemma, Albert, XLNet
WordPiece	WordPieceTokenizer	BERT, DistilBERT, Electra

The split regex, byte-level flag, merge order, normalizer (including SentencePiece's Precompiled charsmap), and special tokens are all read from the file itself. Output is verified id-for-id against HuggingFace tokenizers across every family — GPT-2, RoBERTa, BART, Qwen, Whisper (BPE); T5, Albert, XLNet (Unigram); BERT, DistilBERT (WordPiece). (Rust: splintr::from_json_path / from_json_bytes.)

Strict by design. Rather than silently approximate a config it doesn't fully support (which would emit wrong tokens with no signal), from_json raises — UnsupportedModelType, UnsupportedNormalizer, InvalidNormalizerRegex, or UnsupportedPreTokenizer (a declared pre-tokenizer with no recognized split, so the pattern is never guessed).

OpenAI standard tokens:

• cl100k_base: <|endoftext|>, <|fim_prefix|>, <|fim_middle|>, <|fim_suffix|>, <|endofprompt|>
• o200k_base: <|endoftext|>, <|endofprompt|>

Meta Llama 3 standard tokens:

• llama3: <|begin_of_text|>, <|end_of_text|>, <|start_header_id|>, <|end_header_id|>, <|eot_id|>, <|eom_id|> (3.1+), <|python_tag|> (3.1+), <|step_id|> (3.2-Vision), <|image|> (3.2-Vision)

DeepSeek V3 standard tokens:

• deepseek_v3: <｜begin▁of▁sentence｜>, <｜end▁of▁sentence｜>, <think>, </think>, <｜User｜>, <｜Assistant｜>, <|EOT|>, FIM tokens (<｜fim▁hole｜>, <｜fim▁begin｜>, <｜fim▁end｜>), tool calling tokens (<｜tool▁calls▁begin｜>, <｜tool▁call▁begin｜>, etc.)

Mistral standard tokens:

• mistral_v1: <unk>, <s>, </s> (SentencePiece native)
• mistral_v2: Same as V1 + control tokens: [INST], [/INST], [TOOL_CALLS], [AVAILABLE_TOOLS], [/AVAILABLE_TOOLS], [TOOL_RESULTS], [/TOOL_RESULTS]
• mistral_v3: <unk>, <s>, </s> + control tokens (Tekken/Tiktoken-based, NOT SentencePiece)

Agent Tokens (54 per model)

Splintr extends all vocabularies with 54 specialized tokens for building agent systems:

from splintr import Tokenizer, CL100K_AGENT_TOKENS

tokenizer = Tokenizer.from_pretrained("cl100k_base")
text = "<|think|>Let me reason...<|/think|>The answer is 42."
tokens = tokenizer.encode_with_special(text)
print(CL100K_AGENT_TOKENS.THINK)      # 100282
print(CL100K_AGENT_TOKENS.FUNCTION)   # 100292

Category	Example Tokens	Purpose
Conversation	system, user, assistant, im_start, im_end	ChatML format
Thinking	think	Chain-of-Thought reasoning
ReAct	plan, step, act, observe	Agent action loops
Tools	function, result, error	Function calling
RAG	context, quote, cite, source	Citations

See docs/special_tokens.md for the complete list and API Guide for usage examples.

How It Works

Splintr implements several optimizations that make tokenization faster:

• Regexr with JIT compilation: Pure Rust regex engine with SIMD acceleration
• Rayon parallelism: Leverages multiple CPU cores for batch encoding
• Linked-list BPE algorithm: Avoids O(N²) complexity on pathological inputs
• SentencePiece unigram: Viterbi maximum-score segmentation (true Unigram, not greedy) with byte fallback, for Mistral/Llama/T5-style models
• WordPiece tokenizer: BERT-compatible subword tokenization with ## continuation prefix, BasicTokenizer preprocessing (lowercase, accent stripping, punctuation splitting)
• FxHashMap: Faster lookups than default SipHash for non-adversarial contexts
• Aho-Corasick for special tokens: Fast multi-pattern matching without regex alternation
• LRU cache: Avoids redundant BPE encoding of frequently seen chunks

Use Cases

LLM Applications:

• Tokenizing prompts with 3-4x lower latency
• Streaming decoder for real-time output display
• Token counting for API cost estimation

Agent Systems:

• Building ReAct agents with structured reasoning tokens
• Tool-calling systems with function tokens
• Chain-of-Thought reasoning with thinking tokens

Training Pipelines:

• Fast batch encoding of large datasets (10-12x speedup)
• Preprocessing millions of documents efficiently
• Parallel tokenization across distributed systems

RAG Applications:

• Structured context injection with citation tokens
• Document chunking with section markers
• Source tracking through tokenization

Data Processing:

• Bulk document tokenization
• Multi-language text processing
• Real-time text preprocessing

Contributing

Contributions are welcome! Here's how you can help:

1. Report bugs: Open an issue with a minimal reproduction case
2. Suggest features: Describe your use case and why the feature would be helpful
3. Submit pull requests:
   • Add tests for new functionality
   • Run cargo test and cargo clippy before submitting
   • Update documentation as needed

Development Setup

# Clone the repository
git clone https://github.com/ml-rust/splintr.git
cd splintr

# Install pre-commit hook (recommended)
cp hooks/pre-commit .git/hooks/pre-commit
chmod +x .git/hooks/pre-commit

# Build the Rust library
cargo build --release

# Build Python bindings
pip install maturin
maturin develop --release

# Run tests
cargo test                    # Rust tests
cargo clippy --all-targets    # Linting
cargo fmt --all --check       # Format check

The pre-commit hook automatically runs formatting, clippy, and tests before each commit.

Acknowledgments

Splintr builds upon concepts from:

• tiktoken - OpenAI's reference BPE tokenizer
• SentencePiece - Google's unsupervised text tokenizer
• tokenizers - Hugging Face's tokenization library

The performance optimizations are informed by profiling real-world usage patterns in LLM applications.

Citation

If you use Splintr in your research, please cite:

@software{splintr,
  author = {Farhan Syah},
  title = {Splintr: High-Performance Tokenizer (BPE + SentencePiece + WordPiece)},
  year = {2025},
  url = {https://github.com/ml-rust/splintr}
}