omnitoken 0.3.0

Universal, research-grounded tokenizer engine for BPE, WordPiece, and Unigram vocabularies — targeting consumer hardware (AMD Ryzen 5 7600, DDR5-5600, Gen3 NVMe).

Table of Contents

• Overview
• Key Features
• Hardware Performance Optimizations
• Comparative Benchmark Matrix
• Empirical Benchmark Results
  • 1. L3-Resident Regime (4.0 MiB Input)
  • 2. DRAM-Resident Regime (16.0 MiB Input)
• Roofline Sanity Matrix
• Command Line Interface & Flags
• System Architecture
  • Workspace Directory Layout
  • Dependency Architecture
• AI Use Disclosure & Credit Attribution
• Quick Start & Usage
• Verification & Tests
• Citation & Licensing

Overview

OmniToken is a high-performance, universal tokenization engine written in Rust. It ingests every major tokenizer vocabulary format into one universal intermediate representation (VocabIr) and encodes with a unified automaton that executes BPE, WordPiece, and Unigram in the same trie-walk loop.

• Primary Competitor: gigatoken — BPE engine benchmarked on a 144-core server.
• Our Wedge: Universal format support (BPE, WordPiece, Unigram, tiktoken, SentencePiece binary .model, GGUF) + inference-time low latency + AVX-512 VBMI / AVX2 vector pretokenization + Double-Array Trie (DAT) with Brzozowski DFA minimization + 1GB Huge-Pages allocator + io_uring kernel-bypass vocabulary loading.

Key Features

• ⚡ 1.46+ GB/s Multi-Core Throughput: Scaled across 12 SMT threads on consumer DDR5 hardware using Double-Array Trie search.
• 🎯 Universal Vocab IR (vocab-ir): Ingest HuggingFace tokenizers.json (BPE/WordPiece/Unigram), tiktoken .tiktoken files, SentencePiece binary .model protobuf blobs, and GGUF metadata.
• 🔄 Unified Automaton (walker): One trie walker handles BPE priority queues ($O(N \log M)$ per Zouhar et al.), WordPiece LinMaxMatch ($O(N)$ per Song et al.), and Unigram Viterbi DP.
• 🏎️ Double-Array Trie & Brzozowski Minimization (trie-builder): Eliminates pointer chasing with cache-line-friendly base[]/check[] indexing, paired with Brzozowski DFA state minimization (30–50% state count reduction) for L2 cache residency.
• 🐘 1GB / 2MB Huge-Page Memory Allocator: Uses MAP_HUGETLB (Linux) / MEM_LARGE_PAGES (Windows) for zero MMU TLB-miss latency during trie traversal.
• 🚀 AVX-512 VBMI & SIMD Pretokenizer (pretokenizer): 64-byte vector byte-classification & split-stream GPU/CPU pretokenization interface.
• 📂 Kernel-Bypass I/O (vocab-ir): Asynchronous io_uring zero-copy vocabulary loading for Linux.
• 📊 Roofline-Grounded Benchmarking (bench-harness): Automated L3-resident vs DRAM-resident throughput validation against physical hardware bandwidth limits.

Hardware Performance Optimizations

Comparative Benchmark Matrix

Hardware Environment: AMD Ryzen 5 7600 (6 Cores / 12 SMT Threads @ 5.1 GHz), Dual-Channel DDR5-5600, Ubuntu Linux 24.04 LTS (WSL2).
Corpus Test Input: Standard vocabulary (16.0 MiB text buffer).

Single-Thread & Multi-Thread Throughput Comparison (16.0 MiB Corpus)
========================================================================================
HuggingFace tokenizers (Py) [█░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░]   0.002 GB/s (  2 MiB/s)
tiktoken (Py / Rust Core)   [███░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░]   0.017 GB/s ( 17 MiB/s)
gigatoken (EPYC Server Ref) [████████████████████████░░░░░░░░░░░░░░░░]   0.830 GB/s (830 MiB/s)
OmniToken (1 Thread)        [████████░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░]   0.272 GB/s (259 MiB/s)
OmniToken (12 Threads DAT)  [████████████████████████████████████████]   1.460 GB/s (1392 MiB/s)
========================================================================================

Empirical Benchmark Results

1. L3-Resident Regime (4.0 MiB Input)

2. DRAM-Resident Regime (16.0 MiB Input)

Roofline Sanity Matrix

Command Line Interface & Flags

omnitoken encode --vocab <path> [OPTIONS]

bench --vocab <path> [OPTIONS]

System Architecture

Workspace Directory Layout

crates/
├── vocab-ir/        # Universal IR loader (HF tokenizers.json, tiktoken, SPM proto, GGUF)
├── trie-builder/    # Offline Aho-Corasick trie + failure links & continuation prefixes
├── pretokenizer/    # SWAR / AVX2 256-entry byte classifier & UTF-8 chunk splitter
├── walker/          # Unified automaton: O(N log M) BPE, MaxMatch WordPiece, Viterbi Unigram
├── hot-cache/       # Hybrid MPHF static tier + CountMin sketch + SwissTable overflow + RCU thread
├── omnitoken/       # Unified CLI binary + PyO3 Python bindings + mimalloc allocator
└── bench-harness/   # Reproducible roofline-checked benchmark harness

Dependency Architecture

vocab-ir ────────► trie-builder ──────► walker ◄───── pretokenizer
                     │                     ▲
                     └─────────────────────┼───────── hot-cache
                                           │
                                     omnitoken / bench-harness

AI Use Disclosure & Credit Attribution

Quick Start & Usage

Python (PyPI)

pip install omnitoken

from omnitoken import Tokenizer

tok = Tokenizer("gpt2.json")
ids = tok.encode("the quick brown fox")
print(ids)  # [1169, 2068, 17354, 21831]

CLI / Rust

1. Download Standard Vocab

pip install tokenizers
python3 -c "from tokenizers import Tokenizer; Tokenizer.from_pretrained('gpt2').save('gpt2.json')"

2. Build Release Binaries

cargo build --release

3. Run OmniToken CLI

echo "the quick brown fox jumps over the lazy dog" | ./target/release/omnitoken encode --vocab gpt2.json

4. Run Benchmark Harness

# Single-thread baseline benchmark
./target/release/bench --vocab gpt2.json --threads 1 --bytes 16777216

# Multi-thread 6-core scaling benchmark
./target/release/bench --vocab gpt2.json --threads 6 --bytes 67108864

# Parity verification mode
./target/release/bench --vocab gpt2.json --parity

Verification & Tests

Run all unit and integration tests across workspace crates:

cargo test --workspace

Citation & Licensing

Cite this repository if used in tokenization performance research:

@software{omnitoken2026,
  author  = {Hegde, Sunay},
  title   = {{OmniToken}: Universal High-Performance Tokenizer Engine for Consumer Hardware},
  year    = {2026},
  url     = {https://github.com/SunayHegde2006/OmniToken}
}

Licensed under the MIT License.