omnitoken 0.2.0

Universal, research-grounded tokenizer engine for BPE, WordPiece, and Unigram vocabularies — targeting consumer hardware (AMD Ryzen 5 7600, DDR5-5600, Gen3 NVMe).

Table of Contents

• Overview
• Key Features
• Comparative Benchmark Matrix
• Empirical Benchmark Results
  • 1. L3-Resident Regime (4.0 MiB Input)
  • 2. DRAM-Resident Regime (64.0 MiB Input)
• Roofline Sanity Matrix
• Command Line Interface & Flags
• System Architecture
  • Workspace Directory Layout
  • Dependency Architecture
• AI Use Disclosure & Credit Attribution
• Quick Start & Usage
• Verification & Tests
• Citation & Licensing

Overview

OmniToken is a high-performance, universal tokenization engine written in Rust. It ingests every major tokenizer vocabulary format into one universal intermediate representation (VocabIr) and encodes with a unified automaton that executes BPE, WordPiece, and Unigram in the same trie-walk loop.

• Primary Competitor: gigatoken — BPE engine benchmarked on a 144-core server.
• Our Wedge: Universal format support (BPE, WordPiece, Unigram, tiktoken, SentencePiece binary .model, GGUF) + inference-time low latency + SWAR/AVX2 pretokenization + hybrid hot-tier cache grounded in PtrHash literature.

Key Features

• ⚡ 1.77+ GB/s Multi-Core Throughput: Scaled across 6 physical cores / 12 SMT threads on consumer DDR5 hardware.
• 🎯 Universal Vocab IR (vocab-ir): Ingest HuggingFace tokenizers.json (BPE/WordPiece/Unigram), tiktoken .tiktoken files, SentencePiece binary .model protobuf blobs, and GGUF metadata.
• 🔄 Unified Automaton (walker): One trie walker handles BPE priority queues ($O(N \log M)$ per Zouhar et al.), WordPiece LinMaxMatch ($O(N)$ per Song et al.), and Unigram Viterbi DP.
• 🚀 SWAR / SIMD Pretokenizer (pretokenizer): 256-entry byte-class table with 8-byte u64 branchless SWAR dispatch and AVX2 vectorization (x86-64-v3).
• 💎 Hybrid Hot-Tier Cache (hot-cache): Lock-free RCU double-buffered static tier, automatic background rebuild thread, XXH3-64 fingerprint verification, 64-byte padded CountMinSketch, and SwissTable fallback.
• 📊 Roofline-Grounded Benchmarking (bench-harness): Automated L3-resident vs DRAM-resident throughput validation against physical hardware bandwidth limits.

Comparative Benchmark Matrix

Hardware Environment: AMD Ryzen 5 7600 (6 Cores / 12 SMT Threads @ 5.1 GHz), Dual-Channel DDR5-5600, Ubuntu Linux 24.04 LTS (WSL2).
Corpus Test Input: GPT-2 standard vocabulary (16.0 MiB text buffer).

Single-Thread & Multi-Thread Throughput Comparison (16.0 MiB Corpus)
========================================================================================
HuggingFace tokenizers (Py) [█░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░]   0.002 GB/s (  2 MiB/s)
tiktoken (Py / Rust Core)   [███░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░]   0.017 GB/s ( 17 MiB/s)
gigatoken (EPYC Server Ref) [████████████████████████░░░░░░░░░░░░░░░░]   0.830 GB/s (830 MiB/s)
OmniToken (1 Thread)        [████████░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░]   0.279 GB/s (266 MiB/s)
OmniToken (6 Cores / 12 T)  [████████████████████████████████████████]   1.771 GB/s (1689 MiB/s)
========================================================================================

Empirical Benchmark Results

1. L3-Resident Regime (4.0 MiB Input ≤ 32MB L3 Cache)

2. DRAM-Resident Regime (64.0 MiB Input > 32MB L3 Cache)

Roofline Sanity Matrix

Command Line Interface & Flags

omnitoken encode --vocab <path> [OPTIONS]

bench --vocab <path> [OPTIONS]

System Architecture

Workspace Directory Layout

crates/
├── vocab-ir/        # Universal IR loader (HF tokenizers.json, tiktoken, SPM proto, GGUF)
├── trie-builder/    # Offline Aho-Corasick trie + failure links & continuation prefixes
├── pretokenizer/    # SWAR / AVX2 256-entry byte classifier & UTF-8 chunk splitter
├── walker/          # Unified automaton: O(N log M) BPE, MaxMatch WordPiece, Viterbi Unigram
├── hot-cache/       # Hybrid MPHF static tier + CountMin sketch + SwissTable overflow + RCU thread
├── omnitoken/       # Unified CLI binary + PyO3 Python bindings + mimalloc allocator
└── bench-harness/   # Reproducible roofline-checked benchmark harness

Dependency Architecture

vocab-ir ────────► trie-builder ──────► walker ◄───── pretokenizer
                     │                     ▲
                     └─────────────────────┼───────── hot-cache
                                           │
                                     omnitoken / bench-harness

AI Use Disclosure & Credit Attribution

Quick Start & Usage

1. Download Standard Vocab

pip install tokenizers
python3 -c "from tokenizers import Tokenizer; Tokenizer.from_pretrained('gpt2').save('gpt2.json')"

2. Build Release Binaries

cargo build --release

3. Run OmniToken CLI

echo "the quick brown fox jumps over the lazy dog" | ./target/release/omnitoken encode --vocab gpt2.json

4. Run Benchmark Harness

# Single-thread baseline benchmark
./target/release/bench --vocab gpt2.json --threads 1 --bytes 16777216

# Multi-thread 6-core scaling benchmark
./target/release/bench --vocab gpt2.json --threads 6 --bytes 67108864

# Parity verification mode
./target/release/bench --vocab gpt2.json --parity

Verification & Tests

Run all unit and integration tests across workspace crates:

cargo test --workspace

Citation & Licensing

Cite this repository if used in tokenization performance research:

@software{omnitoken2026,
  author  = {Hegde, Sunay},
  title   = {{OmniToken}: Universal High-Performance Tokenizer Engine for Consumer Hardware},
  year    = {2026},
  url     = {https://github.com/SunayHegde2006/OmniToken}
}

Licensed under the MIT License.