xerv-crayon 4.0.4

The Omni-Backend Tokenizer (CPU/CUDA/ROCm)

🖍️ XERV Crayon v4.0

The Omni-Backend Tokenizer for Specialized AI

Why force a single bloated vocabulary on every problem?
Crayon is a next-generation tokenizer designed for specialization. Hot-swap vocabulary profiles ("Cartridges") optimized for your domain—Quantum Physics, Rust Programming, Financial Law, or anything in between.

---

🚀 Key Features

Feature	Description
💾 Cartridge System	Instantly hot-swap specialized vocabularies (science, code, multilingual)
🚀 Omni-Backend	Auto-detects & runs on CPU (AVX2), NVIDIA (CUDA), or AMD (ROCm)
⚡ Native GPU Kernels	"Bare Metal" C++/HIP kernels (no wrappers) for >100M tokens/sec
🗺️ Zero-Copy Mapping	DAT files loaded via mmap for instant startup & minimal RAM
🌊 Zero-Disk Streaming	Build profiles directly from Hugging Face—no multi-GB downloads
🛡️ Offline Resilience	Seamless local bootstrap fallback. Works offline out-of-the-box

---

📊 Benchmarks — The Numbers Speak

100% HONEST. NO SUGARCOATING. DATA-DRIVEN.

Run python benchmark_competitive.py to reproduce these results yourself.

⚡ Speed Comparison (Omni-Backend)

Tokenizer	Tokens/sec	vs CRAYON
🖍️ CRAYON (CPU - AVX2)	21,863,777	baseline
🖍️ CRAYON (CUDA - A100)	140,000,000+	6.4x faster
tiktoken (GPT-4)	524,469	41x slower
HF LLaMA (SP-BPE)	281,558	77x slower
HF GPT-2 (BPE)	237,117	92x slower
HF BERT (WordPiece)	202,269	108x slower

📈 CPU Optimization Verification

Measured on Intel Core i3-7020U (Low-Power Laptop CPU)

Metric	Result
✅ AVX2 Status	Active (Simd-Ops v4)
✅ Load Time	0.54ms (Instant hot-swap)
✅ Throughput	21.1M tokens/sec (!?!)

---

⚡ Quick Start: The "Omni-Backend"

Run on any hardware with a single line of code. Crayon automatically detects AVX2, CUDA, or ROCm presence.

1. Hardware-Aware Initialization

from crayon.core.vocabulary import CrayonVocab

# 🔵 CPU (Intel/AMD) - AVX2/AVX-512 Native
vocab = CrayonVocab(device="cpu")

# 🟢 NVIDIA GPUs (All Tensor Core Architectures)
vocab = CrayonVocab(device="cuda")

# 🔴 AMD GPUs (Instinct/Radeon HIP/ROCm)
vocab = CrayonVocab(device="rocm")

2. The "Context Manager" Hot-Swap

Instantly switch between specialized vocabularies within the same script without reloading the model.

vocab = CrayonVocab(device="cpu")
vocab.load_profile("lite")

# ... standard tokenization ...

# ⚡ TEMPORARY SWITCH to 'code' profile for a function block
with vocab.using_profile("code"):
    tokens = vocab.tokenize("def fast_inverse_sqrt(x):")
    # Uses the compact Code vocabulary here
    
# 🔥 AUTOMATICALLY REVERT to 'lite' here

3. Basic Example

import json
import mmap
from crayon.c_ext.dat_builder import DATBuilder
from crayon.c_ext import crayon_cpu # Auto-renamed from crayon_fast

# Load any trained vocabulary
with open("trained_vocab_code.json", "r") as f:
    vocab_list = json.load(f)

# Compile to DAT (one-time, few seconds)
builder = DATBuilder()
builder.build(vocab_list)
builder.save("vocab_code.dat")

# Load into C++ engine via memory mapping (instant, <1ms)
with open("vocab_code.dat", "rb") as f:
    mm = mmap.mmap(f.fileno(), 0, access=mmap.ACCESS_READ)
    crayon_cpu.load_dat(mm)

# Ultra-fast tokenization 🚀
code = 'fn main() { println!("Hello, World!"); }'
tokens = crayon_cpu.tokenize(code)
print(f"Tokens: {tokens}")

---

📦 Installation

git clone https://github.com/Xerv-AI/crayon.git
cd crayon
pip install -e .

Build the Extensions

PowerShell (Windows):

python setup.py build_ext --inplace

Bash (Linux/Mac):

python setup.py build_ext --inplace

Note: The setup script auto-detects nvcc and hipcc. If found, GPU backends are built automatically.

---

🏎️ Omni-Backend Architecture (v4.0)

Crayon now uses a "God Tier" multi-backend implementation combining:

┌─────────────┐      ┌──────────────┐      ┌─────────────┐      ┌──────────────┐
│ vocab.json  │ ──▶  │ DATBuilder   │ ──▶  │  vocab.dat  │ ──▶  │ Omni-Engine  │
│   (List)    │      │  (Python)    │      │  (Binary)   │      │ CPU/CUDA/HIP │
└─────────────┘      └──────────────┘      └─────────────┘      └──────────────┘

Component	File	Accelerators
CPU Backend	c_ext/cpu_engine.cpp	AVX-512 / AVX2 (Intel/AMD)
CUDA Backend	c_ext/gpu_engine_cuda.cu	Tensor Cores (NVIDIA Tesla/Ampere)
ROCm Backend	c_ext/rocm_engine.cpp	CDNA2 / RDNA3 (AMD Instinct/Radeon)
Zero-Copy Loader	mmap + buffer protocol	Instant startup (0.5ms)

---

🧩 Available Cartridges

5 production-ready profiles defined in src/crayon/core/profiles.py:

Profile	Size	Optimized For	Sources
lite	50k	Speed & Mobile	WikiText, RainDrop
science	250k	Reasoning (LaTeX, Quantum, Grad Math)	GRAD, Physics-700
code	250k	Syntax (Python, Rust, C++, JS)	CodeParrot, The Stack
multilingual	250k	Global (EU langs, Chinese, Hindi)	OSCAR, Wikipedia
arts_commerce	250k	Business (Legal, Finance, Lit)	PG19, Fin Phrasebank

vocab = CrayonVocab.load_profile("science")
vocab = CrayonVocab.load_profile("multilingual")

---

☁️ Verify on Google Colab

Want to test the CUDA Backend for free?

1. Open the notebook.
2. Change Runtime type to T4 GPU.
3. Run the cells to verify crayon_cuda compiles and smashes tokens at >100M/sec.

---

🧪 Testing & Verification

# Full verification (Benchmarks + Tests)
python verify_dat_engine.py

# Benchmark all backends
python benchmark_competitive.py

============================================================
XERV CRAYON V2.0 - HYPER-PRODUCTION DAT ENGINE VERIFICATION
============================================================
Vocabulary Size: 50,000 tokens
DAT Nodes: 163,000+
Throughput: 14,255,305 tokens/sec
STATUS: ✅ HYPER-PRODUCTION READY

---

📜 Citation

@techreport{xerv2026crayon,
  title={XERV Crayon: A First-Principles Analysis of Production-Grade Tokenization},
  author={Pal, Soham and Xerv Research},
  year={2026},
  institution={Xerv Research Engineering Division}
}

---

📄 License

Copyright (c) 2025-2026 Xerv Research. Released under the MIT License.

---

Built with 💙 by Xerv Research Engineering Division

_{⭐ Star this repo if Crayon helps your project!}