DeepLatent SARF Tokenizer

Part of Suhail Project - Independent Research by Mohammed Almaghrabi

This is the SARF (Sarf-Aware Representation Framework) tokenizer designed for the DeepLatent language model, trained on bilingual Arabic/English data.

What is SARF?

SARF (صَرْف) is the Arabic term for morphology. In classical and modern Arabic linguistics, ṣarf refers to the system that governs:

• Word formation
• Roots and patterns (جذر / وزن)
• Prefixes, suffixes, infixes
• Tense, gender, number, and derivation

Ṣarf is the exact linguistic layer that makes Arabic hard for naive tokenizers.

SARF combines morphological analysis with BPE tokenization to achieve better compression, especially for morphologically rich languages like Arabic.

Most tokenizers treat Arabic as bytes or characters. SARF treats Arabic as a language.

Features

• Arabic-Optimized: Designed specifically for Arabic and morphologically-rich languages
• Fast: Rust core with Python bindings (up to 43,000+ texts/sec with parallel processing)
• Accurate: 100% roundtrip accuracy on 1,000,000 test samples
• Edge Case Handling: Proper handling of diacritics (tashkeel), prefixes, suffixes, and special characters
• Unicode Support: Full support for Arabic diacritics, and mixed scripts
• Parallel Processing: Excellent thread scaling (5x+ speedup with 8 threads)

Installation

pip install deeplatent-nlp

Quick Start

from deeplatent import SARFTokenizer

# Load tokenizer
tok = SARFTokenizer.from_pretrained("SARFTokenizer")

# Encode text
ids = tok.encode("مرحبا بالعالم")
print(ids)

# Decode back
text = tok.decode(ids)
print(text)

Edge Cases Handled

Performance

Tokenizer Benchmark Results

Comparison with state-of-the-art tokenizers on 60,000 samples (30k Arabic + 30k English).

Dataset: almaghrabima/deeplatent-benchmark-data

Metrics explained:

• Fertility: Average tokens per word (lower is better - more efficient encoding)
• C/T: Characters per token (higher is better - more characters encoded per token)
• Parity: AR chars/token ÷ EN chars/token (1.0 = equal treatment of both languages)

Key findings:

• SARFTokenizer achieves best Arabic fertility (1.72 tokens/word) - 35% better than GPT-4o
• Lowest average fertility (1.64) among all tokenizers tested
• Best Arabic characters/token (3.45) - encodes more Arabic per token than any competitor
• Compact vocabulary (64k) while maintaining top performance
• ALLaM-7B shows similar efficiency (both use morpheme-aware approaches)
• Falcon-H1-7B has best parity (0.926) but 28% higher fertility than SARF
• GPT-4 and Mistral struggle with Arabic (4.6-5.6 tokens/word vs 1.7 for SARF)

Throughput Benchmark (1M samples, 680 MB)

Comparison with tiktoken on 1,000,000 documents:

Key findings:

• SARFTokenizer outperforms tiktoken at 8 threads (13.72 MB/s vs 8.47-10.60 MB/s)
• Excellent parallel scaling: 4.4x speedup from 1 to 8 threads
• tiktoken degrades with more threads (peaks at 4T, drops at 8T)

Million-Scale Roundtrip Accuracy

Tested on 999,999 samples from real-world data:

Edge Case Tests (58/58 Passed)

All 12 edge case categories pass with 100% success:

Reproduce Benchmark Results

Datasets:

• Benchmark data (60k samples): almaghrabima/deeplatent-benchmark-data
• Eval test data: almaghrabima/eval-test-data

# Install dependencies
pip install deeplatent-nlp pyarrow tiktoken transformers huggingface-hub

# Run parity benchmark (vs GPT-4o, Gemma, etc.)
python benchmark_pypi.py

# Run throughput benchmark (vs tiktoken)
python benchmark_tiktoken_style.py --samples 1000000 --threads 1 2 4 8

# Run comprehensive tests (roundtrip + edge cases)
python test_comprehensive_million.py --samples 1000000 --report

Requirements

• Python 3.9+
• Rust 1.70+ (for building from source)

License

CC-BY-NC-4.0

Citation

@misc{sarf-tokenizer-2026,
  title={SARF: A Morpheme-Aware Tokenization Framework for Arabic-English - Suhail Project},
  author={Almaghrabi, Mohammed},
  year={2026},
  url={https://huggingface.co/almaghrabima/SARFTokenizer},
  note={Independent research, part of Suhail Project}
}