deeplatent-nlp 0.2.4

DeepLatent - Morphology-aware tokenizer for Arabic/English bilingual text with native core

DeepLatent

DeepLatent - SARF Tokenizer for Arabic/English bilingual text with native Rust core.

This package provides the SARF (Sarf-Aware Representation Framework) tokenizer that achieves excellent Arabic/English parity (1.09) by applying morpheme-level preprocessing before BPE tokenization.

Installation

pip install deeplatent-nlp

Building from Source

If installing from source, you'll need Rust installed:

# Install Rust (if not already installed)
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh

# Install from source
pip install .

Quick Start

from deeplatent import SARFTokenizer

# Load tokenizer from HuggingFace
tokenizer = SARFTokenizer.from_pretrained("almaghrabima/deeplatent-tokenizer")

# Encode text (SARF preprocessing is applied automatically for Arabic)
arabic_text = "مرحبا بكم في هذا الاختبار"
tokens = tokenizer.encode(arabic_text)
print(f"Token count: {len(tokens)}")

# Decode back to text
decoded = tokenizer.decode(tokens)
print(f"Decoded: {decoded}")

# Works with English too
english_text = "Hello world, this is a test"
tokens = tokenizer.encode(english_text)
print(f"English token count: {len(tokens)}")

Performance

Metric	With SARF Preprocessing	Without Preprocessing
Arabic Fertility	2.29	5.65
English Fertility	2.10	2.91
Parity (Ar/En)	1.09	1.94
Interpretation	EXCELLENT	Moderate

Fertility = average tokens per word. Lower is better. Parity closer to 1.0 means more equal treatment between languages.

Supported Platforms

Pre-built wheels are available for:

• Linux (manylinux2014, x86_64)
• macOS (x86_64, arm64)
• Windows (x86_64)

For other platforms, the package will build from source (requires Rust).

What is SARF?

SARF (صَرْف) is the Arabic term for morphology. In Arabic linguistics, ṣarf refers to the system that governs:

• Word formation
• Roots and patterns (جذر / وزن)
• Prefixes, suffixes, infixes
• Tense, gender, number, and derivation

Most tokenizers treat Arabic as bytes or characters. SARF treats Arabic as a language.

API Reference

SARFTokenizer

from deeplatent import SARFTokenizer

# Load from HuggingFace
tokenizer = SARFTokenizer.from_pretrained("almaghrabima/deeplatent-tokenizer")

# Load from local directory
tokenizer = SARFTokenizer.from_directory("./my_tokenizer")

# Disable preprocessing (not recommended for Arabic)
tokenizer = SARFTokenizer.from_pretrained(
    "almaghrabima/deeplatent-tokenizer",
    use_preprocessing=False
)

Encoding

# Simple encoding
tokens = tokenizer.encode("مرحبا بكم")

# With options
result = tokenizer.encode(
    "مرحبا بكم",
    add_special_tokens=True,
    padding=True,
    truncation=True,
    max_length=512,
    return_tensors="pt"  # or "tf" for TensorFlow
)

# Batch encoding
texts = ["مرحبا", "Hello", "مرحبا بكم في العالم"]
batch_tokens = tokenizer.encode_batch(texts)

Decoding

# Simple decoding
text = tokenizer.decode([1234, 5678, 9012])

# Batch decoding
texts = tokenizer.decode_batch([[1234, 5678], [9012, 3456]])

# Keep special tokens
text = tokenizer.decode(tokens, skip_special_tokens=False)

License

This tokenizer is released under CC-BY-NC-4.0 (Creative Commons Attribution-NonCommercial 4.0 International).

For commercial licensing, please contact: [e-mail]

Author

• Mohammed Almaghrabi
• Email: [e-mail]

Links

• HuggingFace Model
• Evaluation Dataset