مِسار — Misar
Adaptive $O(n)$ Context-Aware Arabic NLP Preprocessing Router for Transformer Language Models
الموجّه اللغوي التكيفي لتعزيز كفاءة ودقة معالجة اللغة العربية في نماذج الذكاء الاصطناعي
Architecture • Benchmarks • Installation • Quick Start • CLI & Workbench • Citation • النسخة العربية
Why Misar?
Modern Arabic NLP deployments face a structural bottleneck: forcing heterogeneous Arabic text (Modern Standard Arabic, classical diacritized prose, Gulf/Egyptian/Levantine/Maghrebi dialects, and noisy social media) through a single static preprocessing pipeline actively degrades both accuracy and inference economics:
- Legacy Normalization Damages Modern Transformers: Classical regex pipelines blindly collapse
ة → هandأ/إ/آ → ا. However, modern Transformer vocabularies such as AraBERTv2 (64kWordPiece) and CAMeLBERT explicitly preserve Hamzas and Taa Marbuta—meaning legacy normalization shatters known whole words (e.g.,الحوسبةas1token splits intoالحو+##سبهas2tokens). - Dialectal Sentiment & Negation Erasure: Over-aggressive cleaning distorts critical colloquial negation and sentiment markers (
مو,مب,مافيه,ماكو), reversing downstream classification polarity. - Diacritic, Tatweel & Elongation Token Explosion: Unstripped harakat (
تشكيل), kashida (ـ), and character stretches (بطييييء,السيرفررررر) trigger OOV[UNK]blindness or inflate subword token counts—multiplying Transformer $O(L^2)$ self-attention cost and API billing.
Misar (مِسار) solves all three challenges without retraining any model (Zero Parameter Update) by acting as an ultra-lightweight, zero-dependency, linear-time $O(n)$ routing gateway directly in front of your Transformer or LLM pipeline.
Architecture
Every incoming text passes once through LinguisticProfiler (< 0.25 ms on CPU) to compute five quantitative signals (dialect_ratio $P_{\text{dial}}$, diacritic_density $D_{\text{diac}}$, elongation_noise_score $E_{\text{noise}}$, character Shannon Entropy $H_{\text{char}}$, and code_switching_ratio $R_{\text{latin}}$), then dispatches dynamically to the optimal micro-pipeline:
flowchart LR
A["Raw Arabic Input<br/>(MSA / Diacritized / Dialect / Social)"] --> B["LinguisticProfiler O(n)<br/>P_dial · D_diac · E_noise · H_char · R_latin"]
B -->|"High Noise / Tatweel / Stretches ≥ 3"| C["NOISY_SOCIAL (+ Hybrid Dialect)<br/>Gemination-Safe Collapse + Tag Unpacking"]
B -->|"Dialect Markers / Protected Negations"| D["DIALECT_PRESERVE<br/>Negation Protection + Vocab-Aligned Cleanup"]
B -->|"Clean MSA / Classical Diacritized"| E["FAST_MSA<br/>Strip Harakat · Preserve أ/إ/آ, ة, ى"]
C --> F["AraBERTv2 / CAMeLBERT / ALLaM / LLMs<br/>(-46.57% Tokens · 0 [UNK])"]
D --> F
E --> F
Empirical Benchmarks
Measured reproducibly via python benchmarks/run_benchmark.py against the authentic 64,000-token aubmindlab/bert-base-arabertv02 WordPiece vocabulary across 15 curated multi-domain samples (latest_benchmark_report.json):
| Domain Category | Samples | Raw Tokens | Legacy Static (ة→ه, أ→ا) |
Misar Adaptive | Misar Gain vs. Raw |
|---|---|---|---|---|---|
| MSA Engineering & Code-Switch | 3 | 57 | 79 (+38.6% fragmentation) | 57 | Preserves 100% orthography (-27.85% vs Legacy) |
| Classical Diacritized (Heritage) | 3 | 186 (147 OOV [UNK]) |
48 | 39 | -79.03% tokens & 100% word recovery (0 [UNK]) |
| Saudi / Gulf / Levantine / Maghrebi Dev | 6 | 121 | 131 | 117 | -3.31% tokens (-10.69% vs Legacy) + negations intact |
| Noisy Tech Social & Code-Switching | 3 | 102 | 80 | 36 | -64.71% tokens + gemination preserved |
| Full Corpus Summary | 15 | 466 | 338 | 249 | -46.57% vs Raw · -26.33% vs Legacy · 0.223 ms/text |
Key Scientific Proof: On Standard Arabic technical prose (
MSA Tech / Engineering), blind legacy normalization inflated token count from 57 → 79 due toة → هsubword splitting, while Misar maintained 57 intact vocabulary tokens (TFR ≈ 1.0). Across all 15 multi-domain inputs, Misar saved 46.57% of total tokens vs. raw text and 26.33% vs. legacy normalization (recovering 156 → 0[UNK]tokens) with 0.223 ms mean latency on pure CPU.
Installation
Core misar requires zero mandatory external packages (Python 3.9+ Standard Library only):
# Install directly from GitHub
pip install git+https://github.com/RealMtrx/misar.git
# Or from PyPI once published
pip install misar
Optional extras for live Hugging Face Transformers wrapping and package development:
pip install "misar[huggingface]" # transformers + torch
pip install "misar[dev]" # pytest + build + twine
(Note: On first use of the built-in WordPieceTokenizer, Misar caches the official 64k vocabulary file locally under ~/.cache/misar/. Pass allow_vocab_download=False or --offline to run completely air-gapped using HeuristicSubwordTokenizer.)
Quick Start
1. Dynamic Text Routing & Token Telemetry
from misar import MisarRouter
router = MisarRouter() # Uses real AraBERTv2 64k WordPiece vocabulary
result = router.route("السيرفررررر بطييييء مررررة ومافيه استجابة في API أبدددداً 😡😡👎 #هندسة_البرمجيات")
print("Route: ", result.route_name) # NOISY_SOCIAL
print("Processed: ", result.processed_text) # السيرفر بطيء مرة ومافيه استجابة في API أبدا هندسة البرمجيات
print("Tokens: ", result.raw_token_count, "->", result.processed_token_count) # 28 -> 14
print("Reduction: ", f"-{result.token_reduction_pct:.1f}%") # -50.0%
print("Latency: ", f"{result.latency_ms:.3f} ms")
2. Zero-Latency Drop-In String Preprocessing & Batching
from misar import MisarRouter, preprocess
# One-line module helper
clean = preprocess("الســــلام علــــيكم ومافيه مشكلة #تطوير_البرمجيات")
# -> "السلام عليكم ومافيه مشكلة تطوير البرمجيات"
# Fast batch routing for production ETL pipelines
router = MisarRouter()
results = router.route_batch([
"أصدرت عدة دول عربية بياناً مشتركاً لأول مرة عن صافي الأرباح", # -> FAST_MSA (zero dialect false positives)
"والللله الكود مو معقد أبداً والأخطاء متكررررة", # -> NOISY_SOCIAL (preserves والله and مو)
])
3. One-Line Hugging Face Transformers Integration
from transformers import pipeline
from misar import MisarRouter
sentiment = pipeline("sentiment-analysis", model="CAMeL-Lab/bert-base-arabic-camelbert-mix-sentiment")
smart_sentiment = MisarRouter(tokenizer=sentiment.tokenizer).wrap_hf_pipeline(sentiment)
print(smart_sentiment("الكود مو معقد أبداً بس السيرفررررر بطييييء #مشكلة"))
CLI & Web Workbench
Misar includes a clean command-line interface and a self-hosted HTTP server + Arabic RTL Interactive Workbench:
# Run 6 built-in cross-dialect samples with live AraBERTv2 tokenization
misar demo
# Route a single sentence and inspect JSON telemetry
misar route "أعلنت منصة المطورين عن ترقية واجهة البرمجة API لدعم نماذج المحولات" --json
# Run the reproducible 15-sample benchmark suite
misar bench
# Launch the interactive RTL Web Workbench & hardened REST API on http://127.0.0.1:8050
misar serve
Embedded REST API Endpoints (misar serve)
| Method | Endpoint | Description |
|---|---|---|
GET |
/api/health |
Service status, version, and active tokenizer backend |
GET |
/api/suite |
Evaluates the 6 standard cross-domain reference samples |
POST |
/api/route |
Body {"text": "..."} → Returns full RoutingResult telemetry |
POST |
/api/batch |
Body {"texts": ["...", "..."]} → Batch routes up to 500 strings (1 MiB payload limit) |
Repository Structure
misar/
├── .github/workflows/ci.yml # Cross-OS Python 3.9–3.13 CI pipeline
├── docs/misar-architecture.png # High-resolution system architecture diagram
├── benchmarks/
│ ├── run_benchmark.py # Reproducible benchmark runner
│ └── latest_benchmark_report.json # Verified telemetry snapshot
├── src/misar/
│ ├── __init__.py # Public API exports
│ ├── profiler.py # O(n) 5-signal LinguisticProfiler & Shannon entropy
│ ├── lexicons.py # MSA-safe multi-region dialect & negation lexicons
│ ├── pipelines.py # FAST_MSA, DIALECT_PRESERVE & NOISY_SOCIAL passes
│ ├── tokenizer.py # Pure-Python WordPieceTokenizer + HF & offline adapters
│ ├── router.py # MisarRouter orchestration & wrap_hf_pipeline
│ ├── benchmark.py # Shared benchmark suite module
│ ├── cli.py # `misar` CLI (`demo`, `route`, `bench`, `serve`)
│ └── web/
│ ├── server.py # Hardened zero-dependency ThreadingHTTPServer REST API
│ └── index.html # Interactive Arabic RTL Workbench & token visualizer
├── tests/test_misar.py # 23 unit, security, server & real-vocab tests
├── CITATION.cff # Native GitHub repository citation metadata
├── CONTRIBUTING.md # Developer setup & contribution standards
├── SECURITY.md # Built-in API & frontend hardening controls
├── CHANGELOG.md # Formal release history
├── LICENSE # MIT License
└── pyproject.toml # PEP 621 PyPI package specification
Testing & Verification
python -m unittest discover -s tests -v
All 23 automated tests verify profiler mathematics, MSA homograph false-positive resistance, Saudi/Egyptian/Levantine/Maghrebi negation preservation, gemination protection (والللله → والله), markdown stripping, tatweel/diacritic splitting, authentic AraBERTv2 WordPiece reduction, and REST server HTTP security bounds.
Citation
If you use Misar in academic research, benchmarks, or commercial NLP pipelines, please cite:
@software{altheeb2026misar,
author = {Muneer Jarab Matrak Al-Theeb},
title = {Misar (مِسار): Adaptive Context-Aware Arabic NLP Preprocessing Router for Transformer Language Models},
year = {2026},
version = {1.0.0},
url = {https://github.com/RealMtrx/misar},
license = {MIT}
}
📖 نظرة عامة بالعربية
مِسار (Misar) هي مكتبة وموجّه بايثون مفتوح المصدر وخفيف الوزن (بلا أي تبعيات خارجية إجبارية) صُمِّم ليعمل كطبقة بوابة ذكية في زمن خطي $O(n)$ (~0.22 مللي ثانية للنص) قبل نماذج المحولات اللغوية العربية مثل AraBERTv2 و CAMeLBERT والنماذج التوليدية (ALLaM و LLMs) دون الحاجة إلى إعادة تدريب النموذج (Zero Parameter Update).
- توجيه ثلاثي المسارات: يفحص النص في تمريرة واحدة ويختار بين مسار الفصحى الحافظ للإملاء (
FAST_MSA)، أو مسار حفظ اللهجة وأدوات النفي (DIALECT_PRESERVE) لحماية كلمات مثلمو، مب، مافيه، ماكو، أو مسار تنقية ضجيج شبكات التواصل (NOISY_SOCIAL) الذي يضغط المدود والتطويل مع الحفاظ على الكلمات المشددة (والللله ← والله). - توفير التوكنات وتكاليف الـ APIs: أظهرت التجارب المرجعية الموثقة على معجم
AraBERTv2خفض إجمالي التوكنات بنسبة 46.57% مقارنة بالنص الخام و26.33% مقارنة بمناهج التطبيع القديمة، مع استعادة 100% من الكلمات المتأثرة بالتشكيل والكشيدة (156 → 0 [UNK]).
Authored and maintained by Muneer Jarab Matrak Al-Theeb (منير جراب مترك الذيب)
Released under the MIT License © 2026
Metadata
Release files for misar 1.0.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| misar-1.0.0.tar.gz | 50.0 kB | With provenance | Details for misar-1.0.0.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| misar-1.0.0-py3-none-any.whl | Python 3 | none | any | Details for misar-1.0.0-py3-none-any.whl |
Total release size: 95.1 kB
Release files / misar-1.0.0.tar.gz
| Download URL | misar-1.0.0.tar.gz |
|---|---|
| Size | 50.0 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
ee879ebbdba470d292841efd2bf3df759f3151e029c0182d691847b03a47fe19
|
|
BLAKE2b-256 checksum How to use checksums |
8155f808ebcb9f06020409c3530622efe19dc15ff5b8a6d365ad154063be8b46
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 19, 2026.
Transparency logRelease files / misar-1.0.0-py3-none-any.whl
| Download URL | misar-1.0.0-py3-none-any.whl |
|---|---|
| Size | 45.1 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
75b27794be9f672e2e5c30e6eb8b45d52f5a5a17eb35362f22a03ced6476f793
|
|
BLAKE2b-256 checksum How to use checksums |
2a265b1dba8ca0f0ed2e327ac17118e8ae65d045b6479cc53f011ffb8303126b
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 19, 2026.
Transparency logSecurity for misar 1.0.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
Full provenance
Both files in this release have been uploaded with provenance metadata.
Attestation for misar-1.0.0.tar.gz
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 19, 2026.
Transparency logA note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.