Skip to main content

مِسار — Misar

Adaptive $O(n)$ Context-Aware Arabic NLP Preprocessing Router for Transformer Language Models
الموجّه اللغوي التكيفي لتعزيز كفاءة ودقة معالجة اللغة العربية في نماذج الذكاء الاصطناعي

CI Python 3.9+ Dependencies License: MIT Latency Token Reduction

Architecture • Benchmarks • Installation • Quick Start • CLI & Workbench • Citation • النسخة العربية


Misar Adaptive Arabic NLP Preprocessing Router Architecture

Why Misar?

Modern Arabic NLP deployments face a structural bottleneck: forcing heterogeneous Arabic text (Modern Standard Arabic, classical diacritized prose, Gulf/Egyptian/Levantine/Maghrebi dialects, and noisy social media) through a single static preprocessing pipeline actively degrades both accuracy and inference economics:

  1. Legacy Normalization Damages Modern Transformers: Classical regex pipelines blindly collapse ة → ه and أ/إ/آ → ا. However, modern Transformer vocabularies such as AraBERTv2 (64k WordPiece) and CAMeLBERT explicitly preserve Hamzas and Taa Marbuta—meaning legacy normalization shatters known whole words (e.g., الحوسبة as 1 token splits into الحو + ##سبه as 2 tokens).
  2. Dialectal Sentiment & Negation Erasure: Over-aggressive cleaning distorts critical colloquial negation and sentiment markers (مو, مب, مافيه, ماكو), reversing downstream classification polarity.
  3. Diacritic, Tatweel & Elongation Token Explosion: Unstripped harakat (تشكيل), kashida (ـ), and character stretches (بطييييء, السيرفررررر) trigger OOV [UNK] blindness or inflate subword token counts—multiplying Transformer $O(L^2)$ self-attention cost and API billing.

Misar (مِسار) solves all three challenges without retraining any model (Zero Parameter Update) by acting as an ultra-lightweight, zero-dependency, linear-time $O(n)$ routing gateway directly in front of your Transformer or LLM pipeline.


Architecture

Every incoming text passes once through LinguisticProfiler (< 0.25 ms on CPU) to compute five quantitative signals (dialect_ratio $P_{\text{dial}}$, diacritic_density $D_{\text{diac}}$, elongation_noise_score $E_{\text{noise}}$, character Shannon Entropy $H_{\text{char}}$, and code_switching_ratio $R_{\text{latin}}$), then dispatches dynamically to the optimal micro-pipeline:

flowchart LR
    A["Raw Arabic Input<br/>(MSA / Diacritized / Dialect / Social)"] --> B["LinguisticProfiler O(n)<br/>P_dial · D_diac · E_noise · H_char · R_latin"]
    B -->|"High Noise / Tatweel / Stretches ≥ 3"| C["NOISY_SOCIAL (+ Hybrid Dialect)<br/>Gemination-Safe Collapse + Tag Unpacking"]
    B -->|"Dialect Markers / Protected Negations"| D["DIALECT_PRESERVE<br/>Negation Protection + Vocab-Aligned Cleanup"]
    B -->|"Clean MSA / Classical Diacritized"| E["FAST_MSA<br/>Strip Harakat · Preserve أ/إ/آ, ة, ى"]
    C --> F["AraBERTv2 / CAMeLBERT / ALLaM / LLMs<br/>(-46.57% Tokens · 0 [UNK])"]
    D --> F
    E --> F

Empirical Benchmarks

Measured reproducibly via python benchmarks/run_benchmark.py against the authentic 64,000-token aubmindlab/bert-base-arabertv02 WordPiece vocabulary across 15 curated multi-domain samples (latest_benchmark_report.json):

Domain Category Samples Raw Tokens Legacy Static (ة→ه, أ→ا) Misar Adaptive Misar Gain vs. Raw
MSA Engineering & Code-Switch 3 57 79 (+38.6% fragmentation) 57 Preserves 100% orthography (-27.85% vs Legacy)
Classical Diacritized (Heritage) 3 186 (147 OOV [UNK]) 48 39 -79.03% tokens & 100% word recovery (0 [UNK])
Saudi / Gulf / Levantine / Maghrebi Dev 6 121 131 117 -3.31% tokens (-10.69% vs Legacy) + negations intact
Noisy Tech Social & Code-Switching 3 102 80 36 -64.71% tokens + gemination preserved
Full Corpus Summary 15 466 338 249 -46.57% vs Raw · -26.33% vs Legacy · 0.223 ms/text

Key Scientific Proof: On Standard Arabic technical prose (MSA Tech / Engineering), blind legacy normalization inflated token count from 57 → 79 due to ة → ه subword splitting, while Misar maintained 57 intact vocabulary tokens (TFR ≈ 1.0). Across all 15 multi-domain inputs, Misar saved 46.57% of total tokens vs. raw text and 26.33% vs. legacy normalization (recovering 156 → 0 [UNK] tokens) with 0.223 ms mean latency on pure CPU.


Installation

Core misar requires zero mandatory external packages (Python 3.9+ Standard Library only):

# Install directly from GitHub
pip install git+https://github.com/RealMtrx/misar.git

# Or from PyPI once published
pip install misar

Optional extras for live Hugging Face Transformers wrapping and package development:

pip install "misar[huggingface]"   # transformers + torch
pip install "misar[dev]"           # pytest + build + twine

(Note: On first use of the built-in WordPieceTokenizer, Misar caches the official 64k vocabulary file locally under ~/.cache/misar/. Pass allow_vocab_download=False or --offline to run completely air-gapped using HeuristicSubwordTokenizer.)


Quick Start

1. Dynamic Text Routing & Token Telemetry

from misar import MisarRouter

router = MisarRouter()  # Uses real AraBERTv2 64k WordPiece vocabulary

result = router.route("السيرفررررر بطييييء مررررة ومافيه استجابة في API أبدددداً 😡😡👎 #هندسة_البرمجيات")

print("Route:     ", result.route_name)          # NOISY_SOCIAL
print("Processed: ", result.processed_text)      # السيرفر بطيء مرة ومافيه استجابة في API أبدا هندسة البرمجيات
print("Tokens:    ", result.raw_token_count, "->", result.processed_token_count)  # 28 -> 14
print("Reduction: ", f"-{result.token_reduction_pct:.1f}%")                       # -50.0%
print("Latency:   ", f"{result.latency_ms:.3f} ms")

2. Zero-Latency Drop-In String Preprocessing & Batching

from misar import MisarRouter, preprocess

# One-line module helper
clean = preprocess("الســــلام علــــيكم ومافيه مشكلة #تطوير_البرمجيات")
# -> "السلام عليكم ومافيه مشكلة تطوير البرمجيات"

# Fast batch routing for production ETL pipelines
router = MisarRouter()
results = router.route_batch([
    "أصدرت عدة دول عربية بياناً مشتركاً لأول مرة عن صافي الأرباح",  # -> FAST_MSA (zero dialect false positives)
    "والللله الكود مو معقد أبداً والأخطاء متكررررة",                  # -> NOISY_SOCIAL (preserves والله and مو)
])

3. One-Line Hugging Face Transformers Integration

from transformers import pipeline
from misar import MisarRouter

sentiment = pipeline("sentiment-analysis", model="CAMeL-Lab/bert-base-arabic-camelbert-mix-sentiment")
smart_sentiment = MisarRouter(tokenizer=sentiment.tokenizer).wrap_hf_pipeline(sentiment)

print(smart_sentiment("الكود مو معقد أبداً بس السيرفررررر بطييييء #مشكلة"))

CLI & Web Workbench

Misar includes a clean command-line interface and a self-hosted HTTP server + Arabic RTL Interactive Workbench:

# Run 6 built-in cross-dialect samples with live AraBERTv2 tokenization
misar demo

# Route a single sentence and inspect JSON telemetry
misar route "أعلنت منصة المطورين عن ترقية واجهة البرمجة API لدعم نماذج المحولات" --json

# Run the reproducible 15-sample benchmark suite
misar bench

# Launch the interactive RTL Web Workbench & hardened REST API on http://127.0.0.1:8050
misar serve

Embedded REST API Endpoints (misar serve)

Method Endpoint Description
GET /api/health Service status, version, and active tokenizer backend
GET /api/suite Evaluates the 6 standard cross-domain reference samples
POST /api/route Body {"text": "..."} → Returns full RoutingResult telemetry
POST /api/batch Body {"texts": ["...", "..."]} → Batch routes up to 500 strings (1 MiB payload limit)

Repository Structure

misar/
├── .github/workflows/ci.yml           # Cross-OS Python 3.9–3.13 CI pipeline
├── docs/misar-architecture.png        # High-resolution system architecture diagram
├── benchmarks/
│   ├── run_benchmark.py               # Reproducible benchmark runner
│   └── latest_benchmark_report.json   # Verified telemetry snapshot
├── src/misar/
│   ├── __init__.py                    # Public API exports
│   ├── profiler.py                    # O(n) 5-signal LinguisticProfiler & Shannon entropy
│   ├── lexicons.py                    # MSA-safe multi-region dialect & negation lexicons
│   ├── pipelines.py                   # FAST_MSA, DIALECT_PRESERVE & NOISY_SOCIAL passes
│   ├── tokenizer.py                   # Pure-Python WordPieceTokenizer + HF & offline adapters
│   ├── router.py                      # MisarRouter orchestration & wrap_hf_pipeline
│   ├── benchmark.py                   # Shared benchmark suite module
│   ├── cli.py                         # `misar` CLI (`demo`, `route`, `bench`, `serve`)
│   └── web/
│       ├── server.py                  # Hardened zero-dependency ThreadingHTTPServer REST API
│       └── index.html                 # Interactive Arabic RTL Workbench & token visualizer
├── tests/test_misar.py                # 23 unit, security, server & real-vocab tests
├── CITATION.cff                       # Native GitHub repository citation metadata
├── CONTRIBUTING.md                    # Developer setup & contribution standards
├── SECURITY.md                        # Built-in API & frontend hardening controls
├── CHANGELOG.md                       # Formal release history
├── LICENSE                            # MIT License
└── pyproject.toml                     # PEP 621 PyPI package specification

Testing & Verification

python -m unittest discover -s tests -v

All 23 automated tests verify profiler mathematics, MSA homograph false-positive resistance, Saudi/Egyptian/Levantine/Maghrebi negation preservation, gemination protection (والللله → والله), markdown stripping, tatweel/diacritic splitting, authentic AraBERTv2 WordPiece reduction, and REST server HTTP security bounds.


Citation

If you use Misar in academic research, benchmarks, or commercial NLP pipelines, please cite:

@software{altheeb2026misar,
  author       = {Muneer Jarab Matrak Al-Theeb},
  title        = {Misar (مِسار): Adaptive Context-Aware Arabic NLP Preprocessing Router for Transformer Language Models},
  year         = {2026},
  version      = {1.0.0},
  url          = {https://github.com/RealMtrx/misar},
  license      = {MIT}
}

📖 نظرة عامة بالعربية

مِسار (Misar) هي مكتبة وموجّه بايثون مفتوح المصدر وخفيف الوزن (بلا أي تبعيات خارجية إجبارية) صُمِّم ليعمل كطبقة بوابة ذكية في زمن خطي $O(n)$ (~0.22 مللي ثانية للنص) قبل نماذج المحولات اللغوية العربية مثل AraBERTv2 و CAMeLBERT والنماذج التوليدية (ALLaM و LLMs) دون الحاجة إلى إعادة تدريب النموذج (Zero Parameter Update).

  • توجيه ثلاثي المسارات: يفحص النص في تمريرة واحدة ويختار بين مسار الفصحى الحافظ للإملاء (FAST_MSA)، أو مسار حفظ اللهجة وأدوات النفي (DIALECT_PRESERVE) لحماية كلمات مثل مو، مب، مافيه، ماكو، أو مسار تنقية ضجيج شبكات التواصل (NOISY_SOCIAL) الذي يضغط المدود والتطويل مع الحفاظ على الكلمات المشددة (والللله ← والله).
  • توفير التوكنات وتكاليف الـ APIs: أظهرت التجارب المرجعية الموثقة على معجم AraBERTv2 خفض إجمالي التوكنات بنسبة 46.57% مقارنة بالنص الخام و26.33% مقارنة بمناهج التطبيع القديمة، مع استعادة 100% من الكلمات المتأثرة بالتشكيل والكشيدة (156 → 0 [UNK]).

Authored and maintained by Muneer Jarab Matrak Al-Theeb (منير جراب مترك الذيب)
Released under the MIT License © 2026

Metadata

Release files for misar 1.0.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for misar 1.0.0
File Size Uploaded Provenance Actions
misar-1.0.0.tar.gz 50.0 kB With provenance Details for misar-1.0.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for misar 1.0.0
File Interpreter ABI Platform Actions
misar-1.0.0-py3-none-any.whl Python 3 none any Details for misar-1.0.0-py3-none-any.whl

Total release size: 95.1 kB

Release files / misar-1.0.0.tar.gz

Download URL misar-1.0.0.tar.gz
Size 50.0 kB
Tags Source
SHA-256 checksum
How to use checksums
ee879ebbdba470d292841efd2bf3df759f3151e029c0182d691847b03a47fe19
BLAKE2b-256 checksum
How to use checksums
8155f808ebcb9f06020409c3530622efe19dc15ff5b8a6d365ad154063be8b46
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 19, 2026.

Transparency log

Release files / misar-1.0.0-py3-none-any.whl

Download URL misar-1.0.0-py3-none-any.whl
Size 45.1 kB
Tags Python 3
SHA-256 checksum
How to use checksums
75b27794be9f672e2e5c30e6eb8b45d52f5a5a17eb35362f22a03ced6476f793
BLAKE2b-256 checksum
How to use checksums
2a265b1dba8ca0f0ed2e327ac17118e8ae65d045b6479cc53f011ffb8303126b
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 19, 2026.

Transparency log

Security for misar 1.0.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

Both files in this release have been uploaded with provenance metadata.

Select from 2 files

Attestation for misar-1.0.0.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 19, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page