darija-router 0.15.0

Moroccan Darija NER and semiotic class tagging — rule-based NER engine that outperforms GPT-4o-mini on Darija text. Built in Rust for O(1) lookup, 750x faster than LLMs.

darija-router

Moroccan Darija NER and semiotic class tagging — rule-based, zero-training, O(1) lookup. Built in Rust, wrapped for Python.

63.0% F1 on Darija NER — outperforms GPT-4o-mini (57.0%) at 750x the speed.

Benchmark

Evaluated on 500 Moroccan Darija sentences (998 annotations, 12 entity types). All LLM baselines zero-shot via OpenRouter API.

System	Precision	Recall	F1	Speed
darija-router	57.8%	69.3%	63.0%	1.2s
GPT-4o-mini	62.9%	52.2%	57.0%	~15min
GPT-4o	64.6%	43.2%	51.8%	~20min
Qwen3.6-35B	61.4%	23.1%	33.6%	~45min
GPT-5-mini	65.9%	8.9%	15.7%	~10min

darija-router runs on a single CPU core. No GPU. No API calls. No training data.

Ablation Study

Configuration	P	R	F1
Full system	57.8%	69.3%	63.0%
− Lexicon lookup	57.8%	69.3%	63.0%
− Morphological analysis	57.8%	69.3%	63.0%
− NER triggers (PER/LOC/ORG)	57.6%	70.0%	63.2%
− Temporal patterns	62.0%	59.7%	60.9%
− All semantic tagging	0.0%	0.0%	0.0%

See eval/README.md for full methodology and per-label results.

Install

pip install darija-router

Requires Python ≥ 3.8. Wheels available for Linux, macOS, and Windows (x86_64 and ARM64).

Quick Start

import darija_router

# NER — extract named entities
result = darija_router.run_ner("شريت 3 كيلو ديال التفاح ب 20 درهم ليوم")
print(result["entities"])
# [{"text": "3 كيلو", "entity_type": "MEASURE", ...},
#  {"text": "20 درهم", "entity_type": "MONEY", ...},
#  {"text": "ليوم", "entity_type": "DATE", ...}]

# Filter to specific entity types
result = darija_router.run_ner(
    "شريت 3 كيلو ب 20 درهم",
    labels=["MONEY", "MEASURE"]
)
print(result["entities"])
# [{"text": "3 كيلو", "entity_type": "MEASURE", ...},
#  {"text": "20 درهم", "entity_type": "MONEY", ...}]

# Code-switching detection
cs = darija_router.run_codeswitch("kifash nchri car b 5000 euro?")
print(cs["stats"]["darija"])  # 0.4
print(cs["stats"]["fr"])      # 0.2

# Full analysis
result = darija_router.analyze("خويا باغي نلقاك غدوة الساعة 3 صباح")

API Reference

run_ner(phrase, labels=None, config=None)

Named Entity Recognition. Returns span-based entities and per-token NER tags.

Parameters:

• phrase (str): Input text (Arabic script or Arabizi)
• labels (list[str], optional): Filter output to only these entity types. Default: all types. Useful for evaluation when the gold schema excludes PERSON, LOC, etc.
• config (dict, optional): Pipeline configuration for ablation studies. Keys: disable_lexicon, disable_morphology, disable_semantic, disable_temporal, disable_ner_triggers. All default to False.

Returns:

{
  "entities": [
    {"text": "500 درهم", "entity_type": "MONEY", "start": 0, "end": 8,
     "token_indices": [0, 1], "confidence": 0.95}
  ],
  "tokens": [
    {"token": "500", "ner_tag": "MONEY"},
    {"token": "درهم", "ner_tag": "MONEY"}
  ]
}

analyze(phrase, mode=None, fields=None)

Tokenizes a phrase and classifies each token.

Parameters:

• phrase (str): Input text in Arabic script
• mode (str, optional): Output preset — "full" (default), "compact", "ner", "codeswitch"
• fields (list[str], optional): Custom field selection — e.g. ["token", "lang", "confidence"]

Returns: list[dict] with per-token results.

run_codeswitch(phrase)

Code-switching detection with language spans and statistics.

Returns:

{
  "spans": [{"text": "خويا واش", "lang": "darija", "start": 0, "end": 9}],
  "tokens": [{"token": "خويا", "lang": "darija", "latin": null, "confidence": 1.0}],
  "stats": {"darija": 50.0, "fr": 25.0, "en": 0.0, "num": 12.5, "punct": 0.0, "other": 0.0, "unknown": 12.5}
}

lookup(word)

Single word lookup. Same format as one element from analyze().

normalize_arabic(text) / arabizi_to_arabic(text) / normalize(text)

Normalization functions. normalize() auto-detects script and applies the appropriate function.

tokenize(text)

Tokenize text into tokens with character offsets.

detect_script(text)

Detect script: "arabic", "latin", "mixed", or "unknown".

__version__

Package version string (e.g. "0.15.0").

Entity Types

Type	Detection	Example
CARDINAL	Digits or Arabic number words	50 عشرين مية
ORDINAL	Arabic ordinal words (standard + Darija)	تاني تالت اللول
DECIMAL	Decimal numbers	3.5 ١٢.٥
MONEY	Cardinal + currency word	500 درهم
MEASURE	Cardinal + unit word	2 كيلو 24 ساعة
TIME	Clock times, periods, durations	الساعة 3 صباح 8 د الصباح
DATE	Date words, years, day names, durations	غدوة 15 مارس 2026
PHONE	Phone number patterns (Moroccan)	[phone] 0661234567
EMAIL	Email addresses	[e-mail]
URL	URLs	http://google.com
PERCENT	Percentage suffixes	20% بالمئة
ADDRESS	Address keywords + following word	زنقة الحسن
PERSON	Honorifics + name expansion	سيدي محمد لالة فاطمة
LOC	Location triggers + expansion	شارع محمد الخامس مدينة الدار البيضاء
ORG	Organization triggers + expansion	شركة مسير بنك الشعب
FACILITY	Facility triggers + expansion	سبيسان مسجد الكبير
EVENT	Event triggers + expansion	عيد الفطر عرس
MISC	Nationalities, languages, religions	المغربي الفرنساوي الإسلام
FRACTION	Arabic fraction words	نص ربع تلت

Semantic Tags

Semantic tags are internal per-token labels that feed into the NER engine. They correspond to entity types above but may include intermediate labels like PUNCT.

Tag	Detection	Example
PUNCT	Punctuation characters	؟ ، .
CARDINAL	Digits or Arabic number words	50 عشرين مية
ORDINAL	Arabic ordinal words	تاني تالت اللول
MONEY	Cardinal + currency word	50 درهم → both tokens
MEASURE	Cardinal + unit word	2 كيلو → both tokens
TIME	Time patterns	الساعة 3 صباح
DATE	Date patterns	غدوة 15 مارس
FRACTION	Fraction words	نص ربع
PERCENT	Percentage suffix	20%
PHONE	Phone patterns	+212...
EMAIL	Email patterns	user@...
URL	URL patterns	http://...
ADDRESS	Address keywords	زنقة شارع

Output Modes

Mode	Description
"full" (default)	All fields: token, index, lang, entries, stem, semantic, confidence
"compact"	token + lang only (no entries/stem)
"ner"	NER preset: entities + tokens with ner_tag
"codeswitch"	Code-switching preset: spans + tokens + stats

Confidence Scoring

Match Type	Score
Exact match	1.0
Normalized (ة→ه)	0.95
Single prefix/suffix strip	0.85
Double strip (prefix + suffix)	0.70
Semantic tag only (no lexicon match)	0.60
Unknown	0.0

Pipeline Architecture

darija-router processes text through a six-phase pipeline:

1. Script Detection — classify each token as Arabic, Latin, Mixed, or Numeric
2. Normalization — Arabizi (Latin script) → Arabic script using dictionary + heuristics (427 exceptions)
3. Morphological Analysis — strip Arabic prefixes (7) and suffixes (5) withت→ه normalization
4. Lexicon Lookup — O(1) HashMap lookup against 2,187 embedded entries, with morphological fallback
5. Semantic Tagging — context-aware rule engine detecting 14+ categories with multi-token grouping
6. NER Extraction — trigger-based detection for PERSON, LOC, ORG, FACILITY, EVENT, MISC with expansion rules

All lexicons compile into the binary via include_str!(). Zero runtime file I/O. Zero startup cost. 3.2 MB binary.

Temporal Expression Recognition

darija-router recognizes 12 distinct Darija temporal patterns:

Pattern	Example	Structure
Clock time	الساعة 8	article + number
Prepositional	8 د الصباح	number + preposition + period
Contracted	12 دالليل	number + attached preposition
Duration hours	24 ساعة	number + hour unit
Duration minutes	15 دقيقة	number + minute unit
Fraction hour	نص ساعة	fraction + unit
Clock with fraction	6 ونص	number + connector + fraction
Midnight	منتصف الليل	fixed phrase
Noon	منتصف النهار	fixed phrase
Period only	8 مساء	number + period word
Definite hour	السبعة	article + number word
HH:MM	14:30	direct time notation

This specialized handling improved TIME F1 from 0% to 38.1%.

Build from Source

git clone https://github.com/LaamiriOuail/darija-router.git
cd darija-router
pip install maturin
maturin develop --release

Testing

pip install pytest
pytest tests/ -v

695+ tests covering tokenization, normalization, morphology, semantic tagging, NER, code-switching, and span positions.

Citation

If you use darija-router in your research, please cite:

@software{darija_router_2026,
  author = {Laamiri, Ouail and Berrada, Ismail},
  title = {darija-router: Rule-Based NER for Moroccan Darija},
  year = {2026},
  url = {https://github.com/LaamiriOuail/darija-router},
  note = {63.0\% F1 on Darija NER, outperforming GPT-4o-mini at 750x speed}
}

License

MIT