darija-router 0.3.0

Moroccan Darija (Arabic) phrase analyzer — classify each word by source language (French/English/Darija), detect semantic entities (money, time, date, phone...) and strip morphological prefixes. Built in Rust for O(1) lookup.

darija-router

Moroccan Darija language router — O(1) phrase analysis with morphological stripping, semantic tagging, NER, and code-switching detection. Built in Rust, wrapped for Python.

Classifies Arabic-script Darija words by source language (fr, en, darija), Latin spelling, category, probability, semantic role, and confidence score.

Install

pip install darija-router

Quick Start

import darija_router

result = darija_router.analyze(
    "خويا باغي نلقاك غدوة الساعة 3 صباح فالساحة، "
    "واش باغي نشريو شي تيليفون جديد ب 500 درهم؟"
)

API Reference

darija_router.analyze(phrase, mode=None, fields=None)

Tokenizes a phrase and classifies each token.

Parameters:

• phrase (str): Input text in Arabic script
• mode (str, optional): Output preset — "full" (default), "compact", "ner", "codeswitch"
• fields (list[str], optional): Custom field selection — e.g. ["token", "lang", "confidence"]

Returns: list[dict] with per-token results.

darija_router.run_ner(phrase)

Named Entity Recognition. Returns span-based entities and per-token NER tags.

Returns:

{
  "entities": [
    {"text": "500 درهم", "entity_type": "MONEY", "start": 0, "end": 8, "token_indices": [0, 1]}
  ],
  "tokens": [
    {"token": "500", "ner_tag": "MONEY"},
    {"token": "درهم", "ner_tag": "MONEY"}
  ]
}

Entity types: MONEY, MEASURE, TIME, DATE, PHONE, EMAIL, URL, CARDINAL, DECIMAL, ORDINAL, FRACTION, PERCENT, PERSON, LOC

darija_router.run_codeswitch(phrase)

Code-switching detection with language spans and statistics.

Returns:

{
  "spans": [
    {"text": "خويا واش", "lang": "darija", "start": 0, "end": 9}
  ],
  "tokens": [
    {"token": "خويا", "lang": "darija", "latin": null}
  ],
  "stats": {
    "darija": 50.0, "fr": 25.0, "en": 0.0, "num": 12.5, "punct": 0.0, "unknown": 12.5
  }
}

darija_router.lookup(word)

Single word lookup. Same format as one element from analyze().

darija_router.__version__

Package version string.

Semantic Tags

Tag	Detection	Example
PUNCT	Punctuation characters	؟ ، . !
CARDINAL	Digits or Arabic number words	50 عشرين مية
DECIMAL	Decimal numbers	3.5 ١٢.٥
MONEY	Cardinal + currency word	50 درهم → both tokens tagged
MEASURE	Cardinal + unit word	2 كيلو → both tokens tagged
TIME	الساعة + cardinal, or time words	الساعة 3 صباح مساء
DATE	Date words or cardinal + month	غدوة 15 مارس
ORDINAL	Arabic ordinal words	تاني تالت أول
FRACTION	Arabic fraction words	نص ربع تلت
PERCENT	% suffix or بالمئة	20%
PHONE	Phone number patterns	[phone]
EMAIL	Contains @ and .	[e-mail]
URL	Starts with http or www.	http://google.com
ADDRESS	Address keyword + following word	زنقة الحسن

Output Modes

Mode	Description
"full" (default)	All fields: token, index, lang, entries, stem, semantic, confidence
"compact"	token + lang only (no entries/stem)
"ner"	NER preset: entities + tokens with ner_tag
"codeswitch"	Code-switching preset: spans + tokens + stats

Confidence Scoring

Match Type	Score
Exact match	1.0
Normalized (ة→ه)	0.95
Single prefix/suffix strip	0.85
Double strip (prefix + suffix)	0.70
Semantic tag only (no lexicon match)	0.60
Unknown	0.0

How It Works

1. Embedded lexicon — 1,909 Arabic-script keys (2,209 entries) compiled into the binary via include_str!(). Zero file I/O, zero startup cost.

2. O(1) HashMap lookup — exact match against a HashMap<String, Vec<Entry>>.

3. Morphological fallback — when an exact match fails, the engine strips prefixes and suffixes:

   Prefixes: ال و ف ب ل م ك

   Suffixes: هم ها ش ك ي

   Normalization: ة → ه

   Tries: exact → normalize → single strip → double strip. Minimum stem length of 2 prevents over-stripping.

4. Semantic tagger — rule-based second pass detects 14 semantic categories via regex patterns and context-aware grouping (e.g. cardinal + currency word → MONEY).

5. NER engine — extracts span-based entities with context rules for PERSON (سيدي/لالة + next word) and LOC (شارع/زنقة/حي + next words).

6. Code-switching detector — assigns per-token language labels, groups consecutive same-language tokens into spans, and computes aggregate statistics.

7. 23 thematic categories — technology, food, transportation, health, administration, clothing, home, education, sports, work, money, adjectives, family, colors, time, nature, slang, functional grammar, common verbs, common descriptions, time expressions, numbers.

Build from Source

git clone https://github.com/LaamiriOuail/darija-router.git
cd darija-router
pip install maturin
maturin develop --release

License

MIT