arabic-name-romanizer

Offline, CPU-only transliteration of Arabic personal names (a given name, a surname, or a full name) into their likely real-world Latin spelling.

from arabic_name_romanizer import transliterate

transliterate("محمد صلاح")  # 'Mohamed Salah'
transliterate("حبيبة عطيفي")  # 'Habiba Atifi'
transliterate("عبد الرحمن", top_k=3)  # [Candidate('Abdulrahman', ...), Candidate('Abderrahmane', ...), ...]
transliterate("عبد القادر", country="DZ")  # 'Abdelkader'

It produces likely or common spellings, not legally authoritative identity-document transliterations. It is not a general Arabic transliterator and not a translator.

Why Arabic → Latin names are ambiguous

Arabic writing omits short vowels, so محمد is attested as Mohamed, Mohammed, Muhammad and Muhammed. Conventions differ by region: Egypt writes Gamal where the Gulf writes Jamal; Algeria, Morocco and Tunisia follow French spelling (Youcef, Abdelkader, Benali) where the Mashreq follows English (Yusuf, Abdulqadir, Bin Ali). One Arabic name legitimately has several correct Latin spellings, and the library returns ranked candidates rather than one "true" form.

How it works

flowchart TD
    IN["Arabic input<br/>e.g. محمد عبد الرحمن الزهراني<br/>+ optional country / role hint"] --> NORM["Normalization<br/>NFKC, strip harakat and tatweel,<br/>unify Persian letters, punctuation → space"]
    NORM --> WHOLE{"Whole string known?<br/>user dictionary, then shipped index"}
    WHOLE -- "yes, trusted (count ≥ 3)" --> OUT
    WHOLE -- no --> SPLIT["Split into units<br/>whitespace tokens; particles merge:<br/>عبد الرحمن · بن علي · سيف الدين"]
    SPLIT --> UNIT{"Each unit"}
    UNIT --> UD["User dictionary<br/>(TSV, always wins)"]
    UD -- miss --> IDX["Shipped index<br/>SQLite, ~60k attested spellings<br/>from Wikidata + Algerian corpora<br/>ranked by country → region → all"]
    IDX -- miss --> ML["ML fallback<br/>char-level GRU seq2seq with attention<br/>1.7M params, ONNX Runtime, CPU<br/>input: country + role tokens + letters<br/>batched beam search, width 2"]
    ML -- "no model installed" --> RULES["Rule-based romanization<br/>(last resort)"]
    UD --> RANK
    IDX --> RANK
    ML --> RANK
    RULES --> RANK
    RANK["Combine units<br/>score = product of unit scores,<br/>dictionary: frequency × trust,<br/>model: exp(mean log-prob)"] --> OUT["Top-k candidates<br/>text, score, source per unit"]

No ML runs for a name whose units are all known: the dictionary answers, in 0.1–0.4 ms. The network only sees units nobody in the data has ever spelled. Training (PyTorch, training/) is offline; the package ships the exported ONNX graphs and needs only numpy and onnxruntime.

Why dictionary first

Known real-world spellings beat generated ones. The pipeline is:

Arabic input
     |
     v
normalize
     |
     v
level 0: user override dictionary, whole string     (if hit: done)
     |
     v
level 1: built-in index, whole string               (if hit with count >= 3: done)
     |
     v
split on whitespace into units; merge particle + next token
(عبد الرحمن, أبو بكر, بن علي, آل سعود are one unit each)
     |
     v
for each unit:  user dictionary → built-in unit index → neural model (beam search) → rules
     |
     v
combine per-unit candidates (score product) → rank → join with spaces

The built-in index holds attested spellings from Wikidata; the neural model is a 1.7M-parameter GRU sequence-to-sequence model run with ONNX Runtime and is only consulted for name units the index has never seen.

Install

pip install arabic-name-romanizer      # or: pip install dist/arabic_name_romanizer-0.1.0-py3-none-any.whl

Runtime dependencies: onnxruntime and numpy. PyTorch is only needed for training. Requires Python 3.11–3.14. The wheel is 9.3 MB compressed, 16 MB installed (8.6 MB SQLite index, 7.1 MB ONNX graphs).

Python API

from arabic_name_romanizer import Transliterator, UserDictionary, transliterate, transliterate_many

transliterate("محمد")  # 'Mohamed'
transliterate("محمد", country="SA")  # 'Mohammed'
transliterate("حسن", role="surname")  # role hint: 'given' | 'surname'
transliterate_many(["محمد", "علي"])  # ['Mohamed', 'Ali']

for c in transliterate("محمد زقزوق", top_k=3):
    print(c.text, round(c.score, 3), c.source)  # source: dictionary | model | rules | mixed | user | passthrough
    for u in c.units:  # per-unit provenance
        print("  ", u.arabic, "->", u.latin, u.source, u.count)

# your own attested spellings always win
t = Transliterator(user_dictionary=UserDictionary.from_pairs([("حبيبة عطيفي", "Habiba Atify")]))
t.transliterate("حبيبة عطيفي")  # 'Habiba Atify'

score is a ranking heuristic (relative attested frequency for dictionary hits, exp of the mean per-character log-probability for the model), not a calibrated probability. Non-Arabic input is passed through unchanged.

CLI

arabic-name-romanize "محمد صلاح"
arabic-name-romanize "محمد صلاح" --top-k 5
arabic-name-romanize "عبد القادر" --country DZ
arabic-name-romanize --json --top-k 3 "زقزوق"
cat names.txt | arabic-name-romanize --user-dict my_names.tsv     # one name per line on stdin

The user dictionary is a TSV: arabic<TAB>latin[<TAB>count[<TAB>country]].

CPU-only inference

Inference needs no GPU and no PyTorch. The model ships as two ONNX graphs (encoder, decoder step) plus a character vocabulary. Decoding is batched beam search across all name units of a batch; identical units are decoded once. Output is deterministic for a fixed model file, ONNX Runtime version and CPU; floating-point differences across CPUs can reorder near-tied candidates.

Training data

Filters remove titles, regnal ordinals, academic romanizations with macrons and length mismatches. Units are aligned to Latin tokens positionally, or by a small DP aligner when a label drops a middle name or spells a particle unit as several tokens (عبد الرحمن → Abd al Rahman); 99.9% of accepted pairs yield unit pairs. People born before 1900 are excluded from the shipped index and training. Full details in DATA_LICENSES.md and DESIGN.md.

What the errors look like

Measured on 5,000 names composed from a hand-written lexicon (data/llm_test/names.jsonl, country hint on, full report in artifacts/llm_test/report.md). A name counts as correct only if every unit is one of the lexicon's accepted spellings; 3,923 of the 11,902 unit occurrences (33%) were not. Grouped by what actually differs:

About two thirds of the differences are therefore another correct spelling of the same name, not a wrong name. On this set the model is almost never the source: 98% of the units are found in the dictionary, so what is measured is which attested variant gets ranked first. The country hint is worth two points of acceptable matches and eight points once typography is ignored.

Benchmark results

See DESIGN.md (section "Experiment log") for the full tables. Test split (never seen in training, split by whole-name key): 2,472 full names and 4,225 name-unit groups, of which 807 units never occur in the training data. All rows are measured on this (round 4, cleaned) test set.

Exact match is strict against the attested test labels; on full names 82% of top-1 outputs are attested spellings of every unit ("expanded" match), and the dictionary answers directly for every name whose units are known.

An independent set of 5,000 names composed from a hand-written lexicon (scripts/llm_lexicon.py, common names only, country hint on) gives acceptable EM@1 0.367, or 0.522 when hyphen/space/case differences are ignored, CER 0.088. Two thirds of its remaining "errors" are a different attested variant (vowel choice, Al- vs Al , Abdel vs Abdul); see DESIGN.md round 4 for the breakdown and artifacts/llm_test/report.md for every failure. The model also receives the unit's role (given name / surname, from its position or the role argument), which is worth about one point on unknown units.

Throughput on an Apple M4, single process, machine idle, measured by scripts/benchmark.py:

The project requirement was at least 1,000 names/s model-only bulk throughput; the shipped configuration passes it with margin. Run-to-run variance on a laptop is about ±30%.

Coverage by country and language

The index and the model were built from people with citizenship in the 22 Arab League states, so that is where the library is measured. "Index rows" is the number of attested unit spellings for that country in the shipped names.sqlite3; "unseen-unit EM@1" is the model alone on Wikidata test units never seen in training (blank when fewer than 50 test units); "5k set" is acceptable EM@1 on the independent lexicon-based set with the country hint on. Pass the country code as country= to get its conventions.

Names from the 22 covered countries that are not in the index go to the model with the country token; a country with no attested spelling for a unit falls back to its region (Maghreb = DZ, MA, TN, MR; Mashriq/Gulf = the rest) before the global pool. Country codes outside the 22 are accepted but carry no data: the lookup falls through to the global pool and the model sees the unknown-country token.

Persian-script names: Afghanistan, Iran, Pakistan

Since 0.2.0 the package ships a second model and dictionary rows for names written in the Persian script. Routing is automatic: a country of AF, IR or PK, or any Persian-, Pashto- or Urdu-only letter in the input (پ چ ژ گ ک ی, ښ ډ ټ ړ ږ ڼ, ٹ ڈ ڑ ھ ں ے), selects the Persian path. Arabic input never touches it, and the Arabic results are byte-for-byte what they were in 0.1.x. The Persian path folds ک/ی to Arabic letters, treats the zero-width non-joiner as a word boundary (محمد‌جواد becomes two units, نجیب‌الله one unit نجیب الله), keeps the language-specific letters, and falls back country → persian region → Persian-script rows only.

transliterate("حامد کرزی", country="AF")  # 'Hamid Karzai'
transliterate("احمد شاه مسعود", country="AF")  # 'Ahmad Shah Massoud'
transliterate("ذبیح‌الله مجاهد", country="AF")  # 'Zabihullah Mujahid'
transliterate("حسن روحانی")  # 'Hassan Rouhani' (Persian letters detected)

Data: 33k people with citizenship in Afghanistan, Iran or Pakistan and a fa, ps or ur label on Wikidata, plus their English aliases (CC0). Afghanistan alone is 1.4k people, so the Afghan conventions come from the <AF> country token and the Afghan rows; the model itself is trained on all three countries. Measured:

Still weak: Pashto tribal names that occur only once in the data, vowel choices between Afghan and Iranian conventions (Zahir vs Zaher), and Kurdish (Sorani) orthography, which is not covered at all. Names from your own Afghan records loaded as a user dictionary win over everything else.

Limitations

• Foreign names written in Arabic (جون سميث) are out of scope; they will be transliterated as if Arabic.
• Historical figures are excluded from the data; academic romanizations (Muḥammad) are never produced.
• A dictionary entry seen once is one observation, not the truth. A whole-name entry below the trust threshold (3) is ranked jointly with the per-unit candidates; a unit entry is still preferred over the model, because attested spellings beat generated ones 9 points of the time (Transliterator(model_below_trust=True) changes this and gains 1.4 points of strict exact match).
• Scores are not calibrated probabilities.
• Regional spelling depends on the country hint; without it the most frequent attested spelling across all countries wins.

Reproducing

uv sync
uv run python scripts/download_data.py --with-dzirinames
uv run python scripts/build_dataset.py
uv run python training/train.py --config training/configs/gru_role.yaml --run-dir artifacts/runs/gru_role
uv run python training/evaluate.py --model artifacts/runs/gru_role --beam-width 2 --out artifacts/eval/gru_role.json
uv run python scripts/export_onnx.py --run artifacts/runs/gru_role --out artifacts/onnx/gru_role --beam-width 2
uv run python scripts/build_lookup.py
uv run python scripts/benchmark.py --model-dir src/arabic_name_romanizer/resources
uv run pytest

Licensing

Code: MIT (LICENSE). The shipped index and weights derive from Wikidata (CC0), the Algerian Name Transcription Corpus (CC0) and the DziriNames benchmark (MIT, which in turn credits the Amoura Kaggle dataset, MIT). The required MIT notices are in THIRD_PARTY_NOTICES.md, which is packaged into the wheel's dist-info directory; keep that file with any redistribution. Details per source in DATA_LICENSES.md.

CPU only

Inference, the test suite and the benchmarks run on CPU: ONNX Runtime is created with CPUExecutionProvider only, the training smoke tests pin device: cpu, and scripts/benchmark.py measures the shipped ONNX files, never a torch model. A GPU is neither required nor used at runtime; torch is a dev dependency for training only.