urdunorm

Urdu text normalization for ASR evaluation, with explicit and reproducible protocols.

pip install urdunorm

from urdunorm import normalize

normalize("کیلئے", mode="eval")  # 'کے لیے'
normalize("بھائی", mode="eval")  # 'بھائی'   aspiration preserved
normalize("ہاں", mode="eval")  # 'ہاں'     nasalization preserved

urdunorm --mode eval < ref.txt > ref.norm.txt
urdunorm --mode lums-compat --jsonl-field text < hyp.jsonl

Why this exists

Urdu word error rate is not currently a well-defined quantity. Published numbers for the same model on the same benchmark differ by tens of points across studies. Most of that spread is not modelling. It comes from two choices that papers rarely state: which held-out set was used, and how the text was normalized before scoring.

Urdu makes the normalization choice unusually consequential.

The script has near-duplicate characters that carry no phonemic contrast, so ي U+064A and ی U+06CC are the same letter typed on different keyboards. A normalizer that ignores this charges a model for a keyboard layout.

It also has near-duplicate characters that carry a real contrast, and merging them is the standard bug. ھ U+06BE marks aspiration: بھائی is "brother", بہائی is not the same word. ں U+06BA marks nasalization: ہاں is "yes", ہان is not a word. A normalizer that folds either of these into its lookalike reports a lower WER by destroying information.

And the same phrase is routinely written with or without an internal space. کیلئے and کے لیے are the same two words. WER counts whitespace-separated tokens, so that spelling difference alone scores as two errors. This is where most of the gap between published numbers lives.

Scoring

wer and cer take a string or a list of strings, normalize both sides under a named mode, and pool the result over the corpus.

from urdunorm import wer, cer, score

wer(references, hypotheses, mode="eval")
cer(references, hypotheses, mode="eval")

result = score(references, hypotheses, mode="eval")
result.rate, result.substitutions, result.deletions, result.insertions
str(result)  # 'WER 12.40% [eval]'

Tokenization is a whitespace split, and the corpus rate is total errors over total reference tokens, never the mean of per-utterance rates. Both are stated here because leaving them to the caller is how the same model ends up with two different published numbers.

There is no dependency on a WER library. Edit distance is thirty lines, and inheriting another package's default transform would reintroduce exactly the unstated normalization this one removes. The test suite checks agreement with jiwer where it is installed: WER, CER, and total edit distance match exactly. The substitution/deletion/insertion split does not always match, because edit distance has several equally optimal alignments and the split is a tie-breaking choice rather than a fact.

Modes

eval is the protocol this project reports under.

display is for human-facing transcripts. It fixes keyboard variants, removes tatweel and zero-width controls, and folds أ إ ٱ to plain ا. It keeps diacritics, punctuation, digits and case.

lums-compat reproduces the preprocessing of WER We Stand (COLING 2025), the only published Urdu ASR benchmark, so numbers can be compared against it. Their scoring code is published; one data file it loads is not, so the character whitelist here is a reconstruction, validated by feeding it to their own unmodified scorer and recovering their published table to a mean absolute error of 0.046 points. urdunorm.lums carries the faithful asymmetric reproduction, and its docstrings record what the paper leaves ambiguous and what we assumed.

Every number this project reports appears twice, once under eval and once under lums-compat. A single-protocol Urdu WER is not a comparable quantity.

What is preserved

These are phonemic in Urdu. No mode merges them, and the package refuses to import if a rule table would:

ھ do-chashmi heh, ں noon ghunna, ے barree yeh, ۓ, آ alef madda, ئ, ؤ, ۂ, and the retroflex and Perso-Urdu consonants ٹ ڈ ڑ پ چ ژ گ.

Protocols are data

A mode is a frozen dataclass, not a code path. That makes ablations cheap and makes the exact configuration reportable.

from urdunorm import EVAL, apply_protocol

no_repair = EVAL.with_(repair_spaces=False)
apply_protocol(text, no_repair)

urdunorm --show-protocol --mode eval

Rules live in reviewable data files

The space-repair lexicon and the Urdu number tables are tab-separated files under src/urdunorm/data/, not Python literals, so a native Urdu speaker can review them without reading code. See docs/data-review.md.

The number tables and the space-repair lexicon have not yet been reviewed by a native speaker. numbers_to_words is off in every mode until they are.

Dependencies

None at runtime, deliberately. urdunorm must stay installable by people who only want the normalizer, so it never imports torch, transformers, or datasets.

Citing

Not yet published. A DOI will be minted at the first tagged release.

License

Apache-2.0.