urdu-text-eval 1.0

Urdu text evaluation & benchmarking library with configurable orthographic normalization (WER, CER, chrF, BLEU).

urdu-text-eval

urdu-text-eval is a Python benchmarking library for Urdu text evaluation (OCR, ASR, transliteration, machine translation, LLM text generation, and text processing).

It computes standard NLP metrics (WER, CER, chrF, BLEU, Exact Match, Character Similarity) with configurable Urdu orthographic and taxonomy normalization (handling Arabic lookalike codepoints, zabar/zeer/pesh diacritics, punctuation, numbers, and presentation forms).

---

Key Features

• Standard Metrics: Word Error Rate (WER), Character Error Rate (CER), Word/Char Accuracy, chrF, BLEU, Exact Match, and Levenshtein Character Similarity.
• Urdu Taxonomy Normalization: Standardizes Arabic variants (ك → ک, ي → ی, ه → ہ), presentation forms (ﺍ, ﺎ, ﺐ), and combined characters (ا+ٓ → آ).
• Fine-Grained Diacritic Controls: Selective removal of اعراب (Zabar َ, Zeer ِ, Pesh ُ, Tanween ً ٌ ٍ, Shadda ّ, Sukun ْ, Maddah, and Hamza marks).
• Flexible Input Formats: Evaluates simple list-of-dicts [{"actual": "...", "pred": "..."}] with support for common key aliases (ref, target, gold, Urdu, hyp, prediction, output).
• Dual Reporting: Always reports both taxonomy-normalized (primary) and raw surface metrics side-by-side.

---

Installation

Install via pip:

pip install urdu-text-eval

(Dependencies: jiwer, sacrebleu, rapidfuzz)

---

Quick Start

from urdu_text_eval import evaluate

pairs = [
    {"actual": "کیا یہ ہے؟", "pred": "كيا يہ ہے؟"},
    {"actual": "آہ جو دل سے", "pred": "آه جو دل سے"},
]

result = evaluate(pairs)

print(result["summary"])
print(f"WER: {result['wer']:.4f} | CER: {result['cer']:.4f} | chrF: {result['chrf']:.4f}")

---

Output Structure

Calling evaluate(pairs, per_sample=True) returns a dictionary containing:

Key	Type	Description
wer	float	Primary Word Error Rate (0.0 = perfect match)
cer	float	Primary Character Error Rate
word_accuracy	float	1.0 - wer
char_accuracy	float	1.0 - cer
normalized_exact_match	float	Ratio of exact matches after normalization
chrf	float	Character n-gram F-score (sacrebleu)
bleu	float	Corpus BLEU score
mean_char_similarity	float	Mean normalized Levenshtein similarity
raw_wer / raw_cer	float	Metrics calculated on raw strings without normalization
raw_exact_match	float	Exact match ratio on raw strings
summary	str	Pre-formatted, printable evaluation report
samples	list	(Optional, if per_sample=True) List of dicts with per-row scores and normalized strings

---

Normalization & Customization

Urdu text often varies in orthography (e.g. Arabic vs. Urdu keyboards, presence of diacritics/اعراب, punctuation). You can control normalization behavior precisely:

1. Master On/Off

# Default: Normalization ON (Fair orthographic evaluation)
result = evaluate(pairs)

# Raw evaluation (No normalization applied)
result = evaluate(pairs, normalize=False)

2. Convenience Overrides

You can pass boolean flags directly to evaluate():

result = evaluate(
    pairs,
    taxonomy=True,            # Convert Arabic codepoints (ك/ي/ه) to Urdu (ک/ی/ہ)
    remove_diacritics=True,   # Remove all اعراب (zabar, zeer, pesh, tanween, etc.)
    remove_zabar=True,        # Remove zabar (َ) only
    remove_zeer=True,         # Remove zeer (ِ) only
    remove_pesh=True,         # Remove pesh (ُ) only
    remove_punctuation=True,  # Remove Urdu (؛،؟۔٪) and ASCII punctuation
    remove_digits=False,      # Remove ASCII and Urdu digits
)

3. Using NormalizeConfig or Presets

For reusability across benchmarks, configure a NormalizeConfig:

from urdu_text_eval import evaluate, NormalizeConfig

# Presets
cfg_default = NormalizeConfig.default()          # Taxonomy + diacritics + whitespace
cfg_raw     = NormalizeConfig.none()             # Raw string comparison
cfg_full    = NormalizeConfig.full()             # Everything on (including punctuation & digit stripping)
cfg_diac    = NormalizeConfig.diacritics_only()  # Only remove اعراب
cfg_punct   = NormalizeConfig.punctuation_only() # Only remove punctuation
cfg_tax     = NormalizeConfig.taxonomy_only()    # Only taxonomy mapping

# Custom Configuration
cfg = NormalizeConfig(
    enabled=True,
    taxonomy=True,             # Map Arabic/presentation characters
    combine_characters=True,   # Combine characters (ا + ٓ → آ)
    remove_zabar=True,         # Remove zabar
    remove_zeer=True,          # Remove zeer
    remove_pesh=True,          # Remove pesh
    remove_tanween=True,       # Remove tanween
    remove_shadda=True,        # Remove shadda
    remove_sukun=True,         # Remove sukun
    remove_punctuation=False,  # Keep punctuation
    remove_digits=False,       # Keep digits
)

result = evaluate(pairs, config=cfg)

---

Standalone Normalizer Function

You can also use the Urdu normalizer directly on individual text strings:

from urdu_text_eval import normalize_urdu, NormalizeConfig

# Default normalization
clean_text = normalize_urdu("كيا يہ ہے؟")
# Output: "کیا یہ ہے؟"

# Remove diacritics (zabar/zeer/pesh) only
clean_text = normalize_urdu("شیرِ پنجاب", remove_diacritics_all=True)
# Output: "شیر پنجاب"

# Punctuation removal
clean_text = normalize_urdu("سلام، دنیا!", config=NormalizeConfig.punctuation_only())
# Output: "سلام دنیا"

---

API Summary

from urdu_text_eval import (
    evaluate,           # Core benchmark function: evaluate([{"actual": "...", "pred": "..."}])
    NormalizeConfig,    # Configuration dataclass & presets for text normalization
    normalize_urdu,     # Single-string Urdu normalization function
    compute_metrics,    # Lower-level function: compute_metrics(references, hypotheses)
    format_metrics,     # Formats metrics dict into a human-readable text summary
    per_sample_errors,  # Returns list of per-item error breakdown dicts
)

---

License

Distributed under the MIT License.