voi-oiwer 0.1.3

Official scorer for the Voice of India (VoI) ASR benchmark — Orthographically Informed WER with a fast Rust core

voi_oiwer

The official scorer for OIWER (Orthographically Informed Word Error Rate) as used in the Voice of India (VoI) ASR benchmark (arXiv:2604.19151) — a large-scale evaluation of ASR across 15 Indian languages. This is the same implementation used to produce the benchmark's official results.

OIWER was introduced in prior work (see below) as an orthography-aware alternative to plain WER; VoI adapts it for the benchmark. Instead of penalizing legitimate spelling and transliteration variants, it scores a hypothesis against a lattice of accepted reference variations per word/phrase, so valid orthographic variants aren't counted as errors.

Normalization runs in Python (indicnlp-based); the edit-distance alignment over the lattice is implemented in Rust for speed.

Install

pip install voi_oiwer

Works on CPython 3.8+ (Linux, macOS, Windows). Prebuilt wheels — no Rust toolchain needed.

Usage

from voi_oiwer import oiwer

score, final_h, final_r, final_ops, (ins, dele, sub), \
    total_ref_words, metadata, std_refs = oiwer(
        hypothesis="नमस्ते दुनिया",
        reference_lists=[["नमस्ते"], ["दुनिया", "विश्व"]],   # lattice: variations per slot
        input_language="hindi",
    )
print(score)   # OIWER, as a percentage

Arguments

arg	type	description
hypothesis	str	the ASR output text
reference_lists	list[list[str]]	the reference lattice — one list of accepted variations per slot (a variation may span multiple words)
input_language	str	a language name, e.g. "hindi", "tamil", "urdu" (see list below)

Returns

A tuple:

(oiwer_score,                    # float — error percentage
 final_h,                        # list[str] — aligned hypothesis words
 final_r,                        # list[str] — aligned reference words (" " = insertion)
 final_ops,                      # list[str] — per slot: 'c' correct, 's' sub, 'd' del, 'i' ins
 (insertions, deletions, substitutions),
 total_reference_words,          # int
 final_metadata,                 # list — [lat_idx, var_idx, word_idx] per reference word (None for insertions)
 standardized_reference_lists)   # list[list[str]] — the normalized lattice used internally

Supported languages

assamese, bengali, bodo, dogri, gujarati, hindi, kannada, kashmiri, konkani, maithili, malayalam, manipuri, marathi, nepali, odia, punjabi, sanskrit, santali, sindhi, tamil, telugu, urdu, chattisgarhi, bhojpuri.

How it works

Scoring runs in two stages. First, text normalization (indicnlp-based) runs in Python — script-specific character handling, punctuation stripping, and Unicode normalization. Then the edit-distance alignment over the reference lattice runs in a compiled Rust core, which is where the speedup comes from. The normalized lattice used internally is returned as standardized_reference_lists so results are inspectable.

The metric, VoI's adaptation of OIWER, and the lattice-based reference design are described in full in the benchmark paper (arXiv:2604.19151).

Development

The Rust core is built with maturin:

pip install maturin
maturin develop --release   # compile the Rust core into your active venv
pytest -q                   # run the test suite

Citation

If you use this metric or the Voice of India benchmark, please cite:

@misc{bhogale2026voiceindialargescalebenchmark,
      title={Voice of India: A Large-Scale Benchmark for Real-World Speech Recognition in India}, 
      author={Kaushal Bhogale and Manas Dhir and Amritansh Walecha and Manmeet Kaur and Vanshika Chhabra and Aaditya Pareek and Hanuman Sidh and Mahima Manik and Sagar Jain and Bhaskar Singh and Utkarsh Singh and Tahir Javed and Shobhit Banga and Mitesh M. Khapra},
      year={2026},
      eprint={2604.19151},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2604.19151}, 
}

This metric builds on the original OIWER formulation:

@misc{bhogale2026orthographicallyinformedevaluationspeechrecognition,
      title={Towards Orthographically-Informed Evaluation of Speech Recognition Systems for Indian Languages}, 
      author={Kaushal Santosh Bhogale and Tahir Javed and Greeshma Susan John and Dhruv Rathi and Akshayasree Padmanaban and Niharika Parasa and Mitesh M. Khapra},
      year={2026},
      eprint={2603.00941},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2603.00941}, 
}

License

MIT © JoshTalks