agwer 0.1.0

Agentic WER: WER/MER/WIL/WIP/CER plus RIR, HER, and n-best oracles for evaluating ASR error correctors and voice agents. Self-contained, single dependency (RapidFuzz).

AgWER: Agent-oriented Word Error Rate

agwer is a simple and fast Python package to evaluate speech recognition and voice agents. It is self-contained: one dependency (RapidFuzz, C++ edit distance), a 40 KB wheel, sub-20 ms import.

It supports the classic ASR similarity measures and the agentic ones:

1. word error rate (WER) — also match error rate (MER), word information lost/preserved (WIL/WIP)
2. character error rate (CER)
3. Recoverable Information Ratio (RIR, the paper's ρ)
4. Harmful Edit Rate (HER)

The agentic measures (3–4) evaluate systems that read $n$-best hypotheses and decide when to edit and when to abstain (LLM error correctors, dictation agents):

measure	question it answers
RIR: Recoverable Information Ratio (ρ)	How much of the 1-best→oracle gap did the correction close? ρ>1 beats the n-best oracle; ρ<0 is the damage regime.
HER: Harmful Edit Rate	Of the edits actually made, what fraction broke a correct token? Isolates over-correction.
o_nb / o_cp	The two HyPoradise oracles: best single hypothesis (reranking bound) / best token recombination (correction bound).

Installation

With uv:

uv add agwer             # as a project dependency
uv pip install agwer     # into the active environment

Or with pip (Python >= 3.9):

pip install agwer

Usage

The simplest use-case is computing the word error rate of a dictated utterance:

from agwer import wer

reference  = "please schedule the quarterly budget review for tuesday march twenty first at nine thirty and invite the design team"
hypothesis = "please schedule the quarterly budget review for tuesday march twenty first at nine thirty and invite the desire team"

wer(reference, hypothesis)   # 0.0526 — one broken word in nineteen

All measures accept a single string or a list of strings; lists are pooled corpus-level (total errors / total reference words), and mer, wil, wip, cer work the same way:

import agwer

refs = ["send the revised contract to the legal team before the board meeting on friday afternoon",
        "remind me to pick up the prescription from the pharmacy after the dentist appointment"]
hyps = ["send the revised contract to the legal team before the bored meeting on friday afternoon",
        "remind me to pick up the prescription from the pharmacy after the dentist appointment"]

agwer.wer(refs, hyps)     # 0.0345 — corpus WER
agwer.cer(refs, hyps)     # 0.0116 — corpus CER

Evaluating a corrector / voice agent

With $n$-best input, one call computes everything. Long dictations are where correction matters — a single homophone in a 21-word command ("forth floor") is the difference between the right room and a support ticket:

refs = ["move my three thirty meeting with the product team to thursday and book the large conference room on the fourth floor"]
nbest = [[   # nbest[i][0] is the ASR 1-best
    "move my three thirty meeting with the product team to thursday and book the large conference room on the forth floor",
    "move my three thirty meeting with the product team to thursday and book the large conference room on the fourth floor",
    "move my three thirty meeting with the protect team to thursday and book the large conference room on the forth floor",
]]
corrected = ["move my three thirty meeting with the product team to thursday and book the large conference room on the fourth floor"]

out = agwer.evaluate(refs, corrected, nbest=nbest)
out.wer_1best            # 0.0476 -> the raw ASR broke one word
out.wer_corrected        # 0.0    -> the agent fixed it
out.rir                  # 1.0    -> closed the 1-best-to-oracle gap exactly
out.her                  # 0.0    -> and broke nothing while doing it
out.wer_oracle           # o_nb: best single hypothesis
out.wer_compositional    # o_cp: best token recombination (o_cp <= o_nb)

HER comes in two granularities — her_granularity="utterance" (default; the accounting behind the paper's reported values) and "token" (the formal per-edit definition). A sentence where the corrector fixes one token and breaks another is neutral at utterance granularity but helpful=1, harmful=1 at token granularity; report which one you used.

Normalization

Normalization is the main reason WER numbers are incomparable across papers. Every entry point takes normalize= (any Callable[[str], str]); agwer ships the standards:

A dictation agent that says amounts out loud looks 87% wrong against the written form — until you normalize:

ref = "the invoice total came to $1,250.75 after the 15% discount was applied on march 3rd"
hyp = ("the invoice total came to one thousand two hundred fifty dollars "
       "and seventy five cents after the fifteen percent discount was applied on march third")

agwer.wer(ref, hyp)                                          # 0.867 (!)
agwer.wer(ref, hyp, normalize=agwer.EnglishTextNormalizer())  # 0.0

normalizer	what
None (general measures' default)	score strings exactly as given
agwer.default_normalize (agentic default)	conservative: lowercase, keep apostrophes, strip other punctuation
BasicTextNormalizer()	language-agnostic: symbols, brackets, optional diacritic folding
EnglishTextNormalizer()	the Whisper English normalizer (numbers→digits, currency, contractions, British→American); cached=True adds an LRU for agent loops

The Whisper normalizers are vendored (MIT, © 2022 OpenAI, attribution included) with behavior pinned byte-identical to the original by golden tests. Report which normalizer you used; it is part of the metric.

CLI

agwer results.jsonl                 # {"reference","corrected","nbest"} per line
agwer results.jsonl --json --her-granularity token

Performance

Batched RapidFuzz hot path; benchmarks ship in the package:

python -m agwer.bench

~0.2 s for a full agentic evaluation (WER×3 + both oracles + RIR + HER) of a 10,000-utterance × 5-best corpus on a laptop; classic corpus WER in ~20 ms.

Compatibility & reproducibility

Measure semantics match jiwer (validated bit-identical on 600-corpus goldens, pinned in tests/), and the default agentic settings reproduce the Voice Memory paper's published evaluation (golden-pinned in tests/test_paper_reference.py).

Citation

If you use RIR/ρ or HER, please cite the Voice Memory paper (Exploring Voice Memory for Agentic Speech Recognition, under review, 2026 — citation entry will be updated at camera-ready) and this package.

License

Apache-2.0. Vendored Whisper normalizers: MIT (see src/agwer/normalizers/LICENSE_WHISPER).