MedWER

MedWER is a reproducible, model-free evaluation protocol for medical ASR: a pinned deterministic text normalizer, a phrase-aware medical-WER, and a fixed, license-clean medical term list.

It is a protocol and a tool, not a new metric concept. Entity-weighted WER variants are well studied; what they typically depend on is an eval-time NER model or cloud API, which makes the denominator irreproducible. MedWER's denominator is a fixed, typed term list — the same strings for everyone, forever, no model in the loop.

Why

• Deterministic. Same inputs, same number, on any machine. No NER model, no API, no versioned black box in the metric.
• Pinned. Normalization is Whisper's EnglishTextNormalizer at an exact pinned version, locked by committed golden fixtures the test suite reproduces in CI — an upstream behavior change fails the build instead of silently moving numbers.
• License-clean. Every term-list source permits open commercial redistribution (see NOTICE); the list ships in the wheel.
• Phrase-honest. Multi-token terms ("insulin glargine") are matched greedily longest-first and scored as single atomic units, so a missed phrase counts as one miss, not free partial credit. The same applies to spelled-letter entries ("c h f") in a custom list; the bundled list carries none.

Install

pip install medwer

CLI

medwer score --refs refs.jsonl --hyps hyps.jsonl [--terms path/to/terms.txt]

Both files: one JSON object per line, {"id": ..., "text": ...}. Hypotheses join to references by id; the two id sets must match exactly, and any mismatch is an error. Output is a single JSON object (rates are fractions, null when the corresponding reference-word denominator is zero):

{"utts": 2620, "ref_words": 52576, "med_ref_words": 4310, "wer": 0.042, "medical_wer": 0.31}

Python API

from medwer import Normalizer, Scorer

scorer = Scorer()                      # bundled 19,373-entry medical term list
result = scorer.corpus(refs, hyps)     # {"wer": ..., "medical_wer": ..., ...}

norm = Normalizer()                    # stock Whisper normalizer, number folding on
norm("Give five hundred milligrams.")  # -> "give 500 milligrams"

The protocol

1. Normalize both sides with Whisper's EnglishTextNormalizer, unmodified — lowercasing, punctuation/diacritic removal, contraction and GB→US spelling standardization, and number-word folding ("five hundred" → "500"), so digit/word surface differences never score as errors. References are expected in written clinical form ("500 mg"). Normalizer(fold_numbers=False) (CLI: --no-fold-numbers) disables the folding step for corpora whose digit/word surface must be scored strictly.
2. WER: token-level Levenshtein over whitespace tokens.
3. Medical-WER: filter both sides to term-list content (greedy longest-match, phrases atomic), then WER on what remains.
4. Corpus aggregation: micro-average (total edits / total reference words); samples whose normalized reference is empty are skipped.

Scope: normalization is Latin-script (ASCII, Latin accents, NFKD-compatibility forms). Unit words are not folded ("milligrams" and "mg" are distinct tokens), so the written-form unit convention of the references is part of the protocol surface.

The term list

medwer/data/medical_terms.txt — 19,373 entries (11,742 multi-word) across drugs, diagnoses, symptoms, and injury mechanisms, stored in the protocol normalizer's output space (every entry is a fixed point of Normalizer()). Sources: Health Canada Drug Product Database + Canadian Clinical Drug Data Set (Open Government Licence – Canada) and ICD-10-CM (US public domain). Every entry derives from one of those sources; none is drafted or model-generated. Full provenance in NOTICE.

Regenerating fixtures

tools/gen_fixtures.py rewrites the golden fixtures; run it only when the pinned whisper-normalizer version is bumped or the fixture corpora change.

python tools/gen_fixtures.py

Every entry of the bundled term list is a fixed point of the protocol normalizer, since an entry the normalizer would rewrite can never match normalized text. A custom list (--terms) must hold the same property under the scorer's own normalizer; Scorer validates it at load and rejects the list otherwise. The test suite asserts it for the bundled list.

License

MIT (code). Term-list data: see NOTICE for per-source terms.