indic-error-taxonomy

What kind of mistake did the OCR make? — not just how many.

Character Error Rate tells you a system is 12% wrong. It does not tell you whether that 12% is dropped vowel signs (a post-corrector could fix most of it) or the wrong script entirely (nothing can fix it — re-run the page). Those need opposite decisions and CER cannot distinguish them.

This library compares a reference transcription against a system's output and names every difference, across nine Indic scripts.

from indic_error_taxonomy import analyze

result = analyze(reference, hypothesis, "hi")
print(result.report())

WORD TIER — how much of the page the model attempted
  reference words        13
  word_correct                6   46.2%
  word_misread                3   23.1%
  word_deleted                4   30.8%
  attempted (correct+misread) 69.2%

CHARACTER TIER — what went wrong where it was attempting
  (high-confidence only)
  matra_missing               1
  matra_wrong                 1
  (31 further errors below high confidence — inspect before use)

LINE TIER — layout / reading order
  matched lines          3
  out of order           1
  kendall tau            +0.333

Install

pip install indic-error-taxonomy

One runtime dependency (rapidfuzz). No models, no downloads, no GPU.

Supported languages: hi mr ne (Devanagari) · bn as (Bengali) · ta (Tamil) · te (Telugu) · kn (Kannada) · ml (Malayalam) · gu (Gujarati) · pa (Gurmukhi) · or (Odia)

Three tiers, and the order to read them in

Read them in that order. A page read perfectly but emitted in the wrong sequence looks like catastrophic failure at the word and character tiers; only the line tier says otherwise. And roughly one in six character errors sits inside a word the system never actually attempted — the character label there describes where the aligner anchored, not a real mistake. The word tier is what tells you which is which.

result.lines      # ReadingOrderResult — layout order
result.words      # WordLevelResult    — 5 outcomes per word
result.chars      # CategorizedResult  — 25 reportable categories, per edit
result.attempted  # share of words the system demonstrably read

The tool tells you which of its own labels to trust

Not every category is equally reliable, and the library says so per instance rather than making you find out by hand.

for e in result.chars.errors:
    print(e.category, e.confidence, e.why)

matra_wrong      high   specific rule, uncontested, well-aligned
conjunct_broken  low    category agrees with a human only 20% of the time
                        (n=15); could also be other_deletion
base_confusion   low    the model wrote a different word here
                        ('பின்,' -> 'என்ன,')

result.summary()                      # high-confidence counts only
result.chars.summary_by_confidence()  # the full breakdown

Confidence combines four measured signals: how often that category agrees with a human annotator, whether competing rules could have claimed the same edit, alignment coherence, and whether the surrounding word was attempted.

The 25 character categories

They are a grid — which part of the writing system × what happened to it:

Plus five that sit outside the grid, which is itself informative: base_mark_confusion (crosses two rows), script_confusion and script_switch_failure (whole-text), hallucination (multiple units), and other_substitution / other_deletion / other_insertion (unclassified — not findings).

CATEGORIES holds a 26th name, reading_order_error. It never fires: plain Levenshtein has no transposition operation, so it cannot emit the anagram-shaped block the rule looks for. Transposition is measured at the line tier instead. The name is kept so the constant stays stable for callers that iterate it.

from indic_error_taxonomy.definitions import describe
print(describe("modifier_lost"))

Every category carries a plain description, the firing rule, a worked example, the normative standard, and what it is easily confused with. Definitions are executable: a test asserts every documented example really produces the category it illustrates.

Note: matra means a dependent vowel sign here, not "any mark". An anusvara (ं) is a modifier. Colloquial Hindi/Marathi uses मात्रा for both; this library does not. Six of eight labelling errors in our validation were this.

Grounded in Unicode, not in hand-typed tables

Categories that depend on a linguistic definition take it from the standard, and tests enforce it on every run:

• Character classes are pinned to Indic_Syllabic_Category. Adding this check found 24 misclassified characters, including the VOCALIC RR/LL letters in four languages and Bengali khanda ta.
• Conjunct clusters follow UAX #29 rule GB9c / Indic_Conjunct_Break. This is why conjunct_broken never fires for Tamil, Kannada or Gurmukhi — Unicode assigns those scripts no conjunct linker.

Validation

Injection sites are chosen from Unicode character names, independently of the library's own tables, so the two can disagree — which is the point.

Known limitations, stated plainly

• conjunct_broken is not reliable. 20% human agreement (n=15); ablation shows it competing with four other categories; it straddles akshara boundaries 10× more than average. Reported, never high-confidence, don't build on it.
• base_mark_confusion is ambiguous (58%, n=24). Partly the writing system's fault — a vowel sign is drawn attached to its consonant.
• hallucination's definition changed after validation; re-validate before quoting it.
• Single annotator, no human–human ceiling, so κ is a lower bound of unknown tightness. Four languages have zero hand-checked errors.
• The line tier is the least validated — unit-tested, but not checked at corpus scale on real multi-column scans.
• Word segmentation is whitespace-based. A missing space merges two words.
• Three deliberate divergences from Indic_Syllabic_Category (Gujarati Khoja transliteration nuktas), documented and pinned by test.

Also useful

from indic_error_taxonomy import categorize_errors, align_words
from indic_error_taxonomy.reading_order import analyze_reading_order

categorize_errors(ref, hyp, "hi")      # character tier alone
align_words(ref, hyp, "hi")            # word tier alone
analyze_reading_order(ref, hyp)        # line tier alone

Errors carry a content-addressed error_key, so hand labels survive future changes to the rules — label once, keep forever.

Development

pip install -e ".[dev]"
pytest indic_error_taxonomy/tests -q

License

MIT.