gurmukhifix

Safe, evidence-gated OCR correction for Gurmukhi and other Indic scripts. Reverses the systematic Unicode-order and diacritic errors that OCR makes on Gurmukhi (Punjabi), Hindi, and Devanagari — and is built so it can never silently corrupt correct text, including Gurbani.

🌐 Live demo & docs → — paste OCR text and watch it become clean Unicode, entirely in your browser.

OCR engine  →  gurmukhifix  →  corrected Unicode + report + metadata
(Tesseract / Surya / Gemini / Google Vision / …)

What it does — and what it refuses to do

OCR engines convert an image to characters but have no linguistic rules. On Gurmukhi they make a small set of systematic, predictable errors:

• Sihari misordering — the vowel sign ਿ is drawn before its base consonant but must be encoded after it. ਸਿਮਰਿ is frequently emitted as ਿਸਮਰਿ.
• I-matra misordering (Devanagari) — the same problem for ि.
• Nukta / dependent-vowel order — e.g. ਖ਼ਾਸ emitted as ਖਾ਼ਸ.
• Character confusions where OCR picks a valid-but-wrong look-alike letter.

gurmukhifix corrects these after OCR. The defining principle is safety:

Every automatic change must clear an evidence gate. A verbatim scripture (Gurbani) word is locked and never altered automatically. Any other edit must either strictly improve script-validity or turn a non-word into a known dictionary word — a blind swap between two valid characters is refused, not guessed at. Re-ordering the same characters into canonical order is always safe.

This matters because the target material includes sacred and heritage text, where turning one valid word into a different valid word is the worst possible failure. The guarantee is not a hope — it is property-tested across every supported script and against the whole Gurbani corpus.

Script support

"Hindi" and "Devanagari" share the Devanagari script; the two config names exist so Hindi-specific rules can layer on top of the shared Devanagari base.

Install

pip install gurmukhifix

Python 3.10+. No system OCR libraries required — gurmukhifix processes OCR output, so your OCR engine is a peer, not a runtime dependency.

First correction in 30 seconds

gurmukhifix demo --lang gurmukhi

Sample [gurmukhi]: gurmukhi.json
  OCR input : ਗੁਰਮੁਖਿ ਜਾਪੈ ਸਬਦਿ ਿਲਵ ਲਾਇ
  Corrected : ਗੁਰਮੁਖਿ ਜਾਪੈ ਸਬਦਿ ਲਿਵ ਲਾਇ
  1 fix(es) : 'ਿਲ'→'ਲਿ' (sihari_order_fix)

Quickstart on your own scans

Stock Tesseract emits TSV and hOCR (it has no JSON renderer) — feed either straight in:

# 1. OCR your page with any engine. With Tesseract:
tesseract my_gurmukhi_page.png out --oem 1 --psm 6 tsv

# 2. Correct it (format auto-detected):
gurmukhifix correct --input out.tsv --lang gurmukhi --output ./results

# 3. Read the results
cat ./results/corrected_text.txt

Python API

from gurmukhifix import process_document

# Feed OCR output from any supported engine: a dict, a file path, a plain string,
# or a list of word dicts.
result = process_document({"words": [{"text": "ਿਸਮਰ", "conf": 72}]}, "gurmukhi")

print(result["corrected_text"])   # ਸਿਮਰ
print(result["correction_report"])

Supported OCR formats

gurmukhifix is OCR-engine-agnostic. process_document (and gurmukhifix correct) auto-detect the format; run gurmukhifix formats to list them:

Because transformer OCR (Surya, Gemini, TrOCR) now beats Tesseract on Indic and Nastaliq scripts, this layer keeps gurmukhifix useful no matter which engine wins.

Each word record is {"text": str, "conf": 0–100, "bbox": [x, y, w, h], "alternatives": [...]}. Confidence bands: ≥ 85 passes through unchanged, 60–85 is fully corrected, < 60 is flagged for human review.

Accuracy

The benchmark runs on real verbatim Gurbani — 300 lines of Sri Guru Granth Sahib Ji (tests/ground_truth/gurbani_sggs.txt) — into which the most common Tesseract Gurmukhi error (word-initial sihari misordering) is injected. It reports Character Error Rate of the corrupted text vs gurmukhifix's output, both against the truth:

python -m tests.benchmark   # reproduce

Honest scope: this measures the engine's ability to reverse a documented, targeted error class on real Gurbani — not end-to-end accuracy on a scanned manuscript, which needs a human-labelled scan corpus. Drop paired {"language","ocr","truth"} JSON into tests/ground_truth/ and it is folded in automatically. The clean lines double as a corruption check: their corrected CER must stay 0.0000.

How it works

The lexicon is derived from the Shabad OS database; see gurmukhifix/data/LEXICON.md for provenance.

Output artifacts

Each gurmukhifix correct / batch run writes:

Per-corpus learning

Confirmed corrections are stored in SQLite; once a pattern is confirmed 10+ times it is promoted. Promoted corrections are still run through the evidence gate — they apply in the context they were confirmed and never override the scripture lock.

gurmukhifix review --flagged ./results/flagged.json --corrections ./my.db
gurmukhifix report --corrections ./my.db --lang gurmukhi

Development

pip install -e ".[dev]"
pytest                      # full suite incl. property-based no-corruption tests
python -m tests.benchmark   # accuracy benchmark
ruff check . && mypy gurmukhifix   # lint + types

Every bug fix must ship with a regression test using the exact input that exposed it. See CONTRIBUTING.md.

Releasing

Publishes to PyPI from a GitHub Release via Trusted Publishing — see PUBLISHING.md.

License

MIT — free and open source for any use, including commercial. See LICENSE.

Acknowledgements

• Shabad OS and BaniDB — the Gurbani corpus behind the lexicon and benchmark.
• Tesseract, Surya, and the transformer-OCR community — the engines whose output this tool corrects.
• The Unicode Consortium — Gurmukhi, Devanagari, and Arabic block documentation.