hindi-normalize 0.1.0

Convert Hindi number words to digits and normalize Devanagari ASR/OCR text

hindi-normalize

Convert Hindi number words to digits and normalize Devanagari text from ASR, OCR, or copy-paste. Handles the encoding variants, punctuation quirks, and number-word spellings that break search, equality checks, tokenization, and downstream extraction. Zero runtime dependencies — pure standard library.

I built this because Sakhi — an offline medical Hindi voice-to-form tool meant for ASHAs— kept reinventing the same regex passes against Whisper output: number words that never became digits, ५ and 5 treated as different characters, | where a danda । belonged, the same word encoded two different ways. This is that pass, extracted and generalized.

Install

pip install hindi-normalize

Quick start

from hindi_normalize import normalize_transcript

normalize_transcript("आपका BP एक सौ दस बटा सत्तर है, वजन अट्ठावन kg")
# 'आपका BP 110/70 है, वजन 58 kg'

normalize_transcript runs the whole pipeline. The individual transforms are also exported and usable on their own.

Numbers

Hindi number words → digits, tolerant of ASR spelling variants (पाँच/पांच/पाच, सत्तर/सतर). Adjacent unrelated numbers are not summed — दो तीन दिन ("two-three days") stays 2 3 दिन, never 5.

from hindi_normalize import convert_numbers, parse_hindi_number

convert_numbers("एक सौ दस बटा सत्तर")   # '110 बटा 70'
parse_hindi_number("नौ सौ निन्यानवे")    # 999

Devanagari

normalize_devanagari folds the character-level variants OCR/ASR/copy-paste introduce, following the operations the Indic NLP Library established as standard.

from hindi_normalize import normalize_devanagari

normalize_devanagari("वजन ५८ किलो | ठीक है")   # 'वजन 58 किलो। ठीक है'

Default (lossless, on):

• Invisibles — strip ZWJ/ZWNJ/ZWSP, BOM, soft hyphen, word joiner; no-break space → space
• NFC — unify nukta encodings (क़ as one codepoint vs क + nukta) and matra order
• Digits — Devanagari digits (५) → ASCII (5)
• Danda — | → ।, || → ॥, collapse runs, drop space before a danda

Opt-in (lossy, off):

• remove_nukta — क़ → क, ज़ → ज
• nasals — fold nasal clusters to anusvara (हिन्दी → हिंदी)
• chandrabindu — ँ → ं
• visarga — ASCII colon after a Devanagari letter → visarga (दु:ख → दुःख)

Unlike a naive unicodedata-category filter, nothing here strips the Unicode Mark category, so matras and the virama survive — the failure mode that silently deletes vowel signs from Indic text.

Terms

Map spoken/mis-recognized terms to canonical forms. Ships a maternal/child-health (ASHA home-visit) dictionary; pass your own for any domain.

from hindi_normalize import replace_terms, MEDICAL_TERMS

replace_terms("बीबी एक सौ दस बटा सत्तर", MEDICAL_TERMS)   # 'BP एक सौ दस / सत्तर'
replace_terms("क ख", {"क": "K", "ख": "KH"})              # 'K KH'

Repetition

Collapse the back-to-back stutter recognisers emit.

from hindi_normalize import collapse_repetition

collapse_repetition("ठीकठीकठीकठीक है")   # 'ठीक है'

For runaway tail loops in long LLM generations (repeated sentences/paragraphs) and for general invisible/typographic cleanup of non-Devanagari text, compose with llmclean.

API

Function	Purpose
normalize_transcript(text, *, terms, numbers, devanagari, repetition, sentence_breaks)	Full pipeline
convert_numbers(text)	Hindi number words → digits
parse_hindi_number(text)	Parse one number expression → int
normalize_devanagari(text, *, ...)	Character-level normalization
strip_zero_width(text)	Remove invisibles; NBSP → space
devanagari_digits_to_ascii(text) / ascii_digits_to_devanagari(text)	Digit conversion
normalize_danda(text)	Danda punctuation
fold_nasals(text, *, chandrabindu)	Nasal clusters → anusvara
replace_terms(text, terms)	Term dictionary replacement
collapse_repetition(text, *, min_repeats)	Collapse inline stutter

Every function is guarded against non-string input and returns the input unchanged rather than raising.

License

MIT