JT_script_normaliser

Language-aware text normalisation for the 22 scheduled Indian languages plus several foreign languages (English, Romanian, Portuguese, Vietnamese, Arabic).

It is built for ASR / speech pipelines: it canonicalises Unicode, strips or preserves punctuation and symbols depending on the mode, normalises digit scripts, and flags text that contains characters foreign to the chosen language.

Install

pip install JT_script_normaliser

Optional mojibake / encoding repair via ftfy:

pip install "JT_script_normaliser[ftfy]"

Quick start

from JT_script_normaliser import verbatim, verbatim_with_numbers, display

verbatim("नमस्ते, दुनिया!", "hindi")
# 'नमस्ते दुनिया'

verbatim_with_numbers("मेरे पास 5 आम", "hindi", number_script="native")
# 'मेरे पास ५ आम'

display("कीमत ₹1,00,000.50 है।", "hindi")
# 'कीमत ₹100000.50 है'

Modes

All modes apply a shared base: Unicode NFC, whitespace collapsing, removal of invisible formatting characters (BOM, bidi marks, ...), language-specific canonicalisation (e.g. Malayalam chillu folding), and a foreign-character gate (see below).

verbatim(text, language, *, remove_zero_width=True, use_ftfy=False)

Spoken text only — punctuation, symbols and numbers are stripped, meaning preserved. If any number or foreign character remains, the whole input is redacted (see below).

verbatim_with_numbers(text, language, *, number_script="latin", remove_zero_width=True, use_ftfy=False)

Like verbatim, but numbers are kept and unified to one digit script. A decimal point between two digits is kept (3.14); a grouping comma between two digits is removed (1,00,000 -> 100000).

display(text, language, *, number_script="latin", remove_zero_width=True, use_ftfy=False)

Readable output for downstream/ASR consumers. Keeps numbers, decimals and essential symbols while removing ordinary punctuation:

• Always kept: currency ($ ₹ €), ° © ™ ®, and % ‰ ‱ @ # &.
• Kept only next to a digit: math symbols (+ − × ÷ = ± < > | ~) and the affixes : - / * ^ (so 5:30, -9, 3/4, 2*3 are kept, but a-b, and/or, x = y are not).
• Vulgar fractions are folded to ASCII: ½ -> 1/2.

display("meet at 5:30, pay $5 + €3 = ¥8 - well-known", "english")
# 'meet at 5:30 pay $5 + €3 = ¥8 well known'

Options

• number_script — "latin" renders all digits as 0-9; "native" renders them in the language's script (e.g. Devanagari ५, Tamil ௫, Urdu ۵). Both Latin and native input digits are unified to the target.
• remove_zero_width — ZWJ/ZWNJ are removed by default. Set False to preserve them (e.g. to keep Telugu vibhakti rendering). Lexically meaningful sequences (Malayalam chillu, Bengali khanda-ta) are folded to their atomic forms first, so they survive either way.
• use_ftfy — when True, runs ftfy.fix_text first to repair mojibake (e.g. हि... -> हि...).

Foreign-character gate

After normalisation, any surviving character that is not a letter/mark of the chosen language — a foreign letter, or any digit in verbatim mode — causes the whole input to be returned as a redaction marker:

verbatim("नमस्ते hello", "hindi")
# 'REDACTED_FOREIGN_CHAR (h)'

verbatim_with_numbers("मेरे पास ৫ आम", "hindi")   # Bengali digit in Hindi
# 'REDACTED_FOREIGN_CHAR (৫)'

Supported languages

from JT_script_normaliser import available_languages
available_languages()

Indian: assamese, bengali, bodo, bhojpuri, chhattisgarhi, dogri, gujarati, hindi, kannada, kashmiri, konkani, maithili, malayalam, manipuri (Meetei Mayek), marathi, nepali, odia, punjabi, sanskrit, santali, sindhi, tamil, telugu, urdu.

Foreign: english, romanian, portuguese, vietnamese, arabic.

Common aliases and ISO codes are accepted (hi, bn, ta, ur, en, ...).

Per-language canonicalisation

Beyond the base pipeline, some languages get script-specific fixes:

• Malayalam — legacy chillu (consonant + virama + ZWJ) folded to atomic chillu (ൻ ർ ൽ ...), so ZWJ removal doesn't corrupt them.
• Bengali / Assamese — legacy khanda-ta folded to atomic ৎ.
• Urdu / Sindhi / Kashmiri / Arabic — presentation forms folded to base letters; tatweel (kashida) removed.
• Romanian — cedilla ş ţ folded to comma-below ș ț.

License

MIT