indic-itn (v0.2.7)

Inverse Text Normalization (ITN) for Multilingual Indian Languages (Hindi, Tamil, Telugu, Kannada, etc.).

indic-itn converts spoken-form ASR (Automatic Speech Recognition) transcriptions into normalized written representations across phone numbers, numbers, dates, times, currency, decimals, percentages, ordinals, and OTPs while strictly preserving all surrounding context words and punctuation.

Architectural Highlights & Guarantees

• Zero Context Deletion: Operating on span-based substitutions (text[:start] + normalized_span + text[end:]), ensuring surrounding text before and after numeric expressions is never lost or corrupted.
• Language Plugin Architecture: Decouples core normalization engine logic from language-specific vocabulary. Adding a new language (e.g. Malayalam ml) requires creating a language plugin directory without touching the core engine.
• Specialized Phone Number Entity: Phone numbers are processed as unformatted digit sequences (9876543210), supporting spoken native/English digits, repeated digit phrases (double/triple), optional country codes (+91), plus, and zero.
• Deterministic Entity Priority Resolution: Candidate spans are classified and resolved in strict priority order (URL > Email > Phone > OTP > Date > Time > Currency > Decimal > Percentage > Ordinal > Number) to prevent overlapping span corruption.
• Code-Switching Support: Seamlessly handles mixed Indic script and English spoken digit expressions (e.g. "mera phone number hai nine eight seven six...").
• 100% Backward Compatibility: Full support for legacy entry points (HindiITN, TeluguITN, KannadaITN, TamilITN, IndicITN, Token, Entity, load_resource).

Architecture Overview

Raw ASR Spoken Text
      │
      ▼
indic_itn.core.tokenizer (IndicTokenizer)
      │
      ▼
indic_itn.core.entity_detector (Span candidate detection & priority resolution)
      ├── Phone Entity Handler
      ├── Date Entity Handler
      ├── Time Entity Handler
      ├── Currency Entity Handler
      ├── Decimal Entity Handler
      ├── Percentage Entity Handler
      ├── Ordinal Entity Handler
      ├── OTP Entity Handler
      └── General Number Entity Handler
      │
      ▼
indic_itn.languages.<lang> (Language Lexical Parser & Semantic Mapper)
      │
      ▼
indic_itn.entities.<entity> (Canonical Renderer)
      │
      ▼
indic_itn.core.span_replacer (Right-to-Left Safe Substring Replacer)
      │
      ▼
indic_itn.normalization.postprocess (Whitespace & Punctuation Cleanup)
      │
      ▼
Final Normalized Written Text

Installation

pip install indic-itn

Or install locally in editable mode for development:

pip install -e ".[dev]"

Usage

1. Auto-Detection & Auto-Normalization (Recommended)

indic-itn can automatically detect the input language/script using deterministic Unicode script analysis with zero external dependencies.

from indic_itn import auto_normalize_text, detect_language, get_supported_languages

# Check supported languages
print(get_supported_languages())
# Output: ['hi', 'kn', 'ta', 'te']

# 1. Direct auto-normalization across any supported language
print(auto_normalize_text("मेरा नंबर नौ आठ सात छह पाँच चार तीन दो एक शून्य है"))
# Output: "मेरा नंबर 9876543210 है"

print(auto_normalize_text("ನನ್ನ ಬಳಿ ಐದು ನೂರು ರೂಪಾಯಿ ಇದೆ"))
# Output: "ನನ್ನ ಬಳಿ ₹500 ಇದೆ"

print(auto_normalize_text("రెండు వందల రూపాయలు"))
# Output: "₹200"

print(auto_normalize_text("ஐந்து நூறு ரூபாய்"))
# Output: "₹500"

# 2. Standalone language detection (returns 2-letter ISO code)
lang = detect_language("ನೂರು ರೂಪಾಯಿ")
# Output: 'kn'

# 3. Detailed diagnostics metadata
details = detect_language("ನನ್ನ phone number ಒಂಬತ್ತು", return_details=True)
print(details.language)       # 'kn'
print(details.script_name)    # 'Kannada'
print(details.confidence)     # 0.7273
print(details.script_counts)  # {'kn': 16, 'latin': 6}
print(details.is_supported)   # True
print(details.is_reliable)    # True

2. Functional API (normalize_text)

You can provide an explicit language code ("hi", "kn", "te", "ta") or pass language="auto":

from indic_itn import normalize_text

# Explicit language
print(normalize_text("call me on nine eight seven six five four three two one zero tomorrow", language="hi"))
# Output: "call me on 9876543210 tomorrow"

# Auto language detection
print(normalize_text("ನೂರು ರೂಪಾಯಿ", language="auto"))
# Output: "₹100"

# Currency
print(normalize_text("I have five hundred rupees in my account", language="hi"))
# Output: "I have 500 rupees in my account"

# Time & Date
print(normalize_text("meeting is at five thirty pm", language="hi"))
# Output: "meeting is at 5:30 pm"

3. Object-Oriented Orchestrators

from indic_itn import IndicITN, HindiITN, TamilITN, TeluguITN, KannadaITN

# Unified Auto Orchestrator
itn = IndicITN(lang="auto")
print(itn.normalize("पांच सौ रुपये")) # Output: "₹500"
print(itn.normalize("ನೂರು ರೂಪಾಯಿ"))  # Output: "₹100"

# Language-Specific Orchestrators
hi = HindiITN()
print(hi.normalize("मेरा नंबर नौ आठ सात छह पाँच चार तीन दो एक शून्य है"))
# Output: "मेरा नंबर 9876543210 है"

ta = TamilITN()
print(ta.normalize("என் போன் நம்பர் ஒன்பது எட்டு ஏழு ஆறு ஐந்து நான்கு மூன்று இரண்டு ஒன்று பூஜ்யம்"))
# Output: "என் போன் நம்பர் 9876543210"

te = TeluguITN()
print(te.normalize("నా ఫోన్ నంబర్ తొమ్మిది ఎనిమిది ఏడు ఆరు ఐదు నాలుగు మూడు రెండు ఒకటి సున్నా ఉంది"))
# Output: "నా ఫోన్ నంబర్ 9876543210 ఉంది"

kn = KannadaITN()
print(kn.normalize("ನನ್ನ ಬಳಿ ಐದು ನೂರು ರೂಪಾಯಿ ಇದೆ"))
# Output: "ನನ್ನ ಬಳಿ ₹500 ಇದೆ"

3. Debug & Entity Metadata Mode

from indic_itn import IndicITNEngine

engine = IndicITNEngine(lang="hi")
debug_info = engine.normalize("call nine eight seven six five four three two one zero at five pm", return_entities=True)

print(debug_info)
# Output:
# {
#   "original_text": "call nine eight seven six five four three two one zero at five pm",
#   "normalized_text": "call 9876543210 at 5:00 pm",
#   "detected_spans": [
#     {"start": 5, "end": 53, "original": "nine eight...", "normalized": "9876543210", "entity_type": "phone"},
#     {"start": 57, "end": 64, "original": "five pm", "normalized": "5:00 pm", "entity_type": "time"}
#   ]
# }

How to Add a New Language

Adding support for a 5th Indian language (e.g. Malayalam ml) requires zero modifications to the core engine:

Step 1: Create Resource Directory

Add JSON files in src/indic_itn/resources/ml/:

• numbers.json (digits, numbers, tens, hundreds, multipliers)
• keywords.json (script_range, script_digits, currency, time, phone, otp, decimal, percentage)
• temporal.json (months, date_words, weekdays)
• ordinals.json (ordinal words mapping)

Step 2: Implement Language Plugin Class

Create src/indic_itn/languages/malayalam/normalizer.py:

from indic_itn.languages.base import BaseLanguage

class MalayalamLanguage(BaseLanguage):
    def __init__(self) -> None:
        super().__init__(lang_code="ml")

Step 3: Register Language

Register the language plugin dynamically or in default registry:

from indic_itn import register_language, normalize_text
from indic_itn.languages.malayalam.normalizer import MalayalamLanguage

register_language("ml", MalayalamLanguage)

# Use immediately
print(normalize_text("spoken text in malayalam", language="ml"))

How to Add a New Entity Type

1. Create a handler class in src/indic_itn/entities/my_entity.py inheriting from BaseEntityHandler.
2. Implement entity_type property and parse(span, lang) method.
3. Register handler in IndicITNEngine.entity_handlers.

Quality Metrics & Benchmark Dataset

indic-itn includes an extensive 400-example benchmark test dataset (tests/fixtures/benchmark_dataset.json) covering 100 test samples each for Hindi, Tamil, Telugu, and Kannada across numbers, phone numbers, dates, times, currency, decimals, percentages, and mixed language contexts.

Running Benchmark Suite

pytest tests/benchmark/test_benchmark.py -s

Testing & Quality Assurance

Running Full Test Suite

pytest --cov=indic_itn --cov-report=term-missing

Running Static Type Checker & Linter

mypy src
ruff check src tests