indic-itn (v0.2.4)

Inverse Text Normalization (ITN) for Multilingual Indian Languages (Hindi, Tamil, Telugu, Kannada, etc.).

indic-itn converts spoken-form ASR (Automatic Speech Recognition) transcriptions into normalized written representations across phone numbers, numbers, dates, times, currency, decimals, percentages, ordinals, and OTPs while strictly preserving all surrounding context words and punctuation.

Architectural Highlights & Guarantees

• Zero Context Deletion: Operating on span-based substitutions (text[:start] + normalized_span + text[end:]), ensuring surrounding text before and after numeric expressions is never lost or corrupted.
• Language Plugin Architecture: Decouples core normalization engine logic from language-specific vocabulary. Adding a new language (e.g. Malayalam ml) requires creating a language plugin directory without touching the core engine.
• Specialized Phone Number Entity: Phone numbers are processed as unformatted digit sequences (9876543210), supporting spoken native/English digits, repeated digit phrases (double/triple), optional country codes (+91), plus, and zero.
• Deterministic Entity Priority Resolution: Candidate spans are classified and resolved in strict priority order (URL > Email > Phone > OTP > Date > Time > Currency > Decimal > Percentage > Ordinal > Number) to prevent overlapping span corruption.
• Code-Switching Support: Seamlessly handles mixed Indic script and English spoken digit expressions (e.g. "mera phone number hai nine eight seven six...").
• 100% Backward Compatibility: Full support for legacy entry points (HindiITN, TeluguITN, KannadaITN, TamilITN, IndicITN, Token, Entity, load_resource).

Architecture Overview

Raw ASR Spoken Text
      │
      ▼
indic_itn.core.tokenizer (IndicTokenizer)
      │
      ▼
indic_itn.core.entity_detector (Span candidate detection & priority resolution)
      ├── Phone Entity Handler
      ├── Date Entity Handler
      ├── Time Entity Handler
      ├── Currency Entity Handler
      ├── Decimal Entity Handler
      ├── Percentage Entity Handler
      ├── Ordinal Entity Handler
      ├── OTP Entity Handler
      └── General Number Entity Handler
      │
      ▼
indic_itn.languages.<lang> (Language Lexical Parser & Semantic Mapper)
      │
      ▼
indic_itn.entities.<entity> (Canonical Renderer)
      │
      ▼
indic_itn.core.span_replacer (Right-to-Left Safe Substring Replacer)
      │
      ▼
indic_itn.normalization.postprocess (Whitespace & Punctuation Cleanup)
      │
      ▼
Final Normalized Written Text

Installation

pip install indic-itn

Or install locally in editable mode for development:

pip install -e ".[dev]"

Usage

1. Functional API (normalize_text)

from indic_itn import normalize_text

# Phone number with surrounding words
print(normalize_text("call me on nine eight seven six five four three two one zero tomorrow", language="hi"))
# Output: "call me on 9876543210 tomorrow"

# Currency
print(normalize_text("I have five hundred rupees in my account", language="hi"))
# Output: "I have 500 rupees in my account"

# Time & Date
print(normalize_text("meeting is at five thirty pm", language="hi"))
# Output: "meeting is at 5:30 pm"

2. Object-Oriented Orchestrators

from indic_itn import HindiITN, TamilITN, TeluguITN, KannadaITN

# Hindi
hi = HindiITN()
print(hi.normalize("मेरा नंबर नौ आठ सात छह पाँच चार तीन दो एक शून्य है"))
# Output: "मेरा नंबर 9876543210 है"

# Tamil
ta = TamilITN()
print(ta.normalize("என் போன் நம்பர் ஒன்பது எட்டு ஏழு ஆறு ஐந்து நான்கு மூன்று இரண்டு ஒன்று பூஜ்யம்"))
# Output: "என் போன் நம்பர் 9876543210"

# Telugu
te = TeluguITN()
print(te.normalize("నా ఫోన్ నంబర్ తొమ్మిది ఎనిమిది ఏడు ఆరు ఐదు నాలుగు మూడు రెండు ఒకటి సున్నా ఉంది"))
# Output: "నా ఫోన్ నంబర్ 9876543210 ఉంది"

# Kannada
kn = KannadaITN()
print(kn.normalize("ನನ್ನ ಬಳಿ ಐದು ನೂರು ರೂಪಾಯಿ ಇದೆ"))
# Output: "ನನ್ನ ಬಳಿ ₹500 ಇದೆ"

3. Debug & Entity Metadata Mode

from indic_itn import IndicITNEngine

engine = IndicITNEngine(lang="hi")
debug_info = engine.normalize("call nine eight seven six five four three two one zero at five pm", return_entities=True)

print(debug_info)
# Output:
# {
#   "original_text": "call nine eight seven six five four three two one zero at five pm",
#   "normalized_text": "call 9876543210 at 5:00 pm",
#   "detected_spans": [
#     {"start": 5, "end": 53, "original": "nine eight...", "normalized": "9876543210", "entity_type": "phone"},
#     {"start": 57, "end": 64, "original": "five pm", "normalized": "5:00 pm", "entity_type": "time"}
#   ]
# }

How to Add a New Language

Adding support for a 5th Indian language (e.g. Malayalam ml) requires zero modifications to the core engine:

Step 1: Create Resource Directory

Add JSON files in src/indic_itn/resources/ml/:

• numbers.json (digits, numbers, tens, hundreds, multipliers)
• keywords.json (script_range, script_digits, currency, time, phone, otp, decimal, percentage)
• temporal.json (months, date_words, weekdays)
• ordinals.json (ordinal words mapping)

Step 2: Implement Language Plugin Class

Create src/indic_itn/languages/malayalam/normalizer.py:

from indic_itn.languages.base import BaseLanguage

class MalayalamLanguage(BaseLanguage):
    def __init__(self) -> None:
        super().__init__(lang_code="ml")

Step 3: Register Language

Register the language plugin dynamically or in default registry:

from indic_itn import register_language, normalize_text
from indic_itn.languages.malayalam.normalizer import MalayalamLanguage

register_language("ml", MalayalamLanguage)

# Use immediately
print(normalize_text("spoken text in malayalam", language="ml"))

How to Add a New Entity Type

1. Create a handler class in src/indic_itn/entities/my_entity.py inheriting from BaseEntityHandler.
2. Implement entity_type property and parse(span, lang) method.
3. Register handler in IndicITNEngine.entity_handlers.

Quality Metrics & Benchmark Dataset

indic-itn includes an extensive 400-example benchmark test dataset (tests/fixtures/benchmark_dataset.json) covering 100 test samples each for Hindi, Tamil, Telugu, and Kannada across numbers, phone numbers, dates, times, currency, decimals, percentages, and mixed language contexts.

Running Benchmark Suite

pytest tests/benchmark/test_benchmark.py -s

Testing & Quality Assurance

Running Full Test Suite

pytest --cov=indic_itn --cov-report=term-missing

Running Static Type Checker & Linter

mypy src
ruff check src tests