uztextnorm 0.3.1

Uzbek Text Normalization Library for TTS and NLP applications

UzTextNorm

Uzbek Text Normalization Library for TTS and NLP applications.

Features

• Unicode Normalization: Normalizes text to NFC form
• Cyrillic to Latin Transliteration: Automatic conversion of Uzbek Cyrillic to Latin script
• Number Expansion: Converts numbers to Uzbek words
  • Integers: 123 → bir yuz yigirma uch
  • Decimals: 3.14 → uch butun o'n to'rt
  • Fractions: 3/4 → to'rtdan uch
  • Currency: 10,000 so'm → o'n ming soʻm
  • Percentages: 15% → o'n besh foiz
• Punctuation Normalization: Standardizes punctuation and whitespace
• URL/Email Preservation: Protects URLs and emails from normalization
• Custom Rules: Support for custom pre/post processing rules

Installation

pip install uztextnorm

Usage

Basic Usage

from uztextnorm import UzbekTextNormalizer

# Create a normalizer instance
normalizer = UzbekTextNormalizer()

# Normalize text
text = "Narx: 10,000 so'm; 15% oshdi"
result = normalizer(text)
print(result)
# Output: "narx: o'n ming soʻm; o'n besh foiz oshdi"

Cyrillic to Latin Transliteration

text = "Ўзбекистон Республикаси"
result = normalizer(text)
print(result)
# Output: "o'zbekiston respublikasi"

Number Expansion

# Integers
normalizer("123")  # "bir yuz yigirma uch"

# Decimals
normalizer("3.14")  # "uch butun o'n to'rt"

# Fractions
normalizer("3/4")  # "to'rtdan uch"
normalizer("1/2")  # "yarim"
normalizer("2 3/4")  # "ikki butun to'rtdan uch"

# Currency
normalizer("10,000 so'm")  # "o'n ming soʻm"
normalizer("2,500 $")  # "ikki ming besh yuz dollar"

# Percentages
normalizer("15%")  # "o'n besh foiz"

Configuration Options

normalizer = UzbekTextNormalizer(
    to_lower=True,  # Convert to lowercase (default: True)
    transliterate_cyrillic=True,  # Transliterate Cyrillic (default: True)
    custom_pre_rules=[],  # Custom pre-processing functions
    custom_post_rules=[]  # Custom post-processing functions
)

Custom Processing Rules

def remove_hashtags(text: str) -> str:
    return text.replace('#', '')

def add_period(text: str) -> str:
    if not text.endswith('.'):
        return text + '.'
    return text

normalizer = UzbekTextNormalizer(
    custom_pre_rules=[remove_hashtags],
    custom_post_rules=[add_period]
)

Module Structure

uztextnorm/
├── __init__.py          # Package initialization
├── constants.py         # Constants and language data
├── utils.py            # Utility functions
├── transliterator.py   # Cyrillic-to-Latin conversion
├── tokenizer.py        # Token shielding (URLs, emails)
├── numbers.py          # Number expansion
└── normalizer.py       # Main normalizer class

Requirements

• Python >= 3.10

Development

Clone the repository and install the development dependencies:

git clone https://gitlab.com/adizbek/uztextnorm.git
cd uztextnorm
pip install -e ".[dev]"

Run the test suite:

pytest

Format and lint:

black src tests
ruff check src tests

Build the distribution artifacts:

python -m build

Links

• PyPI: https://pypi.org/project/uztextnorm/
• Repository: https://gitlab.com/adizbek/uztextnorm
• Issues: https://gitlab.com/adizbek/uztextnorm/-/issues
• Changelog: CHANGELOG.md

Author

Adizbek Ergashev — [e-mail]

License

MIT License — see LICENSE for details.

Contributing

Contributions are welcome! Please feel free to submit a Pull Request.