uztextnorm 0.3.0

Uzbek Text Normalization Library for TTS and NLP applications

UzTextNorm

Uzbek Text Normalization Library for TTS and NLP applications.

Features

• Unicode Normalization: Normalizes text to NFC form
• Cyrillic to Latin Transliteration: Automatic conversion of Uzbek Cyrillic to Latin script
• Number Expansion: Converts numbers to Uzbek words
  • Integers: 123 → bir yuz yigirma uch
  • Decimals: 3.14 → uch butun o'n to'rt
  • Fractions: 3/4 → to'rtdan uch
  • Currency: 10,000 so'm → o'n ming soʻm
  • Percentages: 15% → o'n besh foiz
• Punctuation Normalization: Standardizes punctuation and whitespace
• URL/Email Preservation: Protects URLs and emails from normalization
• Custom Rules: Support for custom pre/post processing rules

Installation

pip install uztextnorm

Usage

Basic Usage

from uztextnorm import UzbekTextNormalizer

# Create a normalizer instance
normalizer = UzbekTextNormalizer()

# Normalize text
text = "Narx: 10,000 so'm; 15% oshdi"
result = normalizer(text)
print(result)
# Output: "narx: o'n ming soʻm; o'n besh foiz oshdi"

Cyrillic to Latin Transliteration

text = "Ўзбекистон Республикаси"
result = normalizer(text)
print(result)
# Output: "o'zbekiston respublikasi"

Number Expansion

# Integers
normalizer("123")  # "bir yuz yigirma uch"

# Decimals
normalizer("3.14")  # "uch butun o'n to'rt"

# Fractions
normalizer("3/4")  # "to'rtdan uch"
normalizer("1/2")  # "yarim"
normalizer("2 3/4")  # "ikki butun to'rtdan uch"

# Currency
normalizer("10,000 so'm")  # "o'n ming soʻm"
normalizer("2,500 $")  # "ikki ming besh yuz dollar"

# Percentages
normalizer("15%")  # "o'n besh foiz"

Configuration Options

normalizer = UzbekTextNormalizer(
    to_lower=True,  # Convert to lowercase (default: True)
    transliterate_cyrillic=True,  # Transliterate Cyrillic (default: True)
    custom_pre_rules=[],  # Custom pre-processing functions
    custom_post_rules=[]  # Custom post-processing functions
)

Custom Processing Rules

def remove_hashtags(text: str) -> str:
    return text.replace('#', '')

def add_period(text: str) -> str:
    if not text.endswith('.'):
        return text + '.'
    return text

normalizer = UzbekTextNormalizer(
    custom_pre_rules=[remove_hashtags],
    custom_post_rules=[add_period]
)

Module Structure

uztextnorm/
├── __init__.py          # Package initialization
├── constants.py         # Constants and language data
├── utils.py            # Utility functions
├── transliterator.py   # Cyrillic-to-Latin conversion
├── tokenizer.py        # Token shielding (URLs, emails)
├── numbers.py          # Number expansion
└── normalizer.py       # Main normalizer class

Requirements

• Python >= 3.8

License

MIT License

Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

Continuous Integration & Publishing

This project ships with a GitLab CI pipeline configured in .gitlab-ci.yml. The pipeline runs on pushed branches and merge requests, executes the test-suite, and, when a git tag is pushed, builds release artifacts and uploads them to both PyPI and the GitLab Package Registry.

Required CI/CD variables

Add the following variables in the project settings under Settings → CI/CD → Variables:

• PYPI_API_TOKEN – a PyPI token with the ability to publish the project (__token__ style token). The publish job is skipped if this variable is not present.

No extra configuration is required for the GitLab Package Registry publishing job; it uses the built-in CI_JOB_TOKEN. Ensure that the token’s default permissions are kept (read_registry/write_registry).

Release workflow

1. Ensure tests pass locally: python -m pip install .[dev] && pytest.
2. Bump the version in pyproject.toml.
3. Commit the change and create an annotated tag matching the new version, e.g. git tag -a v0.2.0 -m "Release v0.2.0".
4. Push the branch and tag: git push origin main --tags (replace main with the relevant branch).

The CI pipeline triggered by the tag will build the package once and publish it to both registries.