uzbek2turkic: Uzbek Normalizer for NLP Models

A robust, rule-based text normalization library designed to prepare and standardize Uzbek text for modern AI and NLP models (BERT, mDeBERTa, Llama, WordPiece, BPE).

The library intelligently maps Uzbek Latin sequences to the Common Turkic Alphabet (CTA). By unifying the Turkic language subword ecosystem, this package prevents tokenizers from hopelessly fragmenting multi-character digraphs (like o', g', sh, ch, and ng), drastically improving training efficiency and downstream NLP performance.

Key Features

• Algorithmic Apostrophe Unification: Flattens all erratic variations of apostrophes (ʻ, ‘, ’, ´, `) into the internationally recognized W3C modifier mark: ʼ (U+02BC).
• Digraph Compression: Securely converts sequential digraphs into single-character CTA equivalents (o' → ö, g' → ğ, sh → ş, ch → ç).
• Advanced NG Disambiguation: Uses a dual-layered algorithm (Grammatical Suffix Matrix + O(1) Dictionary Hash) to distinguish between the velar nasal (ñ) and dative/participle morphology boundaries (n + g).
• Case-Sensitivity: Fully preserves logical structural capitalization (Sh → Ş or SH → Ş).
• Reversible: Supports flawless denormalization (trc2uz) to restore model outputs back to standard standard Uzbek Latin.

Installation

Available on PyPI. Install via pip:

pip install uzbek2turkic

Quick Start

from uzbek2turkic.normalizer import UzbekNormalizer

# Initialize the normalizer
normalizer = UzbekNormalizer()

original_text = "O'g'il bola shahar tomon choy ichgani yo'l oldi."

# Forward Conversion (Original -> CTA)
normalized = normalizer.uz2trc(original_text)
print(normalized)
# Output: Öğil bola şahar tomon çoy içgani yöl oldi.

# Backward Conversion (CTA -> Original)
restored = normalizer.trc2uz(normalized)
print(restored)
# Output: O'g'il bola shahar tomon choy ichgani yo'l oldi.

Deep Dive: Semantic Disambiguation of NG (ñ vs n + g)

Converting ng to the nasal ñ is one of the most profound challenges in computational Uzbek text processing. n and g frequently appear sequentially as separate grammatical units (for example, a root ending in n appended by the dative suffix ga, or a passive verb ending in n appended by -gan).

A simplistic replacement script would erroneously destroy morphology by mapping jonga into joña or ishlangan into işlañan.

uzbek2turkic solves this dynamically by analyzing every single occurrence of ng independently, applying two powerful mechanisms:

1. Grammatical Suffix Concurrency Matrix

The algorithm detects localized combinations of N-suffixes (like -lan, -gan, -qan) colliding with G-suffixes (like -gach, -gan, -ga). It automatically isolates these morphological boundaries without relying on absolute dictionary lookups.

# Example of Grammatical Matrix working its magic:
text = "quvonganning ishlanganlariga"

print(normalizer.uz2trc(text))
# Output: quvonganniñ işlanganlariga

(Notice how the legitimate initial n+g breaks (quvon+gan, ishlan+gan) successfully bypass ñ conversion, while the true genitive suffix -ning at the end smoothly transitions into niñ within the exact same lexical word).

2. O(1) Root Dictionary Filtering

For native nouns that structurally end in n (like tun, kun, jon, vatan), the text is instantly validated against exceptions.txt.

# Example of Dictionary Exclusion:
text = "Tungi sovuq kunga va buguning bir qismiga ta'sir qildi."

print(normalizer.uz2trc(text))
# Output: Tungi sovuq kunga va buguniñ bir qismiga taʼsir qildi.

(You can also load your own root words by passing a custom dictionary during initialization: UzbekNormalizer(exceptions_file='/path/to/custom_roots.txt')).

W3C Typography and BPE Tokenizer Compliance

According to W3C Typography and International Unicode Standards, letters that act as an integral part of a word (like the Uzbek "tutuq") must not be written using standard punctuation apostrophes (', U+0027 or ’, U+2019). The international standard mandates the use of U+02BC (Modifier Letter Apostrophe — ʼ).

The Uzbek tutuq mark (used in words like ma'no, sur'at) is treated as a "punctuation boundary" by standard tokenizers (like HuggingFace BPE), unconditionally shattering the word into useless bytes (e.g., ["ma", "'", "no"]).

By converting standalone apostrophes into U+02BC (ʼ), which is officially classed as a Letter Modifier, models treat maʼno as a unified, physically unbroken linguistic token. This single fix dramatically reduces context window loss during LLM training.

License

MIT License. Open-source and ready for large-scale NLP pipelining.