lemmauz 0.1.0

Deterministic FSM Uzbek lemmatizer

lemmauz

A deterministic finite-state-machine (FSM) lemmatizer for Uzbek, designed for speed and correctness without neural models or external dependencies.

Installation

pip install lemmauz

Or from source:

git clone https://github.com/uzumbek/lemmauz.git
cd lemmauz
pip install -e .

Usage

from lemmauz import fsm_lemmatize

# Bare stem (default)
fsm_lemmatize("kelishilganliklari")        # -> "kelishil"
fsm_lemmatize("o'qiyotganlarimizdan")      # -> "o'qi"
fsm_lemmatize("boshqarmoqda")              # -> "boshqar"

# With -moq infinitive suffix for verbal forms
fsm_lemmatize("ishlayotgan", add_moq=True)  # -> "ishlamoq"
fsm_lemmatize("yuraklarida", add_moq=True)  # -> "yurak" (noun, unchanged)

How It Works

lemmauz processes words through a fixed 7-layer pipeline that strips suffixes right-to-left:

Architecture

Each layer handles a specific morphological category. The FSM tries the longest suffix first (greedy match), checks structural validity, and moves to the next layer if no match is found.

Layer	Category	Examples
1	Word-final nominals	-lik, -siz, -roq → bilimli → bilim
2	Iterative nominal loop (CASE → POSS_LONG → PLURAL → DAGI → LIK_CHAIN → LOC → POSS_SHORT)	shaharlarimizdagi → shahar
3	Participles (with vowel-connective fallback)	kelayotgan → kel, tushunmaganligi → tushun
4	Voice suffixes (passive, reciprocal, frequentative)	ko'rishmoq → ko'rish
5	Direct VTM / VPN (verb tense/mood/person)	yaptim, moqda, di
6	Phonological restoration	lig→lik, g'→q
7	Mutation dictionary fallback	shahr→shahar, yurag→yurak

Key Design Decisions

• No normalization of digraphs: sh → ş and ch → ç are intentionally avoided. Uzbek orthography treats these as two-letter sequences, and normalizing them corrupts stems like do'st.
• Phoneme-aware MIN_STEM: The minimum stem length check operates on phonemes (after normalizing o' → single char, g' → single char), not raw bytes. This correctly counts multi-character sequences as one unit.
• Vowel-connective fallback in participle layer: When a vowel-leading participle suffix (e.g., ayotgan) matches but leaves an unnatural consonant cluster, the FSM tries the shorter variant (yotgan) to preserve what's likely a stem-final vowel. This handles cases like ishla + yotgan → surface form ishlayotgan.
• Mutation dictionary: A small lookup table (~20 entries) restores stems that undergo predictable phonological changes before consonant-initial suffixes:
  • Consonant mutation: k → g (yurak → yurag-), q → g' (quloq → qulog'-)
  • Vowel dropping: a/i → ∅ (shahar → shahr-, burun → burn-)
  • Applied as a post-processing fallback after all structural layers.

Benchmark Results

Main benchmark — 120 complex agglutinative chains

Deeply suffixed words with 5+ morphological layers:

Lemmatizer	Correct	Total	Accuracy
lemmauz	120	120	100.0%
UzMorphAnalyser	79	120	65.8%
UzbekLemma	45	120	37.5%

lemmauz outperforms existing open-source lemmatizers by a wide margin on complex chains, primarily because:

• UzMorphAnalyser leaves participle suffixes attached and corrupts apostrophe-containing stems (o' → o?)
• UzbekLemma over-applies -moq to nouns and fails to strip deep nominal chains

Exceptional words — 163 edge cases

Words with phonological mutations, ambiguous suffixes, and irregular patterns:

Lemmatizer	Correct	Total	Accuracy
UzMorphAnalyser	80	163	49.1%
lemmauz	73	163	44.8%
UzbekLemma	71	163	43.6%

All three lemmatizers struggle on this set. UzMorphAnalyser leads slightly, but at the cost of aggressive over-stripping Arabic loan roots (muhim → muh, iqlim → iql). lemmauz trades a small accuracy loss for correct preservation of known loans via a vowel-count guard. Failures fall into two categories (see "What Doesn't Work Yet"):

• Arabic loan ambiguity (-im): 90 misses — stem-final -im vs. possessive suffix
• Vowel-mutating stems: 75 misses — vowel dropping/assimilation before consonant-initial suffixes

What Doesn't Work Yet

Arabic loan words ending in -im (90 misses)

The hardest remaining class involves words ending in -im, which is ambiguous between:

1. Arabic loan roots where -im is part of the stem: tizim (system), iqlim (climate), muhim (important), taslim (surrender)
2. Possessive suffixes on native stems: o'g'il + im → o'g'lim (my son), yurak + im → yuragim (my heart), sabir + im → sabrim (my patience)

Current behavior uses a vowel-count guard to protect Arabic loans with ≤1 vowel in the remaining stem. This correctly preserves tizim, muhim, etc., but still produces incorrect results on words like:

• shonim → expected shon, got shonim (Arabic loan preserved, but should strip possessive)
• sabrim → expected sabr, got sabrim (possessive not stripped)

A full solution requires either a dictionary of known Arabic loans or morphological analysis that distinguishes stem-final -im from suffix -im.

Possessive suffixes on vowel-mutating stems (75 misses)

When possessive suffixes attach to stems ending in vowels, the stem-final vowel often drops or assimilates:

• o'g'il + ing → o'g'ling (expected stem o'g'il, got o'g'l)
• shahar + i → shahri (expected shahar, got shahr)

The mutation dictionary (layer 7) covers ~15 common stems, but a complete solution would need either:

• A larger mutation table covering all vowel-mutating stems
• Or structural rules that reinsert the dropped vowel when the FSM result ends in an unnatural consonant cluster

Short words and edge cases

Words like orqaside (expected orqa), kelma (expected kelmoq), and proper nouns (Azim) remain challenging due to their brevity or non-standard suffix patterns.

Project Structure

lemmauz/
├── lemmauz/
│   ├── __init__.py        # Public API: fsm_lemmatize()
│   └── lemmauz.py         # Core FSM implementation (single-file, no dependencies)
├── tests/
│   ├── test_without_moq.txt  # Benchmark dataset (120 pairs)
│   └── test_exceptional.txt  # Exceptional words dataset (163 pairs)
├── benchmark.py           # Fair comparison script against UzMorphAnalyser and UzbekLemma
└── pyproject.toml         # Package configuration

Running Benchmarks

pip install -e .
python benchmark.py

License

MIT