kyrgyz-morph-tagger 1.0.0

Morphological tagger for Kyrgyz: UPOS, XPOS, lemma and the full UD feature bundle

kyrgyz-morph-tagger

Morphological tagger for Kyrgyz: predicts UPOS, a Kyrgyz-specific XPOS, the lemma, and the full Universal Dependencies feature bundle for every token.

Kyrgyz is agglutinative — one word stacks number, possession, case, voice, negation, tense, mood and person onto a single stem, and a tag is only right if every feature in the bundle is right. This package is a PyTorch re-implementation of the UDPipe 2 architecture, trained on a manually annotated Kyrgyz UD treebank of 8,485 sentences (~98k tokens).

pip install kyrgyz-morph-tagger

from kyrgyz_morph_tagger import Tagger

tagger = Tagger.load()          # downloads the checkpoint once (~48 MB), then caches it

for sentence in tagger.tag("Мектептерибизден эрте келдик."):
    for token in sentence:
        print(token.form, token.lemma, token.upos, token.feats_string)

Мектептерибизден  Мектептер  NOUN   Case=Abl|Number=Plur|Poss=Yes
эрте              эрте       ADV    _
келдик            кел        VERB   Mood=Ind|Number=Plur|Person=1|Polarity=Pos|Tense=Past|Voice=Act
.                 .          PUNCT  _

The first Tagger.load() also pulls xlm-roberta-base (~1.1 GB) through transformers — the tagger reads frozen contextual embeddings from it. Both downloads are cached, so only the first run pays. A GPU is optional: the full 425-sentence test set tags in 12 s on CPU, 2 s on GPU, with identical output.

Usage

Input	Call
Raw text (splits sentences, tokenizes)	tagger.tag(text)
One raw sentence	tagger.tag_sentence(text)
Already-tokenized sentences	tagger.tag_pretokenized([["Мектеп", "жакшы", "."]])
A CoNLL-U file (keeps its tokenization and syntax)	tagger.tag_conllu("in.conllu", "out.conllu")

The tagger predicts morphology only — HEAD and DEPREL are left untouched.

ky-tag "Мектептерибизден келдик."             # print a table
ky-tag -f text.txt --format conllu            # raw text -> CoNLL-U
ky-tag --conllu in.conllu -o out.conllu       # fill an existing CoNLL-U file

Tagger.load()                        # released model, from the Hugging Face Hub
Tagger.load("path/to/checkpoint")    # a local directory you trained yourself
Tagger.load(revision="v1.0")         # pin a Hub revision for reproducibility
Tagger.load(device="cpu")

Accuracy

Held-out test set (425 sentences), exact match, gold tokenization:

	UPOS	XPOS	UFeats	AllTags	Lemmas
Released checkpoint (seed 42)	93.90	92.63	81.25	79.44	87.96
Mean ± std over 4 seeds	93.80 ±0.12	92.53 ±0.13	80.96 ±0.33	79.19 ±0.40	88.06 ±0.14

UFeats is exact-set match of the whole feature bundle; AllTags additionally requires UPOS and XPOS to be right. A single wrong feature invalidates the token — which is why UFeats (81) sits far below UPOS (94). That is the honest measure of full morphological analysis, and under the same metric three frontier LLMs given the same tagging scheme in the prompt reach only UFeats 58–60 zero-shot.

Tokenizer. tag() on raw text must tokenize first, which costs a little accuracy on top of the table above. The bundled rule-based tokenizer, scored against the corpus's gold tokenization on the same test set: token F1 99.27, 95.3 % of sentences tokenized exactly right. The sentence splitter recovers ~85 % of boundaries in running text; the ceiling is low for a real reason — 11 % of the corpus's sentences end with no final punctuation at all, and those boundaries cannot be recovered from the surface. If you already have tokens, use tag_pretokenized() and take the tokenizer out of the loop.

The tagset

Universal POS and features stay standard UD; Kyrgyz-specific detail lives in XPOS and in a few declared feature values.

• Six cases — Nom, Gen, Dat, Acc, Loc, Abl. The instrumental -менен is analysed as a postposition and the similative -дай as derivation, so neither is a case.
• Deverbal categories — the participle (atooch) and the verbal noun (ktooch) are UPOS=VERB with the appropriate VerbForm, keeping their fine label in XPOS.
• Ideophones and interjections — UPOS=INTJ, subtype in XPOS.
• AdjType=Rel|Qual, AdvType=Tim|Loc|Man|Deg, VerbForm=Part|Conv|Vnoun.

17 UPOS tags, 24 XPOS tags, 14 feature categories. The corpus passes the official UD validator (level 2) with zero errors.

Limitations

• Morphology only; no syntax.
• Trained on news, web and literary text — other domains will be weaker.
• Sentences longer than 512 XLM-R subwords lose contextual embeddings for the overflow. The package warns when this happens; split long sentences.

Development

pip install -e ".[dev]"
pytest -m "not model"     # fast: tokenizer and API surface
pytest                    # everything, incl. downloading and running the checkpoint

To run the tokenizer's full gold-corpus regression check, point KY_TREEBANK at a checkout of the treebank.

Licence

PolyForm Noncommercial 1.0.0 — free for research, teaching, personal projects and noncommercial organisations. Commercial use requires a separate licence; contact the licensor.

Be aware of what this means before you depend on it: PolyForm Noncommercial is not an OSI-approved open-source licence, and corporate licence scanners will treat it accordingly. If you need the tagger inside a product, ask — do not assume.

The training corpus is CC BY-SA 4.0 and is distributed separately; it is not shipped in this package, and this licence does not extend to it.

Related

• Model weights: https://huggingface.co/NurtilekAlibekov/kyrgyz-morph-tagger
• Corpus (CC BY-SA 4.0): https://github.com/AlibekovNurtilek/kyrgyz-ud-treebank
• Research code — training, experiments, the LLM comparison: https://github.com/AlibekovNurtilek/kyrgyz-morph-tagger