turkicnlp 0.1.9

NLP toolkit for Turkic languages

TurkicNLP

NLP toolkit for 20+ Turkic languages — a pip-installable Python library inspired by Stanza, with adaptations for the low-resource, morphologically rich Turkic language family.

Maintained by Sherzod Hakimov

Citation

If you use TurkicNLP in your research, please cite:

@misc{hakimov2026turkicnlpnlptoolkit,
      title={TurkicNLP: An NLP Toolkit for Turkic Languages}, 
      author={Sherzod Hakimov},
      year={2026},
      eprint={2602.19174},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2602.19174}, 
}

Arxiv preprint

PDF

Features

• 24 Turkic languages from Turkish to Sakha, Kazakh to Uyghur
• Script-aware from the ground up — Latin, Cyrillic, Perso-Arabic, Old Turkic Runic
• Automatic script detection and bidirectional transliteration
• Apertium FST morphology for ~20 Turkic languages via Python-native hfst bindings (no system install)
• Stanza/UD integration — pretrained tokenization, POS tagging, lemmatization, dependency parsing, and NER via Stanza models trained on Universal Dependencies treebanks
• NLLB embeddings + translation backend — sentence/document vectors and MT via NLLB-200
• Multiple backends — choose between rule-based, Apertium FST, or Stanza neural backends per processor
• License isolation — library is Apache-2.0; Apertium GPL-3.0 data downloaded separately
• Stanza-compatible API — Pipeline, Document, Sentence, Word

Installation

Requirements: Python 3.9, 3.10, 3.11, or 3.12

pip install turkicnlp                    # core — tokenization, rule-based processing, CoNLL-U I/O
pip install "turkicnlp[hfst]"           # + Apertium FST morphology (Linux and macOS only)
pip install "turkicnlp[stanza]"         # + Stanza neural models (tokenize, POS, lemma, depparse, NER)
pip install "turkicnlp[translation]"    # + NLLB embeddings and machine translation
pip install "turkicnlp[all]"            # everything above (Linux and macOS only)
pip install "turkicnlp[dev]"            # development tools (pytest, black, ruff, mypy)

Platform compatibility

Installation tests run nightly across all combinations of OS, Python version, and install extra (see CI workflow).

Extra	Ubuntu 22.04 / 24.04	macOS 14 / 15	Windows 2025
base	✅ 3.9 – 3.12	✅ 3.9 – 3.12	✅ 3.9 – 3.12
[hfst]	✅ 3.9 – 3.12	✅ 3.9 – 3.12	❌ not available
[stanza]	✅ 3.9 – 3.12	✅ 3.9 – 3.12	✅ 3.9 – 3.12
[translation]	✅ 3.9 – 3.12	✅ 3.9 – 3.12	✅ 3.9 – 3.12
[all]	✅ 3.9 – 3.12	✅ 3.9 – 3.12	❌ not available

Windows users: the hfst Python package has no published wheels for Python 3.7 or later on Windows — this is an upstream limitation with no current workaround. All features except Apertium FST morphology work normally on Windows; use turkicnlp[stanza] or turkicnlp[translation] instead. If you need Apertium FST morphology on Windows, the recommended approach is Windows Subsystem for Linux (WSL), where hfst installs normally.

Quick Start

import turkicnlp

# Download models for a language
turkicnlp.download("kaz")

# Build a pipeline
nlp = turkicnlp.Pipeline("kaz", processors=["tokenize", "pos", "lemma", "ner", "depparse"])

# Process text
doc = nlp("Мен мектепке бардым")

# Access annotations
for sentence in doc.sentences:
    for word in sentence.words:
        print(f"{word.text}\t{word.lemma}\t{word.upos}\t{word.feats}")

# Export to CoNLL-U
print(doc.to_conllu())

Embeddings (NLLB)

import math
import turkicnlp

turkicnlp.download("tur", processors=["embeddings"])
nlp = turkicnlp.Pipeline("tur", processors=["embeddings"])

doc1 = nlp("Bugün hava çok güzel ve parkta yürüyüş yaptım.")
doc2 = nlp("Parkta yürüyüş yapmak bugün çok keyifliydi çünkü hava güzeldi.")

def cosine_similarity(a, b):
    dot = sum(x * y for x, y in zip(a, b))
    norm_a = math.sqrt(sum(x * x for x in a))
    norm_b = math.sqrt(sum(y * y for y in b))
    return dot / (norm_a * norm_b)

print(len(doc1.embedding), len(doc2.embedding))
print(f"cosine = {cosine_similarity(doc1.embedding, doc2.embedding):.4f}")
print(doc1._processor_log)  # ['embeddings:nllb']

Machine Translation (NLLB)

import turkicnlp

# Downloads once into ~/.turkicnlp/models/huggingface/facebook--nllb-200-distilled-600M
turkicnlp.download("tur", processors=["translate"])

nlp = turkicnlp.Pipeline(
    "tur",
    processors=["translate"],
    translate_tgt_lang="eng",
)

doc = nlp("Bugün hava çok güzel ve parkta yürüyüş yaptım.")
print(doc.translation)
print(doc._processor_log)  # ['translate:nllb']

translate_tgt_lang accepts either ISO-639-3 ("eng", "tuk", "kaz") or explicit Flores-200 codes ("eng_Latn", "kaz_Cyrl").

Using the Stanza Backend

from turkicnlp.processors.stanza_backend import (
    StanzaTokenizer, StanzaPOSTagger, StanzaLemmatizer, StanzaNERProcessor, StanzaDepParser
)
from turkicnlp.models.document import Document

# Models are downloaded automatically on first use
doc = Document(text="Merhaba dünya.", lang="tur")

for Proc in [StanzaTokenizer, StanzaPOSTagger, StanzaLemmatizer, StanzaNERProcessor, StanzaDepParser]:
    proc = Proc(lang="tur")
    proc.load()
    doc = proc.process(doc)

for word in doc.words:
    print(f"{word.text:12} {word.upos:6} {word.lemma:12} head={word.head} {word.deprel}")

# Export to CoNLL-U
print(doc.to_conllu())

Mixed Backends

from turkicnlp.processors.tokenizer import RegexTokenizer
from turkicnlp.processors.stanza_backend import StanzaPOSTagger, StanzaNERProcessor, StanzaDepParser
from turkicnlp.models.document import Document

doc = Document(text="Мен мектепке бардым.", lang="kaz")

# Rule-based tokenizer + Stanza POS/parsing (pretokenized mode)
tokenizer = RegexTokenizer(lang="kaz")
tokenizer.load()
doc = tokenizer.process(doc)

pos = StanzaPOSTagger(lang="kaz")
pos.load()
doc = pos.process(doc)

ner = StanzaNERProcessor(lang="kaz")
ner.load()
doc = ner.process(doc)

parser = StanzaDepParser(lang="kaz")
parser.load()
doc = parser.process(doc)

Multi-Script Support

# Kazakh — auto-detects Cyrillic vs Latin
doc = nlp("Мен мектепке бардым")    # Cyrillic
doc = nlp("Men mektepke bardym")     # Latin

# Explicit script selection
nlp_cyrl = turkicnlp.Pipeline("kaz", script="Cyrl")
nlp_latn = turkicnlp.Pipeline("kaz", script="Latn")

# Transliteration bridge — run Cyrillic model on Latin input
nlp = turkicnlp.Pipeline("kaz", script="Latn", transliterate_to="Cyrl")

Uyghur (Perso-Arabic)

nlp_ug = turkicnlp.Pipeline("uig", script="Arab")
doc = nlp_ug("مەن مەكتەپكە باردىم")

Transliteration

The Transliterator class converts text between scripts for any supported language pair:

from turkicnlp.scripts import Script
from turkicnlp.scripts.transliterator import Transliterator

# Kazakh Cyrillic → Latin (2021 official alphabet)
t = Transliterator("kaz", Script.CYRILLIC, Script.LATIN)
print(t.transliterate("Қазақстан Республикасы"))
# → Qazaqstan Respublıkasy

# Uzbek Latin → Cyrillic
t = Transliterator("uzb", Script.LATIN, Script.CYRILLIC)
print(t.transliterate("O'zbekiston Respublikasi"))
# → Ўзбекистон Республикаси

# Uyghur Perso-Arabic → Latin (ULY)
t = Transliterator("uig", Script.PERSO_ARABIC, Script.LATIN)
print(t.transliterate("مەكتەپ"))
# → mektep

# Azerbaijani Latin → Cyrillic
t = Transliterator("aze", Script.LATIN, Script.CYRILLIC)
print(t.transliterate("Azərbaycan"))
# → Азәрбайҹан

# Turkmen Latin → Cyrillic
t = Transliterator("tuk", Script.LATIN, Script.CYRILLIC)
print(t.transliterate("Türkmenistan"))
# → Түркменистан

# Tatar Cyrillic → Latin (Zamanälif)
t = Transliterator("tat", Script.CYRILLIC, Script.LATIN)
print(t.transliterate("Татарстан Республикасы"))
# → Tatarstan Respublikası

Old Turkic Runic Script

TurkicNLP supports transliteration of Old Turkic runic inscriptions (Orkhon-Yenisei script, Unicode block U+10C00–U+10C4F) to Latin:

from turkicnlp.scripts import Script
from turkicnlp.scripts.transliterator import Transliterator

t = Transliterator("otk", Script.OLD_TURKIC_RUNIC, Script.LATIN)

# Individual runic characters
print(t.transliterate("\U00010C34\U00010C07\U00010C2F\U00010C19"))
# → törk  (Türk)

# The transliterator maps each runic character to its standard
# Turkological Latin equivalent, handling both Orkhon and Yenisei
# variant forms (e.g., separate glyphs for consonants with
# back vs. front vowel contexts).

Supported Languages and Components

Geographic distribution of Turkic languages (source: Wikimedia Commons)

The table below shows all supported languages with their available scripts and processor status.

Legend:

Symbol	Backend	Description
■	Rule-based	Regex tokenizer, abbreviation lists
◆	Apertium FST	Finite-state morphology via hfst (GPL-3.0, downloaded separately)
●	Stanza/UD	Neural models trained on Universal Dependencies treebanks
▲	Custom Stanza	Custom-trained Stanza models hosted by turkic-nlp
★	NLLB	Embeddings and machine translation via NLLB-200
○	Planned	Implementation planned
—		Not available yet

Oghuz Branch

Language	Code	Script(s)	Tokenize	Morph	POS	Lemma	DepParse	NER	Embed	Translate
Turkish	tur	Latn	■ ●	◆	●	●	●	●	★	★
Azerbaijani	aze	Latn, Cyrl	■▲	◆	▲	▲	▲	—	★	★
Iranian Azerbaijani	azb	Arab	○	—	—	—	—	—	★	★
Turkmen	tuk	Latn, Cyrl	■▲	◆	▲	▲	▲	—	★	★
Gagauz	gag	Latn	■	◆	—	—	—	—	—	—

Kipchak Branch

Language	Code	Script(s)	Tokenize	Morph	POS	Lemma	DepParse	NER	Embed	Translate
Kazakh	kaz	Cyrl, Latn	■ ●	◆	●	●	●	●	★	★
Kyrgyz	kir	Cyrl	■ ●	◆	●	●	●	—	★	★
Tatar	tat	Cyrl, Latn	■▲	◆	▲	▲	▲	—	★	★
Bashkir	bak	Cyrl	■▲	◆	▲	▲	▲	—	★	★
Crimean Tatar	crh	Latn, Cyrl	■	◆	—	—	—	—	★	★
Karakalpak	kaa	Latn, Cyrl	■	◆	—	—	—	—	—	—
Nogai	nog	Cyrl	■	◆	—	—	—	—	—	—
Kumyk	kum	Cyrl	■	◆	—	—	—	—	—	—
Karachay-Balkar	krc	Cyrl	■	◆	—	—	—	—	—	—

Karluk Branch

Language	Code	Script(s)	Tokenize	Morph	POS	Lemma	DepParse	NER	Embed	Translate
Uzbek	uzb	Latn, Cyrl	■ ▲	◆	▲	▲	▲	—	★	★
Uyghur	uig	Arab, Latn	○ ●	◆	●	●	●	—	★	★

Siberian Branch

Language	Code	Script(s)	Tokenize	Morph	POS	Lemma	DepParse	NER	Embed	Translate
Sakha (Yakut)	sah	Cyrl	■	◆	—	—	—	—	—	—
Altai	alt	Cyrl	■	◆	—	—	—	—	—	—
Tuvan	tyv	Cyrl	■	◆	—	—	—	—	—	—
Khakas	kjh	Cyrl	■	◆	—	—	—	—	—	—

Oghur Branch

Language	Code	Script(s)	Tokenize	Morph	POS	Lemma	DepParse	NER	Embed	Translate
Chuvash	chv	Cyrl	■	◆	—	—	—	—	—	—

Arghu Branch

Language	Code	Script(s)	Tokenize	Morph	POS	Lemma	DepParse	NER	Embed	Translate
Khalaj	klj	Arab	—	—	—	—	—	—	—	—

Historical Languages

Language	Code	Script(s)	Tokenize	Morph	POS	Lemma	DepParse	NER	Embed	Translate
Ottoman Turkish	ota	Arab, Latn	—	—	—	—	—	—	—	—
Old Turkish	otk	Orkh, Latn	○	—	—	—	—	—	—	—

Stanza/UD Model Details

The Stanza backend provides neural models trained on Universal Dependencies treebanks. Official Stanza models (●) are downloaded via Stanza's model hub. Custom-trained models (▲) are hosted at turkic-nlp/trained-stanza-models and downloaded automatically.

Language	Stanza Code	Type	UD Treebank(s)	Stanza Processors	NER Dataset
Turkish	tr	●	IMST (default), BOUN, FrameNet, KeNet, ATIS, Penn, Tourism	tokenize, mwt, pos, lemma, depparse, ner	Starlang NER
Kazakh	kk	●	KTB	tokenize, mwt, pos, lemma, depparse, ner	KazNERD
Uyghur	ug	●	UDT	tokenize, pos, lemma, depparse	—
Kyrgyz	ky	●	KTMU	tokenize, pos, lemma, depparse	—
Uzbek	uz	▲	UzUDT	tokenize, pos, lemma, depparse	—
Uzbek	uz	▲	UzUDT	tokenize, pos, lemma, depparse	—
Turkmen	tk	▲	Tk-TUD	tokenize, pos, lemma, depparse	—
Azerbaijani	az	▲	Az-TUD	tokenize, pos, lemma, depparse	—
Tatar	ta	▲	Ta-TUD	tokenize, pos, lemma, depparse	—
Bashkir	ba	▲	Ba-TUD	tokenize, pos, lemma, depparse	—

Transliteration Support

Bidirectional script conversion is available for all multi-script languages. The transliterator uses a greedy longest-match algorithm with per-language mapping tables.

Language	Direction	Scripts	Standard
Kazakh	↔ Bidirectional	Cyrillic ↔ Latin	2021 official Latin alphabet
Uzbek	↔ Bidirectional	Cyrillic ↔ Latin	1995 official Latin alphabet
Azerbaijani	↔ Bidirectional	Cyrillic ↔ Latin	1991 official Latin alphabet
Tatar	↔ Bidirectional	Cyrillic ↔ Latin	Zamanälif
Turkmen	↔ Bidirectional	Cyrillic ↔ Latin	1993 official Latin alphabet
Karakalpak	↔ Bidirectional	Cyrillic ↔ Latin	2016 Latin alphabet
Crimean Tatar	↔ Bidirectional	Cyrillic ↔ Latin	Standard Crimean Tatar Latin
Uyghur	↔ Bidirectional	Perso-Arabic ↔ Latin	Uyghur Latin Yéziqi (ULY)
Ottoman Turkish	→ One-way	Latin → Perso-Arabic	Academic transcription
Old Turkic	→ One-way	Runic → Latin	Turkological convention

Apertium FST Quality Levels

Level	Description	Languages
Production	>90% coverage on news text	Turkish, Kazakh, Tatar
Stable	Good coverage, actively maintained	Azerbaijani, Kyrgyz, Uzbek
Beta	Reasonable coverage, some gaps	Turkmen, Bashkir, Uyghur, Crimean Tatar, Chuvash
Prototype	Limited coverage, experimental	Gagauz, Sakha, Karakalpak, Nogai, Kumyk, Karachay-Balkar, Altai, Tuvan, Khakas

Model Catalog and Apertium Downloads

TurkicNLP uses a model catalog to define download sources per language/script/processor. The catalog lives in:

• turkicnlp/resources/catalog.json (packaged default)
• Remote override: ModelRegistry.CATALOG_URL (or TURKICNLP_CATALOG_URL)

For each language, the catalog stores the Apertium source repo and the expected FST script. When turkicnlp.download() is called, it reads the catalog and downloads precompiled .hfst binaries from the url fields. If a language has no URL configured, download will fail with a clear error until the catalog is populated with hosted binaries (for example, a turkic-nlp/apertium-data releases repository).

Download folder

All models and resources are downloaded into this folder: ~/.turkicnlp.

Architecture

TurkicNLP follows Stanza's modular pipeline design:

Pipeline("tur", processors=["tokenize", "morph", "pos", "ner", "depparse"])
    │
    ▼
  Document ─── text: "Ben okula vardım"
    │
    ├── script_detect    → script = "Latn"
    ├── tokenize         → sentences, tokens, words
    ├── morph (Apertium) → lemma, pos, feats (via HFST)
    ├── pos (neural)     → refined UPOS, XPOS, feats
    ├── ner (neural)     → BIO tags and entity spans
    └── depparse         → head, deprel
    │
    ▼
  Document ─── annotated with all layers

Pipeline("azb", processors=["embeddings", "translate"], translate_tgt_lang="eng")
    │
    ▼
  Document ─── text: "من کتاب اوخویورام"
    │
    ├── script_detect          → script = "Arab"
    ├── embeddings (NLLB)      → sentence/document vectors
    └── translate (NLLB)       → sentence/document translation
           (src resolved from FLORES map: azb -> azb_Arab,
            tgt resolved from ISO-3: eng -> eng_Latn)
    │
    ▼
  Document ─── annotated with all layers

Key Abstractions

• Document → Sentence → Token → Word hierarchy (maps to CoNLL-U)
• Processor ABC with PROVIDES, REQUIRES, NAME class attributes
• Pipeline orchestrator with dependency resolution and script-aware model loading
• ProcessorRegistry for pluggable backends (rule, Apertium, Stanza, NLLB)
• ModelRegistry with remote catalog and local caching at ~/.turkicnlp/models/
• NLLB FLORES language map for ISO-3 to NLLB code resolution in translation (e.g. tuk -> tuk_Latn)

Model Storage Layout

~/.turkicnlp/models/
├── kaz/
│   ├── Cyrl/
│   │   ├── tokenize/rule/
│   │   ├── morph/apertium/    ← GPL-3.0 (downloaded separately)
│   │   │   ├── kaz.automorf.hfst
│   │   │   ├── LICENSE
│   │   │   └── metadata.json
│   │   ├── pos/neural/
│   │   └── depparse/neural/
│   └── Latn/
│       └── tokenize/rule/
├── tur/
│   └── Latn/
│       └── ...
├── huggingface/
│   └── facebook--nllb-200-distilled-600M/
│       ├── config.json
│       ├── model.safetensors (or pytorch_model.bin)
│       ├── tokenizer.json
│       └── ...
└── catalog.json

# Stanza models are managed by Stanza at ~/stanza_resources/

Notes:

• NLLB embeddings and translation use a shared Hugging Face model under ~/.turkicnlp/models/huggingface/.
• The NLLB model is downloaded once and reused across supported Turkic languages.
• Unlike Apertium/Stanza components, NLLB artifacts are not duplicated per language/script directory.

License

• Library code: Apache License 2.0
• Stanza models: Apache License 2.0 — managed by Stanza's own download mechanism
• Apertium FST data: GPL-3.0 — downloaded separately at runtime, never bundled in the pip package
• NLLB-200 model weights/tokenizer: CC-BY-NC-4.0 — downloaded from Hugging Face at runtime and reused from ~/.turkicnlp/models/huggingface/ (non-commercial license terms apply)

Development

git clone https://github.com/turkic-nlp/turkicnlp.git
cd turkicnlp
pip install -e ".[dev]"
pytest

Contributing

Contributions are welcome, especially:

• New language support — tag mappings, abbreviation lists, test data
• Neural model training — POS taggers, parsers, NER models
• Apertium FST improvements — better coverage for prototype-level languages
• Other - any other aspect that you want

Create issues, Pull Requests etc.

Acknowledgements

TurkicNLP builds on the work of many researchers and communities. We gratefully acknowledge the following:

Stanza

Stanza provides the pretrained neural models for tokenization, POS tagging, lemmatization, dependency parsing, and NER.

Peng Qi, Yuhao Zhang, Yuhui Zhang, Jason Bolton, and Christopher D. Manning. 2020. Stanza: A Python Natural Language Processing Toolkit for Many Human Languages. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations. [paper]

Universal Dependencies Treebanks

The Stanza models are trained on Universal Dependencies treebanks created by the following teams:

Turkish (UD_Turkish-IMST)

Umut Sulubacak, Memduh Gokirmak, Francis Tyers, Cagri Coltekin, Joakim Nivre, and Gulsen Cebiroglu Eryigit. Universal Dependencies for Turkish. COLING 2016. [paper]

Turkish (UD_Turkish-BOUN)

Utku Turk, Furkan Atmaca, Saziye Betul Ozates, Gozde Berk, Seyyit Talha Bedir, Abdullatif Koksal, Balkiz Ozturk Basaran, Tunga Gungor, and Arzucan Ozgur. Resources for Turkish Dependency Parsing: Introducing the BOUN Treebank and the BoAT Annotation Tool. Language Resources and Evaluation 56(1), 2022. [paper]

Turkish (UD_Turkish-FrameNet, KeNet, ATIS, Penn, Tourism)

Busra Marsan, Neslihan Kara, Merve Ozcelik, Bilge Nas Arican, Neslihan Cesur, Asli Kuzgun, Ezgi Saniyar, Oguzhan Kuyrukcu, and Olcay Taner Yildiz. Starlang Software and Ozyegin University. These treebanks cover diverse domains including FrameNet frames, WordNet examples, airline travel, Penn Treebank translations, and tourism reviews.

Kazakh (UD_Kazakh-KTB)

Aibek Makazhanov, Jonathan North Washington, and Francis Tyers. Towards a Free/Open-source Universal-dependency Treebank for Kazakh. TurkLang 2015. [paper]

Uyghur (UD_Uyghur-UDT)

Marhaba Eli (Xinjiang University), Daniel Zeman (Charles University), and Francis Tyers. [treebank]

Kyrgyz (UD_Kyrgyz-KTMU)

Ibrahim Benli. [treebank]

Ottoman Turkish (UD_Ottoman_Turkish-BOUN)

Saziye Betul Ozates, Tarik Emre Tiras, Efe Eren Genc, and Esma Fatima Bilgin Tasdemir. Dependency Annotation of Ottoman Turkish with Multilingual BERT. LAW-XVIII, 2024. [paper]

NER Datasets

Turkish NER (Starlang)

B. Ertopcu, A. B. Kanburoglu, O. Topsakal, O. Acikgoz, A. T. Gurkan, B. Ozenc, I. Cam, B. Avar, G. Ercan, and O. T. Yildiz. A New Approach for Named Entity Recognition. UBMK 2017. [paper]

Kazakh NER (KazNERD)

Rustem Yeshpanov, Yerbolat Khassanov, and Huseyin Atakan Varol (ISSAI, Nazarbayev University). KazNERD: Kazakh Named Entity Recognition Dataset. LREC 2022. [paper]

NLLB Embeddings & Machine Translation

TurkicNLP embeddings backend uses encoder pooling on:

facebook/nllb-200-distilled-600M

Reference:

NLLB Team, Marta R. Costa-jussà, et al. 2022. No Language Left Behind: Scaling Human-Centered Machine Translation. [paper]

Other Organisations

• Apertium — morphological transducers covering 20+ Turkic languages
• SIGTURK — ACL Special Interest Group on Turkic Languages
• ISSAI — Institute of Smart Systems and Artificial Intelligence, Nazarbayev University, for Kazakh NLP resources
• Universal Dependencies — the framework and community behind Turkic treebanks
• Turkic Interlingua — resources for machine translation for Turkic languages
• Turkic UD - group working on harmonizing Turkic UD treebanks