TurkicNLP

NLP toolkit for 20+ Turkic languages — a pip-installable Python library inspired by Stanza, with adaptations for the low-resource, morphologically rich Turkic language family.

Maintained by Sherzod Hakimov

Install · Quick start · Roadmap · Contributing · Changelog · Paper · Notebooks · 🤗 Models & data · Discord

Help us cover every Turkic language. Native speakers, linguists and developers are all welcome — see CONTRIBUTING.md for how to get started, and ROADMAP.md for planned features, known issues and open ideas you can pick up.

Citation

If you use TurkicNLP in your research, please cite:

@misc{hakimov2026turkicnlpnlptoolkit,
      title={TurkicNLP: An NLP Toolkit for Turkic Languages}, 
      author={Sherzod Hakimov},
      year={2026},
      eprint={2602.19174},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2602.19174}, 
}

Links

Paper (arXiv)	arxiv.org/abs/2602.19174
Code samples (Jupyter notebooks)	turkic-nlp/turkic-nlp-code-samples
Datasets & models	🤗 huggingface.co/turkicnlp
Community	Discord
Planned work and known issues	ROADMAP.md
How to contribute	CONTRIBUTING.md

Features

• 24 Turkic languages from Turkish to Sakha, Kazakh to Uyghur
• Script-aware from the ground up — Latin, Cyrillic, Perso-Arabic, Old Turkic Runic
• Automatic script detection and bidirectional transliteration
• Apertium FST morphology for ~20 Turkic languages via Python-native hfst bindings (no system install)
• Stanza/UD integration — pretrained tokenization, POS tagging, lemmatization and dependency parsing via Stanza models trained on Universal Dependencies treebanks (9 languages, 5 of them custom-trained), plus NER for Turkish and Kazakh
• Morpheme segmentation (MorphemeTokenizer) — splits words into labelled morphemes by combining Apertium and the neural analyzer (89% exact segmentation on a 483-word test set in 20 languages)
• Morpheme-aware subword tokenization (sp_tokenize) — SentencePiece over morpheme-segmented text, for training language models
• NLLB embeddings + translation backend — sentence/document vectors and MT via NLLB-200 (11 Turkic languages)
• Language identification (LanguageDetection, GlotLID model) — FastText-based LID with 1,000+ Glottolog language labels
• Speech recognition (SpeechRecognizer) — speech-to-text for 20 Turkic languages via Meta's Omnilingual ASR, with output in TurkicNLP scripts and a speech → text → analysis pipeline
• Multilingual Glot500 neural models — POS tagging & dependency parsing (15 languages), morphological analysis & lemmatization (22 TurkicNLP languages) via shared Glot500 backbone
• LLM backend — use any LLM for morphology, NER, translation, sentiment and spelling correction in all 24 languages: a local model served with vLLM or Ollama, OpenAI, Claude, or an in-process Hugging Face model (details)
• Multiple backends — choose between rule-based, Apertium FST, Stanza, Glot500 neural or LLM backends per processor
• License isolation — library is Apache-2.0; Apertium GPL-3.0 data downloaded separately
• Stanza-compatible API — Pipeline, Document, Sentence, Word

Installation

Requirements: Python 3.10 – 3.12.

TurkicNLP has a small core and optional extras for each group of components:

pip install turkicnlp                    # core — tokenization, rule-based processing, transliteration, CoNLL-U I/O
pip install "turkicnlp[hfst]"           # + Apertium FST morphology (Linux and macOS only)
pip install "turkicnlp[stanza]"         # + Stanza neural models (tokenize, POS, lemma, depparse, NER)
pip install "turkicnlp[lid]"            # + Language detection (GlotLID model; FastText + HF weights)
pip install "turkicnlp[translation]"    # + NLLB embeddings and machine translation
pip install "turkicnlp[transformers]"   # + Glot500 multilingual POS/DepParse/Morph models
pip install "turkicnlp[all]"            # all text components above (Linux and macOS only)
pip install "turkicnlp[asr]"            # + speech recognition (Omnilingual ASR) — see below, use a separate environment
pip install "turkicnlp[dev]"            # development tools (pytest, black, ruff, mypy)

Choosing an environment: text-only vs. speech + text

Speech recognition is built on omnilingual-asr, whose dependency fairseq2 pins torch==2.8, numpy<2 and huggingface_hub<1, and therefore transformers 4.x. All other TurkicNLP components also work with these versions, so there are two tested setups:

	Text environment	Speech + text environment
Extras	[all]	[all,asr]
Use it for	all text components	speech recognition and all text components
Python	3.10 – 3.12	3.10 – 3.12
torch / torchaudio	2.10 / –	2.8 / 2.8
numpy	2.x	1.26
transformers / huggingface_hub	5.x / 1.x	4.57 / 0.36
stanza	1.11	1.11
Exact versions	requirements/lock-text.txt	requirements/lock-speech.txt

⚠️ Do not add [asr] to an existing text environment: it downgrades torch, numpy and huggingface_hub, which breaks transformers 5.x. Create a fresh environment instead.

Text environment

python3 -m venv venv
source venv/bin/activate              # Windows: venv\Scripts\activate
pip install --upgrade pip
pip install "turkicnlp[all]"          # from PyPI
# or, from a clone of this repository:
pip install -e ".[all]"

Speech + text environment

System requirements: libsndfile and a C++ compiler with CMake (kenlm, a dependency of omnilingual-asr, is compiled from source).

# macOS
brew install libsndfile cmake
# Ubuntu / Debian
sudo apt-get install libsndfile1 cmake build-essential

python3.10 -m venv venv-asr
source venv-asr/bin/activate
pip install --upgrade pip
pip install "turkicnlp[all,asr]"      # from PyPI
# or, from a clone of this repository:
pip install -e ".[all,asr]"

Reproducing the tested versions

The lock files pin every package of the two tested environments (Python 3.10, macOS arm64):

pip install -r requirements/lock-text.txt   && pip install -e .   # text environment
pip install -r requirements/lock-speech.txt && pip install -e .   # speech + text environment

Version constraints and known issues

• Stanza < 1.12. The custom Azerbaijani, Uzbek, Turkmen, Tatar and Bashkir Stanza models were trained with Stanza 1.11; Stanza 1.15 fails to load their parsers (size mismatch for deprel.scorer). The extras enforce stanza>=1.8,<1.12.
• sentencepiece is needed to build the Glot500 (XLM-R) tokenizer with transformers 4.x; it is included in the [transformers], [translation] and [all] extras.
• torchaudio 2.8. omnilingual-asr does not pin torchaudio; newer versions fail to load against torch 2.8 (Symbol not found: _torch_library_impl). The [asr] extra pins torchaudio>=2.8,<2.9.
• kenlm build fails on macOS with 'climits' file not found. A stale /Library/Developer/CommandLineTools/usr/include/c++/v1 left by an old Command Line Tools install shadows the SDK's C++ headers. Either remove it once:

  sudo mv /Library/Developer/CommandLineTools/usr/include/c++ /Library/Developer/CommandLineTools/usr/include/c++.old
  

  or point the compiler at the SDK headers for the install:

  export CXXFLAGS="-nostdinc++ -isystem $(xcrun --show-sdk-path)/usr/include/c++/v1"
  export CMAKE_POLICY_VERSION_MINIMUM=3.5      # kenlm's CMakeLists predates CMake 4
  pip install -e ".[all,asr]"
  

• Memory for speech recognition. omniASR_LLM_1B (9 GB checkpoint) runs in about 6 GB of RAM with the default low-memory loading (memory-mapped checkpoint, bfloat16 weights). For smaller machines use omniASR_LLM_300M or a CTC model (omniASR_CTC_300M, 1.3 GB).

Open issues of the components themselves (e.g. low Apertium coverage for some languages) are tracked in ROADMAP.md → Known issues.

Verify the installation

import turkicnlp

# Core (no downloads)
from turkicnlp.scripts import Script
from turkicnlp.scripts.transliterator import Transliterator
print(Transliterator("kaz", Script.CYRILLIC, Script.LATIN).transliterate("Қазақстан"))   # Qazaqstan

# Text components (downloads models on first use)
nlp = turkicnlp.Pipeline("kaz", processors=["tokenize", "pos", "lemma", "depparse"])
doc = nlp("Мен мектепке бардым.")
print([(w.text, w.upos, w.lemma) for w in doc.words])
# [('Мен', 'PRON', 'мен'), ('мектепке', 'NOUN', 'мектеп'), ('бардым', 'VERB', 'бар'), ('.', 'PUNCT', '.')]

# Speech environment only
print(turkicnlp.list_asr_languages(check_model=True))   # {'tur': {'Latn': 'tur_Latn'}, 'aze': {...}, ...}

Platform compatibility

Installation tests run on every push and pull request and nightly, across all combinations of OS, Python version and install extra below; the test suite (pytest -m "not slow") runs on Ubuntu and macOS (see CI workflow).

Extra	Ubuntu 22.04 / 24.04	macOS 14 / 15	Windows 2025
base	✅ 3.9 – 3.12	✅ 3.9 – 3.12	✅ 3.9 – 3.12
[hfst]	✅ 3.9 – 3.12	✅ 3.9 – 3.12	❌ not available
[stanza]	✅ 3.9 – 3.12	✅ 3.9 – 3.12	✅ 3.9 – 3.12
[lid]	✅ 3.9 – 3.12	✅ 3.9 – 3.12	✅ 3.9 – 3.12
[transformers]	✅ 3.9 – 3.12	✅ 3.9 – 3.12	✅ 3.9 – 3.12
[translation]	✅ 3.9 – 3.12	✅ 3.9 – 3.12	✅ 3.9 – 3.12
[all]	✅ 3.9 – 3.12	✅ 3.9 – 3.12	❌ not available
[asr]	not yet in CI	✅ 3.10 (manually tested)	not tested

Windows users: the hfst Python package has no published wheels for Python 3.7 or later on Windows — this is an upstream limitation with no current workaround. All features except Apertium FST morphology work normally on Windows; use turkicnlp[stanza] or turkicnlp[translation] instead. If you need Apertium FST morphology on Windows, the recommended approach is Windows Subsystem for Linux (WSL), where hfst installs normally.

Quick Start

import turkicnlp

# Download models for a language
turkicnlp.download("kaz")

# Build a pipeline
nlp = turkicnlp.Pipeline("kaz", processors=["tokenize", "pos", "lemma", "ner", "depparse"])

# Process text
doc = nlp("Мен мектепке бардым")

# Access annotations
for sentence in doc.sentences:
    for word in sentence.words:
        print(f"{word.text}\t{word.lemma}\t{word.upos}\t{word.feats}")

# Export to CoNLL-U
print(doc.to_conllu())

turkicnlp.Pipeline("kaz") without processors= loads the default set for the language: the tokenizer and the morphology, POS, lemma, feature, parsing and NER processors listed in the catalog (Apertium morphology is preferred over the neural analyzer when both exist). Processors that download large models or need an extra environment — translate, embeddings, sp_tokenize, asr — are loaded only when you name them. See Processors and backends for all names.

Embeddings (NLLB)

import math
import turkicnlp

turkicnlp.download("tur", processors=["embeddings"])
nlp = turkicnlp.Pipeline("tur", processors=["embeddings"])

doc1 = nlp("Bugün hava çok güzel ve parkta yürüyüş yaptım.")
doc2 = nlp("Parkta yürüyüş yapmak bugün çok keyifliydi çünkü hava güzeldi.")

def cosine_similarity(a, b):
    dot = sum(x * y for x, y in zip(a, b))
    norm_a = math.sqrt(sum(x * x for x in a))
    norm_b = math.sqrt(sum(y * y for y in b))
    return dot / (norm_a * norm_b)

print(len(doc1.embedding), len(doc2.embedding))
print(f"cosine = {cosine_similarity(doc1.embedding, doc2.embedding):.4f}")
print(doc1._processor_log)  # ['embeddings:nllb']

Language ID (LanguageDetection, GlotLID model)

import turkicnlp

lid = turkicnlp.LanguageDetection()  # defaults to all Turkic languages supported by TurkicNLP
labels, probs = lid.predict("salam, hemmelere!", k=3)
print(labels, probs)

# Limit to specific labels
limited = turkicnlp.LanguageDetection(
    languages=[
        "__label__eng_Latn",
        "__label__tur_Latn",
        "__label__kaz_Cyrl",
    ]
)
print(limited.predict("Merhaba dünya!", k=1))

Speech Recognition (Omnilingual ASR)

SpeechRecognizer wraps Meta's Omnilingual ASR models (omniASR_LLM_1B by default) behind TurkicNLP conventions:

• languages are given as TurkicNLP ISO 639-3 codes (kaz, uzb, …) and mapped to Omnilingual codes (kaz_Cyrl, uzn_Latn, …);
• only the 20 Turkic languages below are accepted; each code is checked against the model's supported_langs, and non-Turkic languages (eng, rus, …) are rejected;
• transcripts are returned in the language's primary TurkicNLP script (or the one you request), using the toolkit's transliterator where the model writes another script;
• recordings longer than the model's 40-second limit are split at pauses and the pieces are transcribed and joined;
• the model is loaded once and shared by all languages; by default it is loaded memory-mapped in bfloat16 (≈ 6 GB RAM).

Requires the speech + text environment. The model weights are downloaded on first use (9.1 GB for omniASR_LLM_1B) and cached in ~/.cache/fairseq2/assets/.

Transcribe audio files

import turkicnlp

asr = turkicnlp.SpeechRecognizer(
    model_card="omniASR_LLM_1B",   # default; see the model table below
    device="cpu",                  # default: "cuda" if available, else "cpu"
    dtype="bfloat16",              # default ("auto"): bfloat16, or float16 on Apple "mps"
    low_memory=True,               # default: memory-map the checkpoint while loading
)

files = ["asr_samples/kaz_sample.wav", "asr_samples/tur_sample.wav"]
texts = asr.transcribe(files, lang=["kaz", "tur"], batch_size=2)
print(texts)
# ['мен бүгін мектепке бардым алматы қазақстанның ең үлкен қаласы',
#  "bugün hava çok güzel ali dün ankara'ya gitti"]

The interface mirrors Omnilingual ASR's ASRInferencePipeline.transcribe(audio_files, lang=..., batch_size=...); lang is one code for all inputs or a list with one code per input. A single input returns a string, a list returns a list.

Segments, details and output script

res = asr.transcribe("asr_samples/kaz_sample.wav", lang="kaz", return_details=True)
print(res.lang, res.model_lang, res.script)   # kaz kaz_Cyrl Cyrl
print(res.text)
for seg in res.segments:                      # time-aligned pieces (long audio is split at pauses)
    print(f"[{seg.start:.2f}-{seg.end:.2f}] {seg.text}")

# Ask for another script: Kazakh in the 2021 Latin alphabet
print(asr.transcribe("asr_samples/kaz_sample.wav", lang="kaz", script="Latn"))
# men bügın mektepke bardym almaty qazaqstannyñ eñ ülken qalasy

# The model writes Karakalpak and Crimean Tatar in Cyrillic; TurkicNLP returns their primary Latin script
res = asr.transcribe("kaa_audio.wav", lang="kaa", return_details=True)
print(res.model_text)   # Cyrillic, as produced by the model
print(res.text)         # Latin (2016 Karakalpak alphabet)

Other input types

import soundfile as sf

wav, sr = sf.read("asr_samples/tur_sample.wav")
asr.transcribe(wav, lang="tur", sample_rate=sr)                         # waveform array
asr.transcribe({"waveform": wav, "sample_rate": sr}, lang="tur")        # dict
asr.transcribe(open("asr_samples/tur_sample.wav", "rb").read(), lang="tur")   # encoded bytes

# Hugging Face datasets audio columns work directly
# from datasets import load_dataset
# ds = load_dataset(..., split="test")
# asr.transcribe(list(ds["audio"][:8]), lang="kir", batch_size=4)

Stereo audio is converted to mono and resampled to 16 kHz automatically.

Speech → text → analysis

nlp = turkicnlp.Pipeline(
    "kaz",
    processors=["asr", "tokenize", "pos", "lemma", "depparse"],
    asr_model_card="omniASR_LLM_1B",   # asr_* options configure the recognizer
    asr_device="cpu",                  # also: asr_dtype, asr_batch_size, asr_max_segment_seconds
)
doc = nlp.from_audio("asr_samples/kaz_sample.wav")
print(doc.text)
print(doc.audio_segments)              # [{'start': 0.0, 'end': 4.86, 'text': '...'}]
for w in doc.words:
    print(w.text, w.upos, w.lemma, w.head, w.deprel)

Omnilingual ASR outputs lowercase text without punctuation, so tagging and parsing quality on transcripts can be lower than on edited text.

Command line

python run_asr.py asr_samples/kaz_sample.wav asr_samples/tur_sample.wav --lang kaz tur --device cpu
python run_asr.py lecture.wav --lang tat --model omniASR_LLM_300M      # smaller model
python run_asr.py interview.wav --lang uzb --script Cyrl --analyze      # + tokenize/POS/depparse

asr_samples/ contains two short synthetic test recordings (Kazakh and Turkish).

Models and memory

Any Omnilingual ASR model card can be passed as model_card:

Model card	Checkpoint	Notes
omniASR_LLM_300M	6.5 GB	smallest LLM model
omniASR_LLM_1B (default)	9.1 GB	≈ 6 GB RAM with low-memory loading (tested on a 16 GB MacBook)
omniASR_LLM_3B	17.5 GB	GPU recommended
omniASR_LLM_7B	31.2 GB	best quality; GPU required in practice
omniASR_CTC_300M / _1B / _3B	1.3 / 3.9 / 12.3 GB	faster, no language conditioning (the language code is ignored)

If the process is killed for lack of memory, keep low_memory=True and dtype="bfloat16" (the defaults) or use a smaller model. dtype="float32" doubles the memory needed.

Supported languages

Language	Code	Omnilingual code(s)	Output script
Turkish	tur	tur_Latn	Latin
Azerbaijani	aze	aze_Latn, aze_Cyrl	Latin
South Azerbaijani	azb	aze_Arab	Perso-Arabic
Turkmen	tuk	tuk_Latn	Latin
Gagauz	gag	gag_Latn	Latin
Kazakh	kaz	kaz_Cyrl	Cyrillic (Latin via transliteration)
Kyrgyz	kir	kir_Cyrl	Cyrillic
Tatar	tat	tat_Cyrl	Cyrillic (Latin via transliteration)
Bashkir	bak	bak_Cyrl	Cyrillic
Crimean Tatar	crh	crh_Cyrl	Latin (transliterated)
Karakalpak	kaa	kaa_Cyrl	Latin (transliterated)
Nogai	nog	nog_Cyrl	Cyrillic
Kumyk	kum	kum_Cyrl	Cyrillic
Karachay-Balkar	krc	krc_Cyrl	Cyrillic
Uzbek	uzb	uzn_Latn, uzb_Cyrl	Latin
Uyghur	uig	uig_Arab, uig_Cyrl	Perso-Arabic
Sakha	sah	sah_Cyrl	Cyrillic
Altai	alt	alt_Cyrl	Cyrillic
Khakas	kjh	kjh_Cyrl	Cyrillic
Chuvash	chv	chv_Cyrl	Cyrillic

Not available: Tuvan, Khalaj, Ottoman Turkish and Old Turkic (no Omnilingual language code). turkicnlp.list_asr_languages(check_model=True) lists the codes supported by the installed model.

Machine Translation (NLLB)

import turkicnlp

# Downloads once into ~/.turkicnlp/models/huggingface/facebook--nllb-200-distilled-600M
turkicnlp.download("tur", processors=["translate"])

nlp = turkicnlp.Pipeline(
    "tur",
    processors=["translate"],
    translate_tgt_lang="eng",
)

doc = nlp("Bugün hava çok güzel ve parkta yürüyüş yaptım.")
print(doc.translation)
print(doc._processor_log)  # ['translate:nllb']

HTML entities that NLLB sometimes produces (', ") are converted back to plain characters. translate_tgt_lang accepts either ISO-639-3 ("eng", "tuk", "kaz") or explicit Flores-200 codes ("eng_Latn", "kaz_Cyrl").

LLM Backend (local vLLM / Ollama, OpenAI, Claude, or a Hugging Face model)

Five processors (six backends) can use a large language model, for all 24 languages — including languages without any trained model, such as Khakas, Tuvan or Nogai. The toolkit sends the request, so you only say which LLM to use: a server URL plus an API key, or a local Hugging Face model id. The OpenAI-style API is spoken by OpenAI, Azure OpenAI, vLLM, Ollama, llama.cpp, LM Studio, OpenRouter and others; Claude is supported through Anthropic's OpenAI-compatible endpoint. No extra package is needed for the API case.

Backend	What the LLM does	Writes
morph=llm	lemma, UPOS and UD features for every token	word.lemma/upos/feats
morph=apertium_llm	Apertium analyses as usual; for ambiguous words the LLM chooses one of the transducer's readings (it cannot invent an analysis); a configured apertium_tagger_cmd still decides first, and all sentences go to the LLM as one parallel batch	word.lemma/upos/feats
ner=llm	entity spans over the toolkit's tokens (labels: ner_labels=["PER", "LOC", "ORG"])	word.ner, sentence.entities
translate=llm	translation into translate_tgt_lang (ISO code or language name)	sentence.translation, doc.translation
sentiment=llm	positive / negative / neutral (or sentiment_labels=[...])	sentence.sentiment
spellcheck=llm	spelling correction; answers that rewrite >30% of a sentence are rejected	sentence.correction, sentence.corrections, doc.corrected_text

1. Local model with vLLM (GPU server)

Start an OpenAI-compatible server with a small model (Linux + NVIDIA GPU; install vLLM in its own environment, pip install vllm):

bash scripts/start_vllm_server.sh                                   # Qwen/Qwen2.5-1.5B-Instruct on :8000
MODEL=Qwen/Qwen3-8B PORT=8001 bash scripts/start_vllm_server.sh     # another model / port
GPUS=0,1 TP=2 MODEL=Qwen/Qwen3-32B bash scripts/start_vllm_server.sh # two GPUs

Then use it from TurkicNLP (from another machine, replace localhost with the server name):

import turkicnlp

nlp = turkicnlp.Pipeline(
    "kaz",
    processors=["tokenize", "morph", "ner", "sentiment", "spellcheck", "translate"],
    morph_backend="llm", ner_backend="llm", sentiment_backend="llm", translate_backend="llm",
    translate_tgt_lang="eng",
    llm_model="Qwen/Qwen2.5-1.5B-Instruct",       # the model the server was started with
    llm_base_url="http://localhost:8000/v1",
    llm_api_key="turkicnlp-local",                # API_KEY of start_vllm_server.sh
)
doc = nlp("Қасым-Жомарт Тоқаев кеше Астанада шетелдік инвесторлармен кездесті.")

for w in doc.words:
    print(w.text, w.lemma, w.upos, w.feats)
print(doc.entities, doc.sentences[0].sentiment, doc.corrected_text, doc.translation)

Without a GPU (e.g. on a Mac) use Ollama the same way: ollama pull qwen2.5:7b, then llm_model="qwen2.5:7b", llm_base_url="http://localhost:11434/v1", llm_api_key="ollama".

2. OpenAI

import os, turkicnlp

os.environ["OPENAI_API_KEY"] = "sk-..."           # or pass llm_api_key="sk-..."

nlp = turkicnlp.Pipeline(
    "uzb",
    processors=["tokenize", "ner", "translate"],
    ner_backend="llm", translate_backend="llm", translate_tgt_lang="deu",
    llm_model="gpt-4.1-mini",                     # any chat model of your account
)                                                  # llm_base_url defaults to https://api.openai.com/v1
doc = nlp("Shavkat Mirziyoyev Toshkentda yangi maktabni ochdi.")
print(doc.entities, doc.translation)

3. Claude (Anthropic)

import os, turkicnlp

os.environ["ANTHROPIC_API_KEY"] = "sk-ant-..."    # or pass llm_api_key="sk-ant-..."

nlp = turkicnlp.Pipeline(
    "tyv",
    processors=["tokenize", "morph", "spellcheck"],
    morph_backend="llm",
    llm_provider="anthropic",                     # https://api.anthropic.com/v1, key from ANTHROPIC_API_KEY
    llm_model="claude-sonnet-4-6",
)
doc = nlp("Мен бөгүн школага бардым.")
print([(w.text, w.lemma, w.upos, w.feats) for w in doc.words], doc.corrected_text)

Model names change; use any model your account offers. With llm_provider="anthropic" no temperature is sent, because recent Claude models only accept the default.

4. Hugging Face model in-process (no server)

nlp = turkicnlp.Pipeline("tur", processors=["tokenize", "ner"], ner_backend="llm",
                         llm_provider="hf", llm_model="Qwen/Qwen2.5-1.5B-Instruct")

Needs pip install "turkicnlp[transformers]"; requests run one at a time, so this is for small models and small amounts of text. For anything larger, serve the model with vLLM.

Example run

examples/llm_backend_demo.py runs all LLM backends on one text (--backend vllm | ollama | openai | claude | hf):

$ python examples/llm_backend_demo.py --backend hf --model Qwen/Qwen2.5-1.5B-Instruct --lang kaz

Model: Qwen/Qwen2.5-1.5B-Instruct (hf)
Text:  Қасым-Жомарт Тоқаев кеше Астанада шетелдік инвесторлармен кездесті. Бұл кездесу өте пайдалы болды.

Morphology:
  Қасым-Жомарт
  Тоқаев
  кеше
  Астанада
  шетелдік
  инвесторлармен
  кездесті
  .
  Бұл                    бұл                DET    Case=Gen|Definite=Def|Num=One
  кездесу                кездесу            NOUN   Case=Acc|Gender=Fem|Number=Sing|PronType=Dem
  өте                    өте                ADP    Case=Loc
  пайдалы                пайда              NOUN   Case=Acc|Gender=Masc|Number=Sing|PronType=Dem
  болды                  болды              VERB   Aspect=Perf|Mood=Ind|Number=Sing|Person=3|Polarity=Pos|Tense=Pres|VerbForm=Fin
  .                      .                  PUNCT  _

Entities:    [('Қасым-Жомарт Тоқаев', 'PER'), ('Астанада', 'LOC'), ('инвесторлармен', 'ORG')]
Sentiment:   ['positive', 'positive']
Spelling:    Қазым-Жомарт Тоқаев кеше Астанада шетелдік инвесторлармен кездесті. Бұл кездесу өте пайдалы болды. [('Қасым-Жомарт', 'Қазым-Жомарт')]
Translation: Kasym-Jomart Tokayev metalled in Astana to meet investors. It is very useful.

Provenance:  ['tokenize:stanza', 'spellcheck:llm(Qwen/Qwen2.5-1.5B-Instruct, spellcheck/v1)',
              'morph:llm(Qwen/Qwen2.5-1.5B-Instruct, morph/v2)[1 failed]', 'ner:llm(Qwen/Qwen2.5-1.5B-Instruct, ner/v1)',
              'sentiment:llm(Qwen/Qwen2.5-1.5B-Instruct, sentiment/v1)', 'translate:llm(Qwen/Qwen2.5-1.5B-Instruct, translate/v1)']
Time: 274.7s

This run used a 1.5B-parameter model on a laptop, which shows both how the backend works and why the model matters:

• Validated output. For the first sentence the model's morphology answer did not cover the tokens, even after a retry with feedback, so those fields stay empty and the log says [1 failed] — the toolkit does not write shifted or invented analyses. With morph_on_error="raise" it would stop instead.
• Spelling guard. Word changes that are not spelling fixes (rewritten suffixes, added words) are reverted; a plausible-looking one-letter change such as Қасым → Қазым still gets through, which is a model error.
• Quality. Lemmas, features, the ORG label and "metalled" are wrong: a 1.5B model is enough to test the set-up, not for Kazakh analysis. Use a larger model (e.g. Qwen3-32B on vLLM, GPT or Claude) for real work, and evaluate it on your language first.

How it works and where the prompt is

Each LLM backend is an ordinary pipeline processor. It fills its prompt template — one versioned file per task in turkicnlp/resources/prompts/*.json — with the language, script, the toolkit's tokens and a few examples, asks the model for JSON, validates the answer (and asks once more if it is unusable) and writes the normal Document fields. doc._processor_log records model and prompt version, e.g. ner:llm(Qwen/Qwen2.5-1.5B-Instruct, ner/v1). Replace a prompt with ner_prompt_file="my_ner.json" (in a template only {name} placeholders of known variables are filled; other braces, e.g. a JSON example, are left as written) or add examples with ner_examples=[{"vars": {"tokens": [...]}, "answer": {...}}].

Settings (llm_* applies to all LLM processors; <processor>_llm_* overrides one of them, e.g. ner_llm_model="gpt-4.1"):

Option	Default
llm_model	env TURKICNLP_LLM_MODEL	model name / Hugging Face id
llm_base_url	env TURKICNLP_LLM_BASE_URL, else https://api.openai.com/v1	e.g. http://localhost:8000/v1 (vLLM), http://localhost:11434/v1 (Ollama)
llm_api_key	env TURKICNLP_LLM_API_KEY; OPENAI_API_KEY only when llm_base_url is api.openai.com; ANTHROPIC_API_KEY for anthropic	sent as Authorization: Bearer …; masked in repr(LLMConfig) (as are header values)
llm_provider	openai	anthropic (Claude), hf (local transformers model)
llm_headers, llm_extra_body	–	extra HTTP headers (e.g. Azure api-key) / request fields
llm_temperature, llm_max_tokens	0.0, 2048	llm_temperature=None sends no temperature
llm_timeout, llm_max_retries, llm_max_workers	120 s, 3, 4	parallel requests per document; Retry-After waits are capped at 60 s
llm_json_mode	auto	JSON schema → JSON object → plain prompt, stepping down only when the server answers that the response_format is unsupported
llm_cache, llm_cache_dir	on, ~/.turkicnlp/llm_cache.sqlite	identical requests are answered from the cache; only answers that passed validation are stored; an unwritable cache location disables the cache with a warning
<processor>_on_error	warn	raise to stop on unusable answers (an unusable answer — invalid JSON, an HTML error page, a dropped connection — only affects its own sentence)
translate_tgt_lang, translate_tgt_script	eng	target as code (uzb), code with script (uzb_Cyrl) or name; the prompt names the script, e.g. "Uzbek (Cyrillic script)"
llm_client	–	pass one turkicnlp.LLMClient object to share it

Free-form questions with the toolkit's analyses as context:

doc = turkicnlp.Pipeline("kaz", processors=["tokenize", "morph"])("Мен кеше мектепке бардым.")
print(turkicnlp.llm.ask("Explain the verb form in this sentence.", doc=doc,
                        llm_provider="anthropic", llm_model="claude-sonnet-4-6"))

The context sent with the question is limited to max_words=400 words (ask(..., max_words=...)).

LLM quality differs strongly between languages (good for Turkish, Kazakh, Uzbek, Azerbaijani; often weak for Khakas, Tuvan, Nogai or Kumyk). Evaluate a model for your language before relying on it, and remember that the text is sent to the server you configure.

Using the Stanza Backend

from turkicnlp.processors.stanza_backend import (
    StanzaTokenizer, StanzaPOSTagger, StanzaLemmatizer, StanzaNERProcessor, StanzaDepParser
)
from turkicnlp.models.document import Document

# Models are downloaded automatically on first use
doc = Document(text="Merhaba dünya.", lang="tur")

for Proc in [StanzaTokenizer, StanzaPOSTagger, StanzaLemmatizer, StanzaNERProcessor, StanzaDepParser]:
    proc = Proc(lang="tur")
    proc.load()
    doc = proc.process(doc)

for word in doc.words:
    print(f"{word.text:12} {word.upos:6} {word.lemma:12} head={word.head} {word.deprel}")

# Export to CoNLL-U
print(doc.to_conllu())

Mixed Backends

from turkicnlp.processors.tokenizer import RegexTokenizer
from turkicnlp.processors.stanza_backend import StanzaPOSTagger, StanzaNERProcessor, StanzaDepParser
from turkicnlp.models.document import Document

doc = Document(text="Мен мектепке бардым.", lang="kaz")

# Rule-based tokenizer + Stanza POS/parsing (pretokenized mode)
tokenizer = RegexTokenizer(lang="kaz")
tokenizer.load()
doc = tokenizer.process(doc)

pos = StanzaPOSTagger(lang="kaz")
pos.load()
doc = pos.process(doc)

ner = StanzaNERProcessor(lang="kaz")
ner.load()
doc = ner.process(doc)

parser = StanzaDepParser(lang="kaz")
parser.load()
doc = parser.process(doc)

Multi-Script Support

# Kazakh — auto-detects Cyrillic vs Latin
doc = nlp("Мен мектепке бардым")    # Cyrillic
doc = nlp("Men mektepke bardym")     # Latin

# Explicit script selection
nlp_cyrl = turkicnlp.Pipeline("kaz", script="Cyrl")
nlp_latn = turkicnlp.Pipeline("kaz", script="Latn")

# Transliteration bridge — run Cyrillic model on Latin input
nlp = turkicnlp.Pipeline("kaz", script="Latn", transliterate_to="Cyrl")

Uyghur (Perso-Arabic)

nlp_ug = turkicnlp.Pipeline("uig", script="Arab")
doc = nlp_ug("مەن مەكتەپكە باردىم")

Transliteration

The Transliterator class converts text between scripts for any supported language pair:

from turkicnlp.scripts import Script
from turkicnlp.scripts.transliterator import Transliterator

# Kazakh Cyrillic → Latin (2021 official alphabet)
t = Transliterator("kaz", Script.CYRILLIC, Script.LATIN)
print(t.transliterate("Қазақстан Республикасы"))
# → Qazaqstan Respublikasy

# Uzbek Latin → Cyrillic
t = Transliterator("uzb", Script.LATIN, Script.CYRILLIC)
print(t.transliterate("O'zbekiston Respublikasi"))
# → Ўзбекистон Республикаси

# Uyghur Perso-Arabic → Latin (ULY)
t = Transliterator("uig", Script.PERSO_ARABIC, Script.LATIN)
print(t.transliterate("مەكتەپ"))
# → mektep

# Azerbaijani Latin → Cyrillic
t = Transliterator("aze", Script.LATIN, Script.CYRILLIC)
print(t.transliterate("Azərbaycan"))
# → Азәрбайҹан

# Turkmen Latin → Cyrillic
t = Transliterator("tuk", Script.LATIN, Script.CYRILLIC)
print(t.transliterate("Türkmenistan"))
# → Түркменистан

# Tatar Cyrillic → Latin (Zamanälif)
t = Transliterator("tat", Script.CYRILLIC, Script.LATIN)
print(t.transliterate("Татарстан Республикасы"))
# → Tatarstan Respublikası

Common Turkic Script (CTS)

Every supported language except Old Turkic (23 languages) can be converted to the Common Turkic Alphabet — a unified (34 letters) Latin-based script designed for cross-language interoperability across the Turkic family. Turkish, which already uses a CTS-compatible alphabet, converts with no changes to most letters:

from turkicnlp.scripts import Script
from turkicnlp.scripts.transliterator import Transliterator

# Turkish Latin → CTS (near-identical alphabets)
t = Transliterator("tur", Script.LATIN, Script.COMMON_TURKIC)
print(t.transliterate("Türkçe güzel bir dil"))
# → Türkçe güzel bir dil

# Other languages gain a shared representation
t = Transliterator("kaz", Script.CYRILLIC, Script.COMMON_TURKIC)
print(t.transliterate("Қазақстан"))
# → Qazaqstan

t = Transliterator("aze", Script.LATIN, Script.COMMON_TURKIC)
print(t.transliterate("Azərbaycan dili"))
# → Azärbaycan dili

t = Transliterator("tuk", Script.LATIN, Script.COMMON_TURKIC)
print(t.transliterate("Türkmen dili baýdyr"))
# → Türkmen dili baydır

Old Turkic Runic Script

TurkicNLP supports transliteration of Old Turkic runic inscriptions (Orkhon-Yenisei script, Unicode block U+10C00–U+10C4F) to Latin:

from turkicnlp.scripts import Script
from turkicnlp.scripts.transliterator import Transliterator

t = Transliterator("otk", Script.OLD_TURKIC_RUNIC, Script.LATIN)

# Individual runic characters
print(t.transliterate("\U00010C45\U00010C07\U00010C3C\U00010C1A"))  # 𐱅𐰇𐰼𐰚
# → törk  (Türk)

# The transliterator maps each runic character to its standard
# Turkological Latin equivalent, handling both Orkhon and Yenisei
# variant forms (e.g., separate glyphs for consonants with
# back vs. front vowel contexts).

Neural POS Tagger & Dependency Parser (Glot500)

The multilingual Glot500-based model provides UPOS tagging and dependency parsing for 15 Turkic languages (10 trained + 5 zero-shot). Requires pip install "turkicnlp[transformers]".

import turkicnlp

# Download tokenizer + multilingual Glot500 POS/DepParse model
turkicnlp.download("kaz", processors=["tokenize", "pos", "depparse"])

nlp = turkicnlp.Pipeline(
    "kaz",
    processors=["tokenize", "pos", "depparse"],
    pos_backend="multilingual_glot500",
    depparse_backend="multilingual_glot500",
)

doc = nlp("Мен мектепке бардым.")

for sentence in doc.sentences:
    for word in sentence.words:
        print(f"{word.text:12} {word.upos:6} head={word.head} {word.deprel}")

Morpheme Tokenizer (Hybrid FST + Neural)

The MorphemeTokenizer segments inflected Turkic words into labelled surface morphemes. For every word it converts each Apertium HFST reading into a segmentation by matching the reading's tags against language-specific suffix allomorph tables (vowel harmony, consonant alternations, buffer consonants), and keeps the reading that explains the whole word, breaking ties by agreement with the neural (Glot500) lemma and part of speech. Derivational suffixes are split only when the transducer confirms the shorter stem. When the transducer has no usable reading, the neural analysis is segmented instead. Requires pip install "turkicnlp[hfst,transformers]" (works with only one of the two, with lower accuracy).

On a 483-word test set covering 13 word categories in 20 languages, the hybrid segments 89% of the words exactly (Apertium only: 79%, neural only: 68%). The test set and the evaluation script are in the paper's companion repository; segmentation is weakest for derivation and in languages with small Apertium lexicons (see ROADMAP.md).

from turkicnlp.processors.morpheme_tokenizer import MorphemeTokenizer

# --- Kazakh ---
tok = MorphemeTokenizer(lang="kaz")
tok.load()

result = tok.segment("бармадым")
print(result.segments)
# ['бар', 'ма', 'ды', 'м']

print(result.labeled)
# [('бар', 'STEM'), ('ма', 'NEG'), ('ды', 'PST'), ('м', '1SG')]

result = tok.segment("оқығандар")
print(result.labeled)
# [('оқы', 'STEM'), ('ған', 'PTCP.PST'), ('дар', 'PLUR')]

# --- Turkish ---
tok = MorphemeTokenizer(lang="tur")
tok.load()

result = tok.segment("evlerinden")
print(result.labeled)
# [('ev', 'STEM'), ('ler', 'PLUR'), ('in', 'POSS.2SG'), ('den', 'ABL')]

result = tok.segment("gidiyorlar")
print(result.labeled)
# [('gid', 'STEM'), ('iyor', 'PROG'), ('lar', '3PL')]

# Apostrophe boundaries are handled for proper nouns
result = tok.segment("İstanbul'da")
print(result.labeled)
# [('İstanbul', 'STEM'), ("'da", 'LOC')]

# --- Uzbek ---
tok = MorphemeTokenizer(lang="uzb")
tok.load()

result = tok.segment("kitobimdan")
print(result.labeled)
# [('kitob', 'STEM'), ('im', 'POSS.1SG'), ('dan', 'ABL')]

result = tok.segment("bolalarning")
print(result.labeled)
# [('bola', 'STEM'), ('lar', 'PLUR'), ('ning', 'GEN')]

Suffix allomorph tables exist for 23 languages (turkicnlp/resources/morpheme_rules.json; all except Old Turkic); the Apertium path is available for the 20 languages with a transducer. Results are cached per word (MorphemeTokenizer(lang, cache_size=...)).

Morpheme-aware subword tokenization (sp_tokenize)

sp_tokenize encodes text with a SentencePiece model trained on morpheme-segmented text. Input words are segmented with the MorphemeTokenizer first, so pieces do not cross morpheme boundaries. It adds sp_tokens and sp_ids to every sentence. Pretrained models are not released yet (ROADMAP.md); train one on your corpus with turkicnlp/tools/annotate_corpus.py + SentencePiece and pass its path:

import turkicnlp

nlp = turkicnlp.Pipeline(
    "kaz",
    processors=["tokenize", "sp_tokenize"],
    sp_tokenize_model_path="models/sp/sp_kaz.model",
)
doc = nlp("Мен мектептерімізден оқыдым")
print(doc.sentences[0].sp_tokens)

Neural Morphological Analyzer & Lemmatizer (Glot500)

The multilingual Glot500-based morph model provides UPOS tagging, UD morphological features, and lemmatization for 23 Turkic languages. Requires pip install "turkicnlp[transformers]".

import turkicnlp

# Download tokenizer + multilingual Glot500 morph model
turkicnlp.download("tur", processors=["tokenize", "morph_neural"])

nlp = turkicnlp.Pipeline(
    "tur",
    processors=["tokenize", "morph_neural"],
)

doc = nlp("Çocuklar okula gidiyorlar.")

for sentence in doc.sentences:
    for word in sentence.words:
        print(f"{word.text:20} {word.upos:6} {word.lemma:15} {word.feats}")

Output:

Çocuklar             NOUN   çocuk           Case=Nom|Number=Plur
okula                NOUN   okul            Case=Dat|Number=Sing
gidiyorlar           VERB   gitmek          Aspect=Prog|Mood=Ind|Number=Plur|Person=3|Tense=Pres|VerbForm=Fin
.                    PUNCT  .               _

The morph analyzer also works for low-resource languages:

# Sakha (Yakut) — directly trained
turkicnlp.download("sah", processors=["tokenize", "morph_neural"])
nlp = turkicnlp.Pipeline("sah", processors=["tokenize", "morph_neural"])
doc = nlp("Мин оскуолаҕа бардым.")

# Karakalpak — zero-shot via Uzbek proxy embedding
turkicnlp.download("kaa", processors=["tokenize", "morph_neural"])
nlp = turkicnlp.Pipeline("kaa", processors=["tokenize", "morph_neural"])
doc = nlp("Men mektepke bardım.")

Supported Languages and Components

Geographic distribution of Turkic languages (source: Wikimedia Commons)

The table below shows all supported languages with their available scripts and processor status.

Legend:

Symbol	Backend	Description
■	Rule-based	Regex tokenizer, abbreviation lists
◆	Apertium FST	Finite-state morphology via hfst (GPL-3.0, downloaded separately)
●	Stanza/UD	Neural models trained on Universal Dependencies treebanks
▲	Custom Stanza	Custom-trained Stanza models hosted by turkic-nlp
◇	Glot500 Neural	Multilingual POS tagger & dependency parser (Glot500 backbone, 15 languages)
◈	Glot500 Neural Morph	Multilingual morphological analyzer & lemmatizer (Glot500 backbone, 22 languages)
★	NLLB	Embeddings and machine translation via NLLB-200
◎	Omnilingual ASR	Speech recognition (SpeechRecognizer, speech + text environment)
○	Planned	Implementation planned
—		Not available yet

Oghuz Branch

Language	Code	Script(s)	Tokenize	Morph	POS	Lemma	DepParse	NER	Embed	Translate	ASR
Turkish	tur	Latn	■ ●	◆ ◈	● ◇	● ◈	● ◇	●	★	★	◎
Azerbaijani	aze	Latn, Cyrl	■▲	◆ ◈	▲ ◇	▲ ◈	▲ ◇	—	★	★	◎
Iranian Azerbaijani	azb	Arab	■	—	—	—	—	—	★	★	◎
Turkmen	tuk	Latn, Cyrl	■▲	◆ ◈	▲ ◇	▲ ◈	▲ ◇	—	★	★	◎
Gagauz	gag	Latn	■	◆ ◈	◈	◈	—	—	—	—	◎

Kipchak Branch

Language	Code	Script(s)	Tokenize	Morph	POS	Lemma	DepParse	NER	Embed	Translate	ASR
Kazakh	kaz	Cyrl, Latn	■ ●	◆ ◈	● ◇	● ◈	● ◇	●	★	★	◎
Kyrgyz	kir	Cyrl	■ ●	◆ ◈	● ◇	● ◈	● ◇	—	★	★	◎
Tatar	tat	Cyrl, Latn	■▲	◆ ◈	▲ ◇	▲ ◈	▲ ◇	—	★	★	◎
Bashkir	bak	Cyrl	■▲	◆ ◈	▲ ◇	▲ ◈	▲ ◇	—	★	★	◎
Crimean Tatar	crh	Latn, Cyrl	■	◆ ◈	◈	◈	—	—	★	★	◎
Karakalpak	kaa	Latn, Cyrl	■	◆ ◈	◇ ◈	◈	◇	—	—	—	◎
Nogai	nog	Cyrl	■	◆ ◈	◇ ◈	◈	◇	—	—	—	◎
Kumyk	kum	Cyrl	■	◆ ◈	◇ ◈	◈	◇	—	—	—	◎
Karachay-Balkar	krc	Cyrl	■	◆ ◈	◇ ◈	◈	◇	—	—	—	◎

Karluk Branch

Language	Code	Script(s)	Tokenize	Morph	POS	Lemma	DepParse	NER	Embed	Translate	ASR
Uzbek	uzb	Latn, Cyrl	■ ▲	◆ ◈	▲ ◇	▲ ◈	▲ ◇	—	★	★	◎
Uyghur	uig	Arab, Latn, Cyrl	■ ●	◆ ◈	● ◇	● ◈	● ◇	—	★	★	◎

Siberian Branch

Language	Code	Script(s)	Tokenize	Morph	POS	Lemma	DepParse	NER	Embed	Translate	ASR
Sakha (Yakut)	sah	Cyrl	■	◆ ◈	◇ ◈	◈	◇	—	—	—	◎
Altai	alt	Cyrl	■	◆ ◈	◈	◈	—	—	—	—	◎
Tuvan	tyv	Cyrl	■	◆ ◈	◈	◈	—	—	—	—	—
Khakas	kjh	Cyrl	■	◆ ◈	◈	◈	—	—	—	—	◎

Oghur Branch

Language	Code	Script(s)	Tokenize	Morph	POS	Lemma	DepParse	NER	Embed	Translate	ASR
Chuvash	chv	Cyrl	■	◆ ◈	◈	◈	—	—	—	—	◎

Arghu Branch

Language	Code	Script(s)	Tokenize	Morph	POS	Lemma	DepParse	NER	Embed	Translate	ASR
Khalaj	klj	Latn	■	◈	◈	◈	—	—	—	—	—

Historical Languages

Language	Code	Script(s)	Tokenize	Morph	POS	Lemma	DepParse	NER	Embed	Translate	ASR
Ottoman Turkish	ota	Arab, Latn	■	◈	◇ ◈	◈	◇	—	—	—	—
Old Turkic	otk	Orkh, Latn	■	—	—	—	—	—	—	—	—

Stanza/UD Model Details

The Stanza backend provides neural models trained on Universal Dependencies treebanks. Official Stanza models (●) are downloaded via Stanza's model hub. Custom-trained models (▲) are hosted at turkic-nlp/trained-stanza-models and downloaded automatically.

Language	Stanza Code	Type	UD Treebank(s)	Stanza Processors	NER Dataset
Turkish	tr	●	IMST (default), BOUN, FrameNet, KeNet, ATIS, Penn, Tourism	tokenize, pos, lemma, depparse, ner	Starlang NER
Kazakh	kk	●	KTB	tokenize, pos, lemma, depparse, ner	KazNERD
Uyghur	ug	●	UDT	tokenize, pos, lemma, depparse	—
Kyrgyz	ky	●	KTMU	tokenize, pos, lemma, depparse	—
Uzbek	uz	▲	UzUDT	tokenize, pos, lemma, depparse	—
Turkmen	tk	▲	Tk-TUD	tokenize, pos, lemma, depparse	—
Azerbaijani	az	▲	Az-TUD	tokenize, pos, lemma, depparse	—
Tatar	tt	▲	Ta-TUD	tokenize, pos, lemma, depparse	—
Bashkir	ba	▲	Ba-TUD	tokenize, pos, lemma, depparse	—

Multilingual Glot500 Neural Models

TurkicNLP provides two multilingual neural models built on a frozen Glot500 backbone with script adapters, language embeddings, and shared BiLSTM layers. Both models are hosted at turkic-nlp/trained-stanza-models and downloaded automatically.

Model	Symbol	Tasks	Languages	Architecture
POS & DepParser	◇	UPOS, dependency parsing	10 trained + 5 zero-shot (15 total)	Glot500 → ScriptAdapter → LangEmbed → BiLSTM → POS Head + Biaffine Parser
Morph Analyzer	◈	UPOS, UD morph features, lemmatization	20 trained + 3 zero-shot (23 model languages = 22 TurkicNLP codes; both Altai varieties map to alt)	Glot500 → ScriptAdapter → LangEmbed → BiLSTM → POS Head + MorphFeat Head + CharCNN LemmaHead

POS & DepParser supported languages: Turkish, Azerbaijani, Uzbek, Turkmen, Kazakh, Kyrgyz, Bashkir, Tatar, Uyghur, Ottoman Turkish + zero-shot: Karakalpak, Kumyk, Sakha, Karachay-Balkar, Nogai

Morph Analyzer supported languages: Turkish, Azerbaijani, Uzbek, Turkmen, Kazakh, Kyrgyz, Bashkir, Tatar, Uyghur, Ottoman Turkish, Crimean Tatar, Khakas, Sakha, Tuvan, Chuvash, Gagauz, Kumyk, Southern Altai, Khalaj, Northern Altai + zero-shot: Karakalpak, Karachay-Balkar, Nogai

Transliteration Support

Bidirectional script conversion is available for all multi-script languages. The transliterator uses a greedy longest-match algorithm with per-language mapping tables.

Language	Direction	Scripts	Standard
Kazakh	↔ Bidirectional	Cyrillic ↔ Latin	2021 official Latin alphabet
Uzbek	↔ Bidirectional	Cyrillic ↔ Latin	1995 official Latin alphabet
Azerbaijani	↔ Bidirectional	Cyrillic ↔ Latin	1991 official Latin alphabet
Tatar	↔ Bidirectional	Cyrillic ↔ Latin	Zamanälif
Turkmen	↔ Bidirectional	Cyrillic ↔ Latin	1993 official Latin alphabet
Karakalpak	↔ Bidirectional	Cyrillic ↔ Latin	2016 Latin alphabet
Crimean Tatar	↔ Bidirectional	Cyrillic ↔ Latin	Standard Crimean Tatar Latin
Uyghur	↔ Bidirectional	Perso-Arabic ↔ Latin, Perso-Arabic ↔ Cyrillic, Latin ↔ Cyrillic	Uyghur Latin Yéziqi (ULY), Uyghur Cyrillic
Ottoman Turkish	→ One-way	Latin → Perso-Arabic	Academic transcription
Old Turkic	→ One-way	Runic → Latin	Turkological convention
All except Old Turkic (23)	→ native script to CTS	Latin / Cyrillic / Perso-Arabic → CTS	Common Turkic Alphabet

Apertium FST Quality Levels

Level	Description	Languages
Production	>90% coverage on news text	Turkish, Kazakh, Tatar
Stable	Good coverage, actively maintained	Kyrgyz, Uzbek
Beta	Reasonable coverage, some gaps	Azerbaijani (many common verbs missing), Turkmen, Bashkir, Uyghur, Crimean Tatar, Chuvash
Prototype	Limited coverage, experimental	Gagauz, Sakha, Karakalpak, Nogai, Kumyk, Karachay-Balkar, Altai, Tuvan, Khakas

Model Catalog and Apertium Downloads

TurkicNLP uses a model catalog to define download sources per language/script/processor. The catalog lives in:

• turkicnlp/resources/catalog.json (packaged default)
• Remote override: ModelRegistry.CATALOG_URL (or TURKICNLP_CATALOG_URL)

For each language and script, the catalog lists the available processors, their backends, the default backend and the download source. turkicnlp.download() reads it and fetches what is missing: precompiled Apertium .hfst transducers from the turkic-nlp/apertium-data releases (built by CI from the upstream Apertium repositories), custom Stanza and Glot500 checkpoints from turkic-nlp/trained-stanza-models, and Hugging Face models (Glot500 backbone, NLLB). When a release contains several transducers (e.g. krc@Seegmiller, uzb_guesser), the main <lang>.automorf.hfst is used.

Download folder

All models and resources are downloaded into this folder: ~/.turkicnlp.

Architecture

TurkicNLP follows Stanza's modular pipeline design:

Pipeline("tur", processors=["tokenize", "morph", "pos", "ner", "depparse"])
    │
    ▼
  Document ─── text: "Ben okula vardım"
    │
    ├── script_detect    → script = "Latn"
    ├── tokenize         → sentences, tokens, words
    ├── morph (Apertium) → lemma, pos, feats (via HFST)
    ├── pos (Stanza)     → UPOS, XPOS, feats (owns these fields)
    ├── ner (Stanza)     → BIO tags and entity spans
    └── depparse         → head, deprel
    │
    ▼
  Document ─── annotated with all layers

Pipeline("sah", processors=["tokenize", "morph_neural"])
    │
    ▼
  Document ─── text: "Мин оскуолаҕа бардым"
    │
    ├── script_detect          → script = "Cyrl"
    ├── tokenize               → sentences, tokens, words
    └── morph_neural (Glot500) → upos, feats, lemma
    │
    ▼
  Document ─── annotated with morphological analysis

Pipeline("azb", processors=["embeddings", "translate"], translate_tgt_lang="eng")
    │
    ▼
  Document ─── text: "من کتاب اوخویورام"
    │
    ├── script_detect          → script = "Arab"
    ├── embeddings (NLLB)      → sentence/document vectors
    └── translate (NLLB)       → sentence/document translation
           (src resolved from FLORES map: azb -> azb_Arab,
            tgt resolved from ISO-3: eng -> eng_Latn)
    │
    ▼
  Document ─── annotated with all layers

Processors and backends

Use these names in Pipeline(lang, processors=[...]); pick a backend with <name>_backend="..." and pass options as <name>_<option>=... (e.g. translate_tgt_lang="eng"). Dependencies are added automatically.

Processor	Backends (default first where the catalog has several)	Adds	Loaded by default
tokenize	rule, rule_arabic, stanza	sentences, tokens, words	✓
mwt	neural (rule-based), stanza	multi-word token expansion	✓ (where available)
morph	apertium, llm, apertium_llm	lemma, UPOS, feats (HFST / LLM / HFST + LLM choice)	✓ (apertium)
morph_neural	multilingual_glot500_morph	UPOS, feats, lemma	✓ if no morph
pos	stanza, multilingual_glot500_model	UPOS, XPOS, feats	✓
feats	multilingual_glot500_morph	UD features	✓
lemma	stanza, multilingual_glot500_morph	lemma	✓
depparse	stanza, multilingual_glot500_model	head, deprel	✓
ner	stanza (tur, kaz), llm (all)	BIO tags, entity spans	✓ (stanza)
embeddings	nllb	sentence/document vectors	–
translate	nllb, llm	sentence/document translation	–
sentiment	llm	sentence sentiment label	–
spellcheck	llm	sentence.correction, doc.corrected_text	–
sp_tokenize	sentencepiece	sp_tokens, sp_ids	–
asr	Omnilingual ASR (via Pipeline.from_audio)	transcript	–

turkicnlp.list_processors("kaz") lists what the catalog offers for a language; the llm backends work for every language and are not listed there. sentiment without a backend option uses llm. The placeholder backends tokenize=neural, morph=neural, pos=neural, lemma=neural, lemma=dictionary, depparse=neural and sentiment=neural are registered but not implemented; the pipeline never selects them by itself.

Which processor writes which field. Each word field has one owner per pipeline: the last processor in execution order that provides it (pos owns UPOS/XPOS, feats FEATS, lemma LEMMA, depparse HEAD/DEPREL; a full analyser such as morph owns the fields that no later step provides). In the default Turkish pipeline (morph, pos, lemma, depparse) Stanza therefore provides UPOS, FEATS and LEMMA. What other processors wrote is kept per word in word._provenance (e.g. word._provenance["morph"] holds the Apertium lemma, UPOS and FEATS), and FEATS from a different processor than UPOS are dropped for a word when that processor's own UPOS disagrees. When morph=apertium is in the default set, the Glot500 feats/lemma steps of the same language are left out.

Scripts without models. If a language has no models for the input script (e.g. Uzbek Cyrillic, Kazakh Latin), the pipeline transliterates the text to the script that has them and maps the results back: word texts and character offsets refer to the original input, lemmas are transliterated back, and doc._model_text holds the transliterated text.

Key Abstractions

• Document → Sentence → Token → Word hierarchy (maps to CoNLL-U)
• Processor ABC with PROVIDES, REQUIRES, NAME class attributes
• Pipeline orchestrator with dependency resolution and script-aware model loading
• ProcessorRegistry for pluggable backends (rule, Apertium, Stanza, Glot500, NLLB)
• ModelRegistry with remote catalog and local caching at ~/.turkicnlp/models/
• NLLB FLORES language map for ISO-3 to NLLB code resolution in translation (e.g. tuk -> tuk_Latn)

Model Storage Layout

~/.turkicnlp/models/
├── kaz/
│   ├── Cyrl/
│   │   ├── tokenize/rule/
│   │   └── morph/apertium/    ← GPL-3.0 (downloaded separately)
│   │       ├── kaz.automorf.hfst
│   │       ├── LICENSE
│   │       └── metadata.json
│   └── ...
├── tur/
│   └── Latn/
│       └── ...
├── multilingual/
│   ├── multilingual_glot500.pt           ← POS/DepParse checkpoint
│   └── multilingual_morph_glot500.pt     ← Morph analyzer checkpoint
├── huggingface/
│   ├── cis-lmu--glot500-base/            ← Shared Glot500 backbone
│   │   ├── config.json
│   │   ├── model.safetensors
│   │   └── ...
│   └── facebook--nllb-200-distilled-600M/
│       ├── config.json
│       ├── model.safetensors (or pytorch_model.bin)
│       ├── tokenizer.json
│       └── ...
└── stanza/, stanza_custom/                ← Stanza models

# Stanza models (official and custom) are stored under ~/.turkicnlp/models/stanza*/

Notes:

• Omnilingual ASR checkpoints are managed by fairseq2 and cached in ~/.cache/fairseq2/assets/.
• NLLB embeddings and translation use a shared Hugging Face model under ~/.turkicnlp/models/huggingface/.
• The NLLB model is downloaded once and reused across supported Turkic languages.
• The Glot500 backbone is shared between the POS/DepParse and Morph analyzer models under ~/.turkicnlp/models/huggingface/.
• Unlike Apertium/Stanza components, NLLB and Glot500 artifacts are not duplicated per language/script directory.

License

• Library code: Apache License 2.0
• Stanza models: Apache License 2.0 — managed by Stanza's own download mechanism
• Apertium FST data: GPL-3.0 — downloaded separately at runtime, never bundled in the pip package
• Omnilingual ASR models and code: Apache License 2.0 — downloaded by fairseq2 at runtime
• NLLB-200 model weights/tokenizer: CC-BY-NC-4.0 — downloaded from Hugging Face at runtime and reused from ~/.turkicnlp/models/huggingface/ (non-commercial license terms apply)

Development

git clone https://github.com/turkic-nlp/turkicnlp.git
cd turkicnlp
pip install -e ".[dev]"
python -m pytest        # "python -m" guarantees the pytest of the active environment

# Speech recognition tests run without the model (the backend is mocked);
# run the full suite in the speech + text environment as well:
source venv-asr/bin/activate && pip install pytest && python -m pytest

Contributing

Contributions are very welcome — from checking analyses in your language to adding new components. Read CONTRIBUTING.md for the development setup, how to add a language or a processor, tests, code style and the pull-request process, and browse ROADMAP.md for open tasks (items marked S are good first issues). Questions are welcome on Discord or in GitHub issues.

Acknowledgements

TurkicNLP builds on the work of many researchers and communities. We gratefully acknowledge the following:

Stanza

Stanza provides the pretrained neural models for tokenization, POS tagging, lemmatization, dependency parsing, and NER.

Peng Qi, Yuhao Zhang, Yuhui Zhang, Jason Bolton, and Christopher D. Manning. 2020. Stanza: A Python Natural Language Processing Toolkit for Many Human Languages. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations. [paper]

Universal Dependencies Treebanks

The Stanza models are trained on Universal Dependencies treebanks created by the following teams:

Turkish (UD_Turkish-IMST)

Umut Sulubacak, Memduh Gokirmak, Francis Tyers, Cagri Coltekin, Joakim Nivre, and Gulsen Cebiroglu Eryigit. Universal Dependencies for Turkish. COLING 2016. [paper]

Turkish (UD_Turkish-BOUN)

Utku Turk, Furkan Atmaca, Saziye Betul Ozates, Gozde Berk, Seyyit Talha Bedir, Abdullatif Koksal, Balkiz Ozturk Basaran, Tunga Gungor, and Arzucan Ozgur. Resources for Turkish Dependency Parsing: Introducing the BOUN Treebank and the BoAT Annotation Tool. Language Resources and Evaluation 56(1), 2022. [paper]

Turkish (UD_Turkish-FrameNet, KeNet, ATIS, Penn, Tourism)

Busra Marsan, Neslihan Kara, Merve Ozcelik, Bilge Nas Arican, Neslihan Cesur, Asli Kuzgun, Ezgi Saniyar, Oguzhan Kuyrukcu, and Olcay Taner Yildiz. Starlang Software and Ozyegin University. These treebanks cover diverse domains including FrameNet frames, WordNet examples, airline travel, Penn Treebank translations, and tourism reviews.

Kazakh (UD_Kazakh-KTB)

Aibek Makazhanov, Jonathan North Washington, and Francis Tyers. Towards a Free/Open-source Universal-dependency Treebank for Kazakh. TurkLang 2015. [paper]

Uyghur (UD_Uyghur-UDT)

Marhaba Eli (Xinjiang University), Daniel Zeman (Charles University), and Francis Tyers. [treebank]

Kyrgyz (UD_Kyrgyz-KTMU)

Ibrahim Benli. [treebank]

Ottoman Turkish (UD_Ottoman_Turkish-BOUN)

Saziye Betul Ozates, Tarik Emre Tiras, Efe Eren Genc, and Esma Fatima Bilgin Tasdemir. Dependency Annotation of Ottoman Turkish with Multilingual BERT. LAW-XVIII, 2024. [paper]

NER Datasets

Turkish NER (Starlang)

B. Ertopcu, A. B. Kanburoglu, O. Topsakal, O. Acikgoz, A. T. Gurkan, B. Ozenc, I. Cam, B. Avar, G. Ercan, and O. T. Yildiz. A New Approach for Named Entity Recognition. UBMK 2017. [paper]

Kazakh NER (KazNERD)

Rustem Yeshpanov, Yerbolat Khassanov, and Huseyin Atakan Varol (ISSAI, Nazarbayev University). KazNERD: Kazakh Named Entity Recognition Dataset. LREC 2022. [paper]

Glot500

The multilingual Glot500 model serves as the frozen backbone for TurkicNLP's neural POS/DepParse and Morph analyzer models.

ImaniGooghari, Ayyoob, Peiqin Lin, Amir Hossein Kargaran, Silvia Severini, Masoud Jalili Sabet, Nora Kassner, Chunlan Ma, Helmut Schmid, André Martins, François Yvon, and Hinrich Schütze. 2023. Glot500: Scaling Multilingual Corpora and Language Technology to 500 Languages. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). [paper]

GlotLID

The language detection model

Amir Hossein Kargaran, Ayyoob Imani, François Yvon, Hinrich Schütze (2023). GlotLID: Language Identification for Low-Resource Languages. The 2023 Conference on Empirical Methods in Natural Language Processing. Model

Wiktextract / Kaikki.org

Morphological training data for extended Turkic languages was extracted from Wiktionary using Wiktextract. The structured data is available at kaikki.org.

Tatu Ylonen. 2022. Wiktextract: Wiktionary as Machine-Readable Structured Data. In Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022). [paper]

UniMorph

The Universal Morphology (UniMorph) project provides morphological paradigms used for training and evaluating the multilingual morph analyzer across Turkic languages.

John Sylak-Glassman. 2016. The Composition and Use of the Universal Morphological Feature Schema (UniMorph Schema). Johns Hopkins University. [paper]

NLLB Embeddings & Machine Translation

TurkicNLP embeddings backend uses encoder pooling on:

facebook/nllb-200-distilled-600M

Reference:

NLLB Team, Marta R. Costa-jussà, et al. 2022. No Language Left Behind: Scaling Human-Centered Machine Translation. [paper]

Omnilingual ASR

Speech recognition uses Meta's Omnilingual ASR models:

facebookresearch/omnilingual-asr, facebook/omniASR-LLM-1B

Reference:

Omnilingual ASR Team. 2025. Omnilingual ASR: Open-Source Multilingual Speech Recognition for 1600+ Languages.

Common Turkic Alphabet

Common Turkic Alphabet

Uyghur Transliteration

Uyghur-Multi-Script-Converter

Reference:

Osman, T. Uyghur Multi-Script Converter (Version 1.0) [Computer software]

Other Organisations

• Apertium — morphological transducers covering 20+ Turkic languages
• SIGTURK — ACL Special Interest Group on Turkic Languages
• ISSAI — Institute of Smart Systems and Artificial Intelligence, Nazarbayev University, for Kazakh NLP resources
• Universal Dependencies — the framework and community behind Turkic treebanks
• Turkic Interlingua — resources for machine translation for Turkic languages
• Turkic UD - group working on harmonizing Turkic UD treebanks
• TurkLang - Conference on Computer Processing of Turkic Languages (2013-present)
• Turkic Academy - facilitates and encourages scientific and educational cooperation among Turkic countries and communities.
• Organization of Turkic States - primary objective is to foster comprehensive cooperation among Turkic States