tajiknlp 1.1.0

Production-ready NLP library for Tajik language (Cyrillic script)

🇹🇯 TajikNLP

Production-ready NLP library for the Tajik language (Cyrillic script).

TajikNLP provides a complete suite of text processing tools specifically designed for the unique challenges of the Tajik language (Cyrillic alphabet). Whether you're building a search engine, a chatbot, or conducting linguistic research, TajikNLP gives you robust, customizable components that work out of the box.

---

✨ Features

🧹 Text Cleaning (100% Complete)

• Remove URLs, emails, HTML tags, mentions, and hashtags
• Strip invisible Unicode characters and normalize whitespace
• Smart bracket cleaning (empty brackets and punctuation‑only brackets are removed)
• Remove spaces before punctuation marks

🔤 Normalization (100% Complete)

• Tajik Cyrillic standardization (e.g., љ → ҷ, ї → ӣ)
• Persian/Arabic digit conversion (۱۲۳ → 123)
• Quote and dash normalization (« » → ", — → -)
• Optional lowercasing and custom character replacement maps
• check_normalization() to validate text normalization

✂️ Tokenization (100% Complete)

• Regex‑based word tokenizer aware of Tajik‑specific letters (ғ, ӣ, қ, ӯ, ҳ, ҷ)
• Handles hyphenated compounds (китоб-ҳо)
• Preserves punctuation as separate tokens (optional)
• Provides accurate character offsets for each token
• Morpheme-level tokenization for linguistic analysis
• NEW: Subword tokenization (BPE) – pretrained BPE tokenizer for Tajik

📝 Sentence Splitting (100% Complete)

• Rule-based sentence boundary detection
• Protects Tajik-specific abbreviations (т.д., и.т.п., ва ғ.)
• Handles initials, decimals, dates, URLs, and emails

🏷️ Part‑of‑Speech Tagging (100% Complete)

• Rule‑based tagger with dictionary lookup and morphological fallback
• Category priority system from suffix rules
• Works on raw text without prior annotation

🔍 Morpheme Segmentation (100% Complete)

• Splits words into prefixes, root, and suffixes
• Uses linguistic rules from external JSON files (fully customizable)
• Returns structured morpheme data for each token

📚 Lemmatization (100% Complete)

• Hybrid approach: direct dictionary lookup + iterative affix stripping
• POS‑aware suffix removal for higher accuracy
• Built‑in dictionary of irregular forms

🌿 Stemming (100% Complete)

• Conservative-to-aggressive rule-based stemmer
• Safe mode for grammatical reductions
• Deep mode for derivational stripping
• Corpus-aware scoring for better accuracy

🛑 Stop Words Filtering (100% Complete)

• Comprehensive stop word list (100+ items)
• Automatically removes function words while preserving content words

🎯 Named Entity Recognition (NER) Ready (100% Complete)

• Character-level span alignment
• BIO tag generation for entity spans
• Offset preservation for training data preparation

🧠 Neural Pipeline (NEW in 1.1.0)

• Pretrained Word2Vec / FastText embeddings – 300‑dimensional vectors for Tajik words
• Cosine similarity – compute semantic similarity between words or custom vectors
• Most similar words – find nearest neighbours in the embedding space
• Automatic download of models from Hugging Face Hub (cached locally)

📊 Feature Extraction (NEW in 1.1.0)

• TajikCountVectorizer – Bag‑of‑Words with n‑gram support
• TajikTfidfVectorizer – TF‑IDF feature extraction
• Seamless integration with Doc objects or raw strings

🏷️ Keyword Classification (NEW in 1.1.0)

• Lightweight KeywordClassifier – assign categories based on user‑defined keywords
• Returns confidence scores and supports case‑insensitive matching
• Works directly with tokenized Doc objects (uses lemmas when available)

🛠️ Utilities (100% Complete)

• Script detection – identify if text is Cyrillic, Latin, Arabic, or mixed
• Language detection – distinguish Tajik from Russian, Persian, English, Arabic
• Quality scoring – evaluate how "Tajik‑like" a piece of text is
• Validation helpers – quickly check if a string is valid Tajik text
• Evaluation Metrics – Precision, Recall, F1, Edit Distance, Word Error Rate, BLEU score, Classification Report

---

📊 Project Status

Module	Status	Tests	Coverage
Preprocessing	✅ Complete	33/33	90%+
Tokenization	✅ Complete	22/22	88%
Subword Tokenization	✅ Complete	12/12	100%
Sentence Splitting	✅ Complete	7/7	93%
POS Tagging	✅ Complete	9/9	73%
Morpheme Segmentation	✅ Complete	8/8	72%
Lemmatization	✅ Complete	23/23	85%
Stemming	✅ Complete	6/6	85%
Stop Words Filtering	✅ Complete	6/6	95%
NER / Alignment	✅ Complete	5/5	84%
Script Detection	✅ Complete	47/47	95%
Validation & Metrics	✅ Complete	18/18	98%
Embeddings	✅ Complete	15/15	83%
Feature Extraction	✅ Complete	13/13	96%
Keyword Classification	✅ Complete	20/20	98%
Overall	✅ 100%	525	96%

---

📦 Installation

TajikNLP requires Python 3.8 or later.

Basic Installation

pip install tajiknlp

With Neural Components (Word2Vec, BPE tokenizer)

pip install "tajiknlp[neural]"

Development Installation

pip install "tajiknlp[dev]"

---

🧪 Try It Now in Google Colab

Click the badge below to run a complete demo notebook directly in your browser:

(Make sure the notebook is placed at examples/demo/tajiknlp_demo.ipynb in your repository)

---

🚀 Quick Start

Load the default pipeline

from tajiknlp import load_pipeline

pipe = load_pipeline("default")
text = "Китобҳоямонро хондам, аммо нафаҳмидам."
doc = pipe(text)

for token in doc.tokens:
    print(f"{token.text:<15} → lemma: {token.lemma:<10} pos: {token.pos}")

Output:

китобҳоямонро   → lemma: китоб      pos: NOUN
хондам          → lemma: хондан     pos: VERB
,               → lemma: ,          pos: PUNCT
аммо            → lemma: аммо       pos: CONJ
нафаҳмидам      → lemma: фаҳмидан   pos: VERB
.               → lemma: .          pos: PUNCT

Available Pipeline Presets

Preset	Components
minimal	Cleaner + Tokenizer + Morpheme splitter
default	Full preprocessing + POS + Morphemes + Stopwords + Lemmatizer
full	Alias for default
stemming	Same as default but with stemmer instead of lemmatizer
ner	Optimized for NER (preserves case)
neural	BPE tokenizer + Word2Vec embeddings (requires [neural])

Neural Pipeline Example

nlp = load_pipeline("neural")
doc = nlp("Эргашбой Мирзоевич Муҳамадиев - риёзидони бузурги тоҷик.")
for token in doc:
    vec = token.metadata.get("vector")
    if vec:
        print(f"{token.text}: vector dim={len(vec)}")

Feature Extraction

from tajiknlp import load_pipeline, TajikCountVectorizer

pipe = load_pipeline("default")
docs = [pipe("Ман китоб хондам."), pipe("Ту китоб нахонди.")]
cv = TajikCountVectorizer(ngram_range=(1, 2))
X = cv.fit_transform(docs)
print(cv.get_feature_names_out())

Keyword Classification

from tajiknlp import KeywordClassifier

clf = KeywordClassifier()
clf.add_category("спорт", {"футбол", "варзиш", "бозӣ"})
clf.add_category("сиёсат", {"президент", "интихобот", "ҳукумат"})
print(clf.predict("Дастаи футбол бозӣ бурд."))  # 'спорт'

---

📖 Documentation

Full documentation is available at TajikNLPWorld.github.io/tajiknlp.

It includes:

• Detailed API references
• Component configuration guides
• Advanced usage examples
• NER and alignment tutorials

---

🧪 Running Tests

If you cloned the repository, you can run the test suite with:

# Install development dependencies
pip install -e ".[dev]"

# Run all tests
pytest tests/ -v

# Run with coverage report
pytest tests/ --cov=tajiknlp --cov-report=html

---

🤝 Contributing

We welcome contributions! Please see CONTRIBUTING.md for guidelines on:

• Setting up the development environment
• Coding standards and style guides
• Testing requirements
• Pull request process

---

📄 License

This project is licensed under the MIT License – see the LICENSE file for details.

Copyright (c) 2026 TajikNLPWorld Copyright (c) 2026 Arabov Mullosharaf Kurbonovich

---

🙏 Acknowledgements

TajikNLP was inspired by the need for robust, open‑source NLP tools for low‑resource languages. Special thanks to all contributors and the Tajik linguistic community.

---

📊 Citation

If you use TajikNLP in your research, please cite:

@software{tajiknlp2026,
  author = {Arabov, Mullosharaf Kurbonovich},
  title = {TajikNLP: Production-ready NLP library for Tajik language},
  year = {2026},
  publisher = {GitHub},
  url = {https://github.com/TajikNLPWorld/tajiknlp}
}

---

Made with ❤️ for the Tajik language.