tajiknlp 1.0.0

Production-ready NLP library for Tajik language (Cyrillic script)

🇹🇯 TajikNLP

Production-ready NLP library for the Tajik language (Cyrillic script).

TajikNLP provides a complete suite of text processing tools specifically designed for the unique challenges of the Tajik language (Cyrillic alphabet). Whether you're building a search engine, a chatbot, or conducting linguistic research, TajikNLP gives you robust, customizable components that work out of the box.

---

✨ Features (v1.0.0)

🧹 Text Cleaning

• Remove URLs, emails, HTML tags, mentions, and hashtags
• Strip invisible Unicode characters and normalize whitespace
• Smart bracket cleaning (empty brackets and punctuation‑only brackets are removed)
• Remove spaces before punctuation marks

🔤 Normalization

• Tajik Cyrillic standardization (e.g., љ → ҷ, ї → ӣ)
• Persian/Arabic digit conversion (۱۲۳ → 123)
• Quote and dash normalization (« » → ", — → -)
• Optional lowercasing and custom character replacement maps

✂️ Tokenization

• Regex‑based word tokenizer aware of Tajik‑specific letters (ғ, ӣ, қ, ӯ, ҳ, ҷ)
• Handles hyphenated compounds (китоб-ҳо)
• Preserves punctuation as separate tokens (optional)
• Provides accurate character offsets for each token
• Morpheme-level tokenization for linguistic analysis

📝 Sentence Splitting

• Rule-based sentence boundary detection
• Protects Tajik-specific abbreviations (т.д., и.т.п., ва ғ.)
• Handles initials, decimals, dates, URLs, and emails

🏷️ Part‑of‑Speech Tagging

• Rule‑based tagger with dictionary lookup and morphological fallback
• Category priority system from suffix rules
• Works on raw text without prior annotation

🔍 Morpheme Segmentation

• Splits words into prefixes, root, and suffixes
• Uses linguistic rules from external JSON files (fully customizable)
• Returns structured morpheme data for each token

📚 Lemmatization

• Hybrid approach: direct dictionary lookup + iterative affix stripping
• POS‑aware suffix removal for higher accuracy
• Built‑in dictionary of irregular forms

🌿 Stemming

• Conservative-to-aggressive rule-based stemmer
• Safe mode for grammatical reductions
• Deep mode for derivational stripping
• Corpus-aware scoring for better accuracy

🛑 Stop Words Filtering

• Comprehensive stop word list (100+ items)
• Automatically removes function words while preserving content words

🎯 Named Entity Recognition (NER) Ready

• Character-level span alignment
• BIO tag generation for entity spans
• Offset preservation for training data preparation

🛠️ Utilities

• Script detection – identify if text is Cyrillic, Latin, Arabic, or mixed
• Language detection – distinguish Tajik from Russian, Persian, English, Arabic
• Quality scoring – evaluate how "Tajik‑like" a piece of text is
• Validation helpers – quickly check if a string is valid Tajik text

---

📦 Installation

TajikNLP requires Python 3.8 or later.

Basic Installation

pip install tajiknlp

Development Installation

pip install "tajiknlp[dev]"

Full Installation (with ML dependencies)

pip install "tajiknlp[full]"

---

🚀 Quick Start

Load the default pipeline

from tajiknlp import load_pipeline

# Create a pipeline with cleaner, normalizer, tokenizer, POS tagger,
# morpheme splitter, stopword filter, and lemmatizer.
pipe = load_pipeline("default")

text = "Китобҳоямонро хондам, аммо нафаҳмидам."
doc = pipe(text)

# Inspect the results
for token in doc.tokens:
    print(f"{token.text:<15} → lemma: {token.lemma:<10} pos: {token.pos}")

Output:

китобҳоямонро   → lemma: китоб      pos: NOUN
хондам          → lemma: хондан     pos: VERB
,               → lemma: ,          pos: PUNCT
аммо            → lemma: аммо       pos: CONJ
нафаҳмидам      → lemma: фаҳмидан   pos: VERB
.               → lemma: .          pos: PUNCT

Available Pipeline Presets

Preset	Components
minimal	Cleaner + Tokenizer + Morpheme splitter
default	Full preprocessing + POS + Morphemes + Stopwords + Lemmatizer
full	Alias for default
stemming	Same as default but with stemmer instead of lemmatizer
ner	Optimized for NER (preserves case)

Use individual components

from tajiknlp import Doc
from tajiknlp.components.tokenizers import RegexTokenizer

tokenizer = RegexTokenizer(lowercase=True, keep_punct=True)
doc = tokenizer(Doc(text="Китоб-ҳо хондам!"))
print([t.text for t in doc.tokens])
# Output: ['китоб-ҳо', 'хондам', '!']

Validate and score text quality

from tajiknlp import quality_score, detect_script

score = quality_score("Ман китоб хондам.")
print(f"Score: {score['score']}")        # e.g., 0.78
print(f"Valid: {score['is_valid']}")     # True
print(f"Language: {score['language']}")  # tajik

script = detect_script("Салом!")
print(script)  # Script.CYRILLIC

Custom pipeline

from tajiknlp import TajikPipeline
from tajiknlp.components.cleaners import TextCleaner
from tajiknlp.components.tokenizers import RegexTokenizer
from tajiknlp.components.stemmers import DictStemmer

# Build custom pipeline
pipe = TajikPipeline()
pipe.add_component(TextCleaner())
pipe.add_component(RegexTokenizer())
pipe.add_component(DictStemmer(deep_stemming=True))

doc = pipe("Ман китобҳоро хондам.")

---

📖 Documentation

Full documentation is available at CodeHunterOfficial.github.io/tajiknlp.

It includes:

• Detailed API references
• Component configuration guides
• Advanced usage examples
• NER and alignment tutorials

---

📁 Project Structure

tajiknlp/
├── src/tajiknlp/           # Main package source
│   ├── alignment/          # Span alignment utilities
│   ├── components/         # NLP components
│   │   ├── cleaners/       # Text cleaning
│   │   ├── embeddings/     # Word embeddings
│   │   ├── filters/        # Token filtering
│   │   ├── lemmatizers/    # Lemmatization
│   │   ├── normalizers/    # Text normalization
│   │   ├── sentencizers/   # Sentence splitting
│   │   ├── stemmers/       # Stemming
│   │   ├── taggers/        # POS tagging
│   │   └── tokenizers/     # Tokenization
│   ├── core/               # Base classes
│   ├── data/               # Static resources
│   ├── pipeline/           # Pipeline system
│   ├── resources/          # Resource management
│   └── utils/              # Utility functions
├── tests/                  # Test suite
├── examples/               # Usage examples
├── docs/                   # Documentation
└── pyproject.toml          # Project configuration

---

🧪 Running Tests

If you cloned the repository, you can run the test suite with:

# Install development dependencies
pip install -e ".[dev]"

# Run all tests
pytest tests/ -v

# Run with coverage report
pytest tests/ --cov=tajiknlp --cov-report=html

---

🤝 Contributing

We welcome contributions! Please see CONTRIBUTING.md for guidelines on:

• Setting up the development environment
• Coding standards and style guides
• Testing requirements
• Pull request process

---

📄 License

This project is licensed under the MIT License – see the LICENSE file for details.

Copyright (c) 2026 CodeHunterOfficial Copyright (c) 2026 Arabov Mullosharaf Kurbonovich

---

🙏 Acknowledgements

TajikNLP was inspired by the need for robust, open‑source NLP tools for low‑resource languages. Special thanks to all contributors and the Tajik linguistic community.

---

📊 Citation

If you use TajikNLP in your research, please cite:

@software{tajiknlp2026,
  author = {Arabov, Mullosharaf Kurbonovich},
  title = {TajikNLP: Production-ready NLP library for Tajik language},
  year = {2026},
  publisher = {GitHub},
  url = {https://github.com/CodeHunterOfficial/tajiknlp}
}

---

Made with ❤️ for the Tajik language.