yakit 0.1.1

Yakut language text normalizer using Word2Vec embeddings

Yakit - Yakut Language Text Normalizer

A Python library for normalizing Yakut (Sakha) language text using Word2Vec embeddings.

Installation

pip install yakit

For automatic model downloading from Hugging Face Hub:

pip install yakit[download]

Quick Start

from yakit.normalizers import Word2VecNormalizer

# Initialize normalizer (auto-downloads model on first use)
normalizer = Word2VecNormalizer()

# Normalize text
text = "Мин сахалыы билэбин"
normalized = normalizer.normalize(text)
print(normalized)

Custom Model Path

If you have your own Word2Vec model:

from yakit.normalizers import Word2VecNormalizer

normalizer = Word2VecNormalizer(
    word2vec_path="/path/to/your/model.bin",
    training_data_path="/path/to/train_pairs.txt"  # optional
)

Command Line Interface

# Normalize text directly
yakit normalize "Мин сахалыы билэбин"

# Normalize a file
yakit normalize -i input.txt -o output.txt

# Download models manually
yakit download

# Show cache info
yakit info

What is Normalization?

Normalization converts text WITHOUT special Yakut characters to text WITH proper Yakut characters:

Input	Output
h →	һ
г →	ҕ (in certain positions)
н →	ҥ (in certain positions)
о →	ө (in certain positions)
у →	ү (in certain positions)

Performance

With optimized hyperparameters:

• Character Accuracy: 97.15%
• Word Accuracy: 92.09%
• Exact Match: 61.77%

Requirements

• Python 3.10–3.13 (3.14 not yet supported: gensim has no compatible build)
• gensim
• numpy
• tqdm

License

MIT