lemmaturk 0.1.0

Deep Learning based high-speed Turkish Lemmatizer

Lemmaturk: High-Speed Turkish Lemmatizer

Lemmaturk is a deep learning-based lemmatization library designed specifically for the Turkish language. Unlike traditional rule-based systems that rely on static dictionaries and strict morphological rules, Lemmaturk utilizes a Transformer-based Sequence-to-Sequence (Seq2Seq) architecture.

This approach allows the model to "generate" lemmas rather than selecting them from a list, making it highly robust against unknown words, spelling errors, and complex agglutinative structures. It is optimized for GPU inference, offering significant speed advantages for large-scale data processing.

Features

• Transformer Architecture: Uses a custom Encoder-Decoder model to understand morphological context.
• Generative Approach: Capable of predicting roots for words not present in the training set (Out-of-Vocabulary).
• High Performance: Fully compatible with CUDA for GPU acceleration.
• Robustness: Can handle minor OCR errors and non-standard input effectively.
• Easy Integration: Simple Python API designed for seamless integration into NLP pipelines.

Installation

You can install the package directly via pip:

pip install lemmaturk

Usage

The library provides a simple interface through the Lemmatizer class. It automatically detects if a GPU is available and moves the model to the appropriate device.

Basic Usage

from lemmaturk import Lemmatizer

# Initialize the lemmatizer
# This loads the model weights and vocabulary
lem = Lemmatizer()

# Analyze a single word
word = "kitaplaştıramadıklarımızdan"
root = lem.lemmatize(word)
print(f"Word: {word} -> Lemma: {root}")
# Output: Word: kitaplaştıramadıklarımızdan -> Lemma: kitap

Batch Processing Example

from lemmaturk import Lemmatizer

lem = Lemmatizer()

words = [
    "gidiyorum",
    "kalemlik",
    "gözlükçü",
    "yapılandırılması",
    "bilgisayarlarımız"
]

print("-" * 30)
print(f"{'Word':<20} | {'Lemma'}")
print("-" * 30)

for w in words:
    lemma = lem.lemmatize(w)
    print(f"{w:<20} | {lemma}")

Technical Details

• Architecture: Transformer (Seq2Seq) with Positional Encoding.
• Embedding Dimension: 256
• Hidden Dimension: 512
• Heads: 8
• Layers: 3 Encoder / 3 Decoder
• Training Data: Trained on a hybrid dataset consisting of academic treebanks, web-scraped literature, and OCR-digitized historical texts (approx. 500k+ samples).

Requirements

• Python >= 3.8
• PyTorch >= 2.0.0
• NumPy

License

This project is licensed under the MIT License.

Citation

If you use this tool in your research, please cite:

Mahmut. (2026). Lemmaturk: Deep Learning based high-speed Turkish Lemmatizer.
GitHub/PyPI Repository.