ArabiNLP

A lightweight, dependency-free Python toolkit for Arabic Natural Language Processing (NLP), providing preprocessing, normalization, morphology, stemming, lemmatization, spell checking, and information retrieval.

Version 2.1.0

Features

• Text cleaning
• Arabic normalization
• Tokenization
• Stopword removal
• Sentence splitting
• Text statistics
• Word and character n-grams
• Keyword extraction
• Language detection
• Jaccard similarity
• Arabic Light Stemmer
• Root Extraction
• Lemmatization
• Morphological Segmentation
• Morphological Analysis
• Arabic Spell Checker
• TF-IDF Vectorizer
• BM25 Ranking

Installation

pip install arabinlp

Quick Start

from arabinlp import (
    normalize_arabic,
    remove_stopwords,
    tokenize,
    ArabicLightStemmer,
)

text = "أنا أحب تعلم الذكاء الاصطناعي في الجامعات."

# Normalize
text = normalize_arabic(text)

# Remove stopwords (expects text, returns filtered tokens)
filtered_tokens = remove_stopwords(text)

# Tokenize after converting tokens back to text
tokens = tokenize(" ".join(filtered_tokens))

# Stem
stemmer = ArabicLightStemmer()
stems = [stemmer.stem(token) for token in tokens]

print(stems)

Expected output

['احب', 'تعلم', 'ذكاء', 'اصطناع', 'جامع']

End-to-End NLP Pipeline

from arabinlp import (
    normalize_arabic,
    remove_stopwords,
    tokenize,
    ArabicLightStemmer,
    ArabicRootExtractor,
    ArabicLemmatizer,
)

text = "والمكتبات جميلة"

text = normalize_arabic(text)
filtered_tokens = remove_stopwords(text)
tokens = tokenize(" ".join(filtered_tokens))

stemmer = ArabicLightStemmer()
extractor = ArabicRootExtractor()
lemmatizer = ArabicLemmatizer()

for token in tokens:
    print(
        token,
        stemmer.stem(token),
        extractor.extract(token),
        lemmatizer.lemmatize(token),
    )

Python Support

• Python >= 3.9

Supported versions

• Python 3.9
• Python 3.10
• Python 3.11
• Python 3.12

Project Links

Homepage

https://github.com/IbtihalMakki/arabic-nlp-toolkit

Repository

https://github.com/IbtihalMakki/arabic-nlp-toolkit

Issues

https://github.com/IbtihalMakki/arabic-nlp-toolkit/issues

API Overview

Core Preprocessing

from arabinlp import (
    preprocess,
    normalize_arabic,
    tokenize,
    remove_stopwords,
    split_sentences,
    word_frequency,
    ngrams,
    character_ngrams,
    detect_language,
    text_statistics,
    extract_keywords,
    jaccard_similarity,
)

text = "السَّلَامُ عَلَيْكُمْ 😊 زوروا https://example.com"

cleaned = preprocess(text)

normalized = normalize_arabic(
    "إسلام وآثار",
    level="light",
)

tokens = tokenize(
    "السلام عليكم ورحمة الله",
)

filtered = remove_stopwords(
    "أنا أحب تعلم الذكاء الاصطناعي في الجامعة",
)

sentences = split_sentences(
    "السلام عليكم. كيف حالك؟ أنا بخير!",
)

freq = word_frequency(
    "الذكاء الذكاء الاصطناعي رائع",
)

word_bigrams = ngrams(
    "السلام عليكم ورحمة الله",
    2,
)

char_trigrams = character_ngrams(
    "السلام",
    3,
)

lang = detect_language(
    "Hello مرحبا",
)

stats = text_statistics(
    "السلام عليكم. كيف حالك؟",
)

keywords = extract_keywords(
    "الذكاء الاصطناعي رائع. الذكاء يتطور بسرعة.",
    top_k=2,
)

sim = jaccard_similarity(
    "أنا أحب الذكاء الاصطناعي",
    "الذكاء الاصطناعي رائع",
)

Cleaning Utilities

from arabinlp import (
    remove_urls,
    remove_mentions,
    remove_hashtags,
    remove_extra_spaces,
    remove_diacritics,
    remove_tatweel,
    remove_emojis,
)

print(remove_urls("Visit https://example.com"))

print(remove_mentions("@ibtihal hello"))

print(remove_hashtags("#AI is awesome"))

print(remove_extra_spaces("Hello     World"))

print(remove_diacritics("السَّلَامُ عَلَيْكُمْ"))

print(remove_tatweel("الســــلام"))

print(remove_emojis("Hello 😊"))

Stemming

from arabinlp import ArabicLightStemmer

stemmer = ArabicLightStemmer()

print(stemmer.stem("والمكتبات"))

Expected output

مكتب

Root Extraction

from arabinlp import ArabicRootExtractor

extractor = ArabicRootExtractor()

print(extractor.extract("المكتبات"))

Expected output

كتب

Lemmatization

from arabinlp import ArabicLemmatizer

lemmatizer = ArabicLemmatizer()

print(lemmatizer.lemmatize("المهندسون"))

print(
    lemmatizer.lemmatize_sentence(
        "المهندسون في الجامعات",
    )
)

print(
    lemmatizer.analyze(
        "المهندسون",
    )
)

Morphological Analysis

from arabinlp import (
    ArabicSegmenter,
    ArabicMorphAnalyzer,
)

segmenter = ArabicSegmenter()

print(segmenter.segment("وبالمكتبات"))

print(
    segmenter.segment_sentence(
        "وبالمكتبات كتابهما",
    )
)

analyzer = ArabicMorphAnalyzer()

print(analyzer.analyze("المكتبات"))

print(
    analyzer.analyze_sentence(
        "وبالمكتبات كتابهما",
    )
)

Spell Checking

from arabinlp import ArabicSpellChecker

checker = ArabicSpellChecker()

print(checker.is_correct("رائع"))

print(
    checker.distance(
        "راءع",
        "رائع",
    )
)

print(checker.suggest("راءع"))

print(
    checker.correct(
        "اللغه العربيه جميله",
    )
)

Expected output

اللغة العربية جميلة

TF-IDF

from arabinlp import ArabicTfidfVectorizer

docs = [
    "أنا أحب الذكاء الاصطناعي",
    "الذكاء الاصطناعي يعالج اللغة العربية",
    "معالجة اللغة الطبيعية مجال مهم",
]

vectorizer = ArabicTfidfVectorizer(
    ngram_range=(1, 2),
    min_df=1,
)

matrix = vectorizer.fit_transform(docs)

print(vectorizer.vocabulary_)
print(matrix)

BM25

from arabinlp import ArabicBM25

docs = [
    "أنا أحب الذكاء الاصطناعي",
    "الذكاء الاصطناعي يعالج اللغة العربية",
    "معالجة اللغة الطبيعية مجال مهم",
]

bm25 = ArabicBM25()

bm25.fit(docs)

print(
    bm25.get_scores(
        "الذكاء الاصطناعي",
    )
)

print(
    bm25.search(
        "الذكاء الاصطناعي",
        top_k=2,
    )
)

Examples

Additional runnable examples are available in the GitHub repository:

• examples/showcase.py
• examples/sample.txt
• examples/sample_errors.txt

Repository:

https://github.com/IbtihalMakki/arabic-nlp-toolkit

Running Tests

The automated test suite is available in the GitHub repository.

git clone https://github.com/IbtihalMakki/arabic-nlp-toolkit
cd arabic-nlp-toolkit
python -m pytest

Current Features (v2.0.0)

<<<<<<< HEAD ✅ Arabic text preprocessing

• Text cleaning
• Arabic normalization
• Tokenization
• Stopword removal
• Sentence splitting
• Word frequency
• N-grams
• Character N-grams
• Keyword extraction
• Language detection
• Text statistics
• Jaccard similarity

✔ 40 unit tests ✔ GitHub Actions CI ✔ PyPI package

Implemented (v2.1.0)

• Complete preprocessing pipeline
• Arabic normalization
• Cleaning utilities
• Tokenization
• Stopword removal
• Morphological segmentation
• Morphological analysis
• Arabic Light Stemmer
• Root Extraction
• Lemmatization
• Spell Checker
• TF-IDF
• BM25

All public APIs exported from arabinlp.__init__ are implemented and covered by automated tests.

Planned

• Named Entity Recognition (NER)
• AraBERT integration
• Additional pretrained Arabic NLP pipelines
• Larger benchmarking datasets

90403ac (Fix README examples and documentation for v2.1.0)

Contributing

Contributions, bug reports, feature requests, and pull requests are welcome.

License

<<<<<<< HEAD This project is licensed under the MIT License.

Author

Ibtihal Makki

AI Engineer • NLP Researcher • Generative AI

MIT License.

90403ac (Fix README examples and documentation for v2.1.0)