swahiliclean 0.2.0

A comprehensive Python package for preprocessing Swahili (Kiswahili) text data. Provides functions for text normalization, stopword removal, slang normalization, typo correction, and tokenization. Includes curated datasets of Swahili stopwords, slang terms, and common typos. Designed for NLP tasks on Swahili text corpora.

swahiliclean (Python)

Python port of the Swahili text preprocessing toolkit for NLP on Kiswahili text.

Quick Start

Installation

pip install swahiliclean

For development with tests:

pip install -e .[test]
python3 -m pytest

Basic Usage

from swahiliclean import preprocess_sw, tokenize_sw_words

# Full preprocessing pipeline
text = "manzi poa sana 2na tufanye kazi na sisi"
cleaned = preprocess_sw(text)
print(cleaned)

# Tokenization
tokens = tokenize_sw_words(text)
print(tokens)  # ["manzi", "poa", "sana", ...]

Batch Processing

All functions support batch processing with lists:

sentences = [
    "manzi poa sana 2na tufanye kazi na sisi",
    "Habari za asubuhi? Nimekuja kwa ajili ya kazi.",
    "Na sisi tunaenda kwa shule leo."
]

# Process all at once
processed = preprocess_sw(sentences)
for original, cleaned in zip(sentences, processed):
    print(f"{original} -> {cleaned}")

Features

• Text Normalization: Lowercase, trim, collapse whitespace
• Stopword Removal: Remove common Swahili stopwords (~258 words)
• Slang Normalization: Replace slang with standard forms (~190 mappings)
• Typo Correction: Fix common typos (~422 corrections)
• Tokenization: Simple word tokenization
• Email & URL Removal: Detect and remove email addresses and URLs
• Text Statistics: Count and analyze stopwords, slang, typos with percentages
• Full Pipeline: Combine all steps with configurable options

Available Functions

Core Functions

• normalize_sw_text() - Text normalization
• remove_sw_stopwords() - Stopword removal
• normalize_sw_slang() - Slang normalization
• correct_sw_typos() - Typo correction
• tokenize_sw_words() - Word tokenization
• preprocess_sw() - Full preprocessing pipeline

Email & URL Removal

• remove_emails() - Remove email addresses
• remove_urls() - Remove URLs
• remove_emails_and_urls() - Remove both emails and URLs
• remove_non_alphanumeric() - Remove non-alphanumeric characters
• remove_non_alphabetic() - Remove non-alphabetic characters
• remove_short_sw_words() - Remove short words by minimum length
• remove_emojis() - Remove emoji characters
• remove_unicode_characters() - Remove non-ASCII Unicode characters

Text Statistics

• count_sw_words() - Count words in text
• count_sw_stopwords() - Count stopwords found
• count_sw_slang() - Count slang terms found
• count_sw_typos() - Count typos found
• get_text_statistics() - Get comprehensive statistics
• get_cleaning_statistics() - Compare before/after cleaning

Custom Data

You can provide your own stopwords, typos, or slang mappings:

from swahiliclean import preprocess_sw

# Custom stopwords
result = preprocess_sw(
    "na sisi tunaenda yangu",
    custom_stopwords=["yangu", "yako"]
)

# Custom typos
result = preprocess_sw(
    "kwahiyo tunaenda",
    custom_typos=[("kwahiyo", "kwa hiyo")]
)

# Custom slang
result = preprocess_sw(
    "hii kazi ni mzito",
    custom_slang=[("mzito", "mzito sana")]
)

Configuration Options

All functions support extensive configuration:

• Case sensitivity: Control case-sensitive matching
• Word boundaries: Toggle word boundary matching for stopwords
• Case preservation: Preserve original case when needed
• Whitespace handling: Preserve newlines/tabs or normalize them
• Custom patterns: Use custom regex patterns for tokenization
• Step control: Enable/disable individual pipeline steps
• Character filtering: Remove non-alphanumeric/non-alphabetic characters
• Emoji/Unicode filtering: Remove emojis and non-ASCII Unicode characters
• Word length filtering: Remove short words using minimum length

Documentation

For complete documentation including:

• Full parameter reference for all functions
• Detailed examples and use cases
• Edge cases and behavior notes
• Advanced configuration options

See DOCUMENTATION.md for the complete manual.

Data

Bundled data files (CSV format):

• swahiliclean/data/Stopwords.csv - ~258 stopwords
• swahiliclean/data/Slangs.csv - ~190 slang mappings
• swahiliclean/data/Typos.csv - ~422 typo corrections

Examples

Step-by-Step Processing

from swahiliclean import (
    normalize_sw_text,
    remove_sw_stopwords,
    correct_sw_typos,
    normalize_sw_slang
)

text = "manzi poa sana 2na tufanye kazi na sisi"

# Step 1: Normalize
normalized = normalize_sw_text(text)

# Step 2: Correct typos
corrected = correct_sw_typos(normalized)

# Step 3: Normalize slang
slang_normalized = normalize_sw_slang(corrected)

# Step 4: Remove stopwords
final = remove_sw_stopwords(slang_normalized)

Advanced Configuration

# Preserve case and newlines
result = preprocess_sw(
    "Habari YAKO\nleo",
    lowercase=False,
    preserve_newlines=True,
    remove_stop=False
)

# Case-sensitive processing
result = preprocess_sw(
    "Na sisi",
    stopwords_case_sensitive=True,
    slang_case_sensitive=True
)

# Remove emails and URLs
result = preprocess_sw(
    "Contact [e-mail] or visit https://site.com",
    remove_emails_flag=True,
    remove_urls_flag=True
)

Text Statistics

from swahiliclean import get_text_statistics, get_cleaning_statistics, preprocess_sw

# Get statistics before cleaning
text = "na sisi tunaenda kwa shule"
stats = get_text_statistics(text)
print(f"Stopwords: {stats['stopwords_count']} ({stats['stopwords_percentage']}%)")
print(f"Total detected: {stats['total_detected']} ({stats['total_percentage']}%)")

# Compare before and after
before = "na sisi tunaenda kwa shule"
after = preprocess_sw(before)
cleaning_stats = get_cleaning_statistics(before, after)
print(f"Words before: {cleaning_stats['words_before']}")
print(f"Words after: {cleaning_stats['words_after']}")
print(f"Words removed: {cleaning_stats['words_removed']} ({cleaning_stats['words_removed_percentage']}%)")

Testing

Run the test suite:

python3 -m pytest

Quick batch testing:

python3 tests/batch_demo.py

Authors

• Bernard Masua [e-mail]
• Noel Masasi [e-mail]

License

MIT License (see LICENSE).