swahiliclean (Python)

Python port of the Swahili text preprocessing toolkit for NLP on Kiswahili text.

Quick Start

Installation

pip install swahiliclean

For development with tests:

pip install -e .[test]
python3 -m pytest

Basic Usage

from swahiliclean import preprocess_sw, tokenize_sw_words

# Full preprocessing pipeline
text = "manzi poa sana 2na tufanye kazi na sisi"
cleaned = preprocess_sw(text)
print(cleaned)

# Tokenization
tokens = tokenize_sw_words(text)
print(tokens)  # ["manzi", "poa", "sana", ...]

Batch Processing

All functions accept a list of strings. Files and directories have their own helpers:

sentences = [
    "manzi poa sana 2na tufanye kazi na sisi",
    "Habari za asubuhi? Nimekuja kwa ajili ya kazi.",
    "Na sisi tunaenda kwa shule leo."
]

# Process all at once
processed = preprocess_sw(sentences)
for original, cleaned in zip(sentences, processed):
    print(f"{original} -> {cleaned}")

# One file, or a directory of .txt files
from swahiliclean import preprocess_sw_file, preprocess_sw_corpus

preprocess_sw_file("notes.txt", output_path="notes.clean.txt", by_line=True)
cleaned_docs = preprocess_sw_corpus("corpus/")

Features

• Text Normalization: Lowercase, trim, collapse whitespace
• Stopword Removal: Remove common Swahili stopwords (~258 words)
• Slang Normalization: Replace slang with standard forms (~190 mappings)
• Typo Correction: Fix common typos (~422 corrections)
• Tokenization: Simple word tokenization
• Email & URL Removal: Detect and remove email addresses and URLs
• Text Statistics: Count and analyze stopwords, slang, typos with percentages
• Full Pipeline: Combine all steps with configurable options

Available Functions

Core Functions

• normalize_text() - Text normalization
• remove_stopwords() - Stopword removal
• normalize_slang() - Slang normalization
• correct_typos() - Typo correction
• tokenize_words() - Word tokenization
• preprocess_sw() - Full preprocessing pipeline

Aliases

The short names above are the same functions as:

• normalize_sw_text()
• remove_sw_stopwords()
• normalize_sw_slang()
• correct_sw_typos()
• tokenize_sw_words()

Batch

• preprocess_sw_file() - Clean one text file
• preprocess_sw_corpus() - Clean a list, a line-delimited file, or a directory of .txt files

Data

• prepare_swahili_data() - Load stopwords, slang, and typos, including optional files and custom entries
• load_stopwords(), load_slang(), load_typos() - Load one bundled or user dictionary

Email & URL Removal

• remove_emails() - Remove email addresses
• remove_urls() - Remove URLs
• remove_emails_and_urls() - Remove both emails and URLs
• remove_non_alphanumeric() - Remove non-alphanumeric characters
• remove_non_alphabetic() - Remove non-alphabetic characters
• remove_short_sw_words() - Remove short words by minimum length
• remove_emojis() - Remove emoji characters
• remove_unicode_characters() - Remove non-ASCII Unicode characters

Text Statistics

• count_sw_words() - Count words in text
• count_sw_stopwords() - Count stopwords found
• count_sw_slang() - Count slang terms found
• count_sw_typos() - Count typos found
• get_text_statistics() - Get comprehensive statistics
• get_cleaning_statistics() - Compare before/after cleaning

Custom Data

You can provide your own stopwords, typos, or slang mappings:

from swahiliclean import preprocess_sw, prepare_swahili_data

# Custom stopwords
result = preprocess_sw(
    "na sisi tunaenda yangu",
    custom_stopwords=["yangu", "yako"]
)

# Custom typos
result = preprocess_sw(
    "kwahiyo tunaenda",
    custom_typos=[("kwahiyo", "kwa hiyo")]
)

# Custom slang
result = preprocess_sw(
    "hii kazi ni mzito",
    custom_slang=[("mzito", "mzito sana")]
)

# Replace a bundled dictionary with a file. Custom entries still merge on top.
result = preprocess_sw(
    "na zzqxstop tunaenda",
    stopwords_path="my_stopwords.txt",
    custom_stopwords=["tunaenda"]
)

data = prepare_swahili_data(stopwords_path="my_stopwords.txt")

Configuration Options

Other controls:

• Step control: Enable or disable individual pipeline steps
• Character filtering: Remove non-alphanumeric or non-alphabetic characters, emojis, and non-ASCII characters
• Word length filtering: Remove short words using min_word_length
• Custom patterns: Use a custom regex for tokenization

Documentation

For complete documentation including:

• Full parameter reference for all functions
• Detailed examples and use cases
• Edge cases and behavior notes
• Advanced configuration options

See DOCUMENTATION.md for the complete manual.

Data

Bundled data files (CSV format):

• swahiliclean/data/Stopwords.csv - ~258 stopwords
• swahiliclean/data/Slangs.csv - ~190 slang mappings
• swahiliclean/data/Typos.csv - ~422 typo corrections

User dictionary files can be CSV or plain text. Stopwords are one word per line, or a CSV with a StopWords column. Slang and typos are source,target lines, or CSV columns such as Slang,Meaning and Typo,Word. Text pairs may also be separated with a tab, |, or =>.

Examples

Step-by-Step Processing

from swahiliclean import (
    normalize_sw_text,
    remove_sw_stopwords,
    correct_sw_typos,
    normalize_sw_slang
)

text = "manzi poa sana 2na tufanye kazi na sisi"

# Step 1: Normalize
normalized = normalize_sw_text(text)

# Step 2: Correct typos
corrected = correct_sw_typos(normalized)

# Step 3: Normalize slang
slang_normalized = normalize_sw_slang(corrected)

# Step 4: Remove stopwords
final = remove_sw_stopwords(slang_normalized)

Advanced Configuration

# Preserve case and newlines
result = preprocess_sw(
    "Habari YAKO\nleo",
    to_lower=False,
    preserve_newlines=True,
    remove_stop=False
)

# Case-sensitive dictionary matching
result = preprocess_sw(
    "Na sisi",
    ignore_case=False,
    to_lower=False
)

# Remove emails and URLs
result = preprocess_sw(
    "Contact test@example.com or visit https://site.com",
    remove_emails_flag=True,
    remove_urls_flag=True
)

Text Statistics

from swahiliclean import get_text_statistics, get_cleaning_statistics, preprocess_sw

# Get statistics before cleaning
text = "na sisi tunaenda kwa shule"
stats = get_text_statistics(text)
print(f"Stopwords: {stats['stopwords_count']} ({stats['stopwords_percentage']}%)")
print(f"Total detected: {stats['total_detected']} ({stats['total_percentage']}%)")

# Compare before and after
before = "na sisi tunaenda kwa shule"
after = preprocess_sw(before)
cleaning_stats = get_cleaning_statistics(before, after)
print(f"Words before: {cleaning_stats['words_before']}")
print(f"Words after: {cleaning_stats['words_after']}")
print(f"Words removed: {cleaning_stats['words_removed']} ({cleaning_stats['words_removed_percentage']}%)")

Testing

Run the test suite:

python3 -m pytest

Quick batch testing:

python3 tests/batch_demo.py

Authors

• Bernard Masua bhrmasua@gmail.com
• Noel Masasi noeliasmasasi@gmail.com

License

MIT License (see LICENSE).