low-resource-nlp-toolkit 0.2.0

A robust Python toolkit for low-resource African language pre-processing, emotion labels, evaluation, and routing.

Low-Resource NLP Toolkit

A public, research-facing Python toolkit for African language pre-processing, emotion-label mapping, evaluation, and language/dialect routing.

The project is designed as a safe open-source wrapper around the kinds of NLP engineering problems that appear in low-resource and multilingual AI research: noisy text, code-switching, uneven label taxonomies, small datasets, and evaluation that must be transparent.

Status: 0.2.0 release. Local checks, CI, isolated wheel builds, metadata checks, and install tests pass.

Why This Exists

Low-resource NLP projects often spend too much time rebuilding the same foundations before modelling begins. This toolkit provides a dependable base layer:

• Text normalisation for noisy social, conversational, and cultural text.
• Lightweight African language routing for Yoruba, Igbo, Hausa, Nigerian Pidgin, Swahili, and English.
• Evidence-first code-switch audits that expose token routes, spans, and abstentions.
• Emotion label harmonisation across categorical and valence-arousal formats.
• Evaluation utilities for classification and routing experiments.
• A CLI and examples that run without downloading model weights.
• Extension points for transformer or embedding backends when a project needs heavier models.

Architecture

flowchart LR
    A["Raw multilingual text"] --> B["Normaliser"]
    B --> C["Tokeniser"]
    C --> D["Language router"]
    C --> E["Emotion label mapper"]
    D --> F["Route decision + confidence"]
    E --> G["Canonical emotion / valence-arousal"]
    F --> H["Evaluation reports"]
    G --> H

Quick Start

python3 -m venv .venv
source .venv/bin/activate
python -m pip install low-resource-nlp-toolkit
low-resource-nlp --version

Route a text sample:

low-resource-nlp route "abeg make una help me check this model output"

Audit code-switched language evidence:

low-resource-nlp audit "abeg make una check this model output"

Normalise text:

low-resource-nlp normalise "Ẹ káàrọ̀!!! Visit https://example.com @user"

Map an emotion label:

low-resource-nlp label joy

Run tests:

make check

Without make:

python3 scripts/quality_gate.py
PYTHONPATH=src python3 -m unittest discover -s tests

Python Usage

from low_resource_nlp import (
    LexicalLanguageRouter,
    audit_code_switching,
    label_to_valence_arousal,
    normalise_text,
)

text = normalise_text("Ẹ káàrọ̀, báwo ni?")
decision = LexicalLanguageRouter.default().route(text)
audit = audit_code_switching("abeg make una check this model output")
emotion = label_to_valence_arousal("joy")

print(decision.language_code, decision.confidence)
print(audit.language_mix, audit.warnings)
print(emotion)

Current Scope

The first public release deliberately avoids bundling private datasets or model weights. The core is deterministic, inspectable, and dependency-light. Optional embedding and transformer backends are outside the current core package.

Supported core modules:

• normalisation: Unicode-aware text cleaning, URL/user normalisation, tokenisation, repeated-character handling.
• routing: script-aware and lexicon-assisted language routing.
• audit: token-level code-switch audits with spans, evidence, and abstention warnings.
• labels: canonical emotion labels and valence-arousal mapping.
• evaluation: precision, recall, F1, macro/micro summaries, and confusion matrices.
• datasets: simple CSV/JSONL readers for experiment scaffolding.

Public Project Materials

• Changelog
• Contributing guide
• Documentation index
• Novelty review
• 0.1.0 release plan
• Adoption notes
• Model card template
• Data statement template
• Citation metadata

Responsible AI Notes

This toolkit is for research and prototyping. Language, dialect, and emotion labels are socially and culturally sensitive. Do not treat routing or emotion predictions as identity labels, clinical assessments, or ground truth. Always evaluate with speakers, domain experts, and context-specific data.

External Use

External use signals should be public and verifiable: issues from real users, pull requests, tutorial use, workshop demos, citations, package downloads, or adoption by a lab/community project. Self-generated activity should not be counted as impact.