TLL — Tanzania Local Language

TLL is a Python library ecosystem for Tanzanian local-language technology. The project starts with Kiswahili (sw) and Kisukuma (suk) and is designed to support additional Tanzanian languages without rewriting the core architecture.

This repository currently focuses on the foundational pieces required for a serious, reusable library:

• language registry
• dataset provenance and validation
• raw/cleaned/reviewed/verified data pipeline
• dictionary-oriented data access
• translation API with conservative behavior
• text normalization and tokenization hooks
• CLI and documentation scaffolding

Important: some features are intentionally scaffolded and documented as future work rather than pretending they are fully implemented.

Quick start

PowerShell:

python -m venv .venv
.\.venv\Scripts\Activate.ps1
python -m pip install --upgrade pip
pip install -e .

To install the published package from PyPI:

pip install Tan_language

The package includes the cleaned Sukuma dataset needed by the dictionary translation API, so LOCAL_LANGUAGE and the repository's root data/ folder are not required after installation.

Then:

from tll import get_language, list_languages

print(get_language("suk").name)
print([lang.code for lang in list_languages()])

Translation and chatbot usage

The translation API now uses the dataset-first algorithm from the LOCAL_LANGUAGE project:

1. exact match
2. word-by-word composition for phrases
3. fuzzy spelling match
4. explicit "not found" response instead of invented translations

PowerShell examples:

tan-language translate --from suk --to sw mwalimu
tan-language translate --from suk --to sw "mwalimu shule"
tan-language chat --from suk --to sw mwalimu

Python usage:

from tll import LocalLanguageChatbot, Translator

translator = Translator(source="suk", target="sw")
print(translator.translate("mwalimu shule"))

chatbot = LocalLanguageChatbot(language="suk")
print(chatbot.chat("mwalimu", source_language="suk", target_language="sw"))

The chatbot is grounded in the local dataset. grounded=True means the response came from a dataset match; a grounded=False response must be reviewed by a native speaker and should not be treated as a verified translation.

Optional offline LLM chatbot

For natural conversation, install the optional GGUF runtime:

pip install "Tan_language[local-llm]"

The first LLM request downloads the small Qwen instruct GGUF model into the Hugging Face cache. Later requests reuse that file and run locally without network access. Model weights are intentionally not included in the wheel. To pre-download or use a model file managed by your organization, pass its local path:

tll chat --llm --model-path .\models\assistant.gguf --from suk --to sw "Habari?"

Or use Python:

from tll import LocalLanguageChatbot

bot = LocalLanguageChatbot(backend="llm")
print(bot.chat("Habari?", source_language="sw", target_language="sw")["response"])
bot.reset()  # clear conversation history; the model stays loaded

Generation is deterministic by default (temperature=0, fixed seed). The LLM is not a verified translation engine: responses have grounded=False and should be reviewed for language accuracy.

Current validation workflow

python scripts/validate_dataset.py --dataset data/raw/suk/sukuma_deduplicated.csv

This validates the provided Kisukuma dataset without modifying the original raw file.

Reviewed and verified dataset workflow

python scripts/review_dataset.py --input data/cleaned/suk/cleaned_sukuma_dataset.csv --output-dir data/reviewed/suk --language suk

This creates the reviewed stage and then promotes the verified subset into a dedicated verified dataset. The library keeps all intermediate data, so the raw source remains untouched and the review decisions remain auditable.

Project status

This is Phase 1 of the roadmap: dataset inspection, architecture, language registry, and validation. The cleaning stage begins only after validation is complete and reviewed.

License

This project is licensed under the MIT license. Dataset licenses are handled separately and must be recorded carefully; if unknown, the metadata must state unknown.