🏝️ NusaNLP: Low-Resource NLP Toolkit for West Nusa Tenggara, Indonesia

NusaNLP is an open-source Natural Language Processing (NLP) toolkit engineered specifically for the regional languages of West Nusa Tenggara (Nusa Tenggara Barat / NTB) and Bahasa Indonesia:

1. Bahasa Sasak (sasak) — Spoken by ~3 million people across the island of Lombok.
2. Bahasa Samawa / Sumbawa (samawa) — Spoken in Sumbawa and West Sumbawa regencies.
3. Bahasa Mbojo / Bima (mbojo) — Spoken in Bima Regency, Bima City, and Dompu Regency.
4. Bahasa Indonesia (indonesian) — Fully integrated with tokenization, clitic handling, stopword filtering, morphological stemming, language identification (100% test accuracy), and bilingual translation alignment.

NusaNLP is built to be production-ready, modular, extensible via a plugin language registry, and research-grade.

---

📦 Installation

From Source

git clone https://github.com/kodetr/nusanlp.git
cd nusanlp
pip install -e ".[dev]"

Optional Dependencies

# REST API (FastAPI + Uvicorn)
pip install -e ".[api]"

# Machine Learning additions
pip install -e ".[ml]"

---

🚀 Quick Start

1. Unified Pipeline (with Auto-Detection)

import nusanlp
from nusanlp import Pipeline

# Initialize pipeline with automatic language detection
nlp = Pipeline(language="auto")

result = nlp.process("Tiang lalo mangan leq bale")
print(result)

Output:

{
  "language": "sasak",
  "original_text": "Tiang lalo mangan leq bale",
  "normalized_text": "tiang lalo mangan leq bale",
  "tokens": ["tiang", "lalo", "mangan", "leq", "bale"],
  "tokens_without_stopwords": ["lalo", "mangan", "bale"],
  "analysis": [
    {"word": "tiang", "root": "tiang", "prefix": null, "suffix": null, "features": []},
    {"word": "lalo", "root": "lalo", "prefix": null, "suffix": null, "features": []},
    {"word": "mangan", "root": "ang", "prefix": "m", "suffix": "an", "features": ["SUFF:an", "PREF:m"]},
    {"word": "leq", "root": "leq", "prefix": null, "suffix": null, "features": []},
    {"word": "bale", "root": "bal", "prefix": null, "suffix": "e", "features": ["SUFF:e"]}
  ],
  "pos_tags": [
    ["tiang", "NOUN"],
    ["lalo", "VERB"],
    ["mangan", "VERB"],
    ["leq", "ADP"],
    ["bale", "NOUN"]
  ]
}

2. Language Detection

Detect whether a sentence is Sasak, Samawa, Mbojo, or Indonesian:

import nusanlp

res = nusanlp.detect_language("Tiang lalo leq peken")
print(res)
# {'language': 'sasak', 'confidence': 1.0}

res_smw = nusanlp.detect_language("Kaji lalo tama ko peken ling ina")
print(res_smw)
# {'language': 'samawa', 'confidence': 1.0}

res_mbj = nusanlp.detect_language("Nahu ngaha di uma labo ita")
print(res_mbj)
# {'language': 'mbojo', 'confidence': 1.0}

3. Text Normalization

# Lowercasing, whitespace, repeated character & slang normalization
norm = nusanlp.normalize("AQ   LEKOOOOK  GATI", language="sasak")
print(norm)
# Output: "aku lekook gati"

4. Modular Tokenizer & Clitic Separation

tokens = nusanlp.tokenize("Kaji lalo-mo ko peken", language="samawa")
print(tokens)
# Output: ['Kaji', 'lalo', '-mo', 'ko', 'peken']

5. Stopword Removal

clean = nusanlp.remove_stopwords("nahu ngaha di uma labo ita", language="mbojo")
print(clean)
# Output: ['ngaha', 'uma']

6. Multilingual Dictionary Lookup

entry = nusanlp.lookup("mangan", language="sasak")
print(entry)
# {'word': 'mangan', 'language': 'sasak', 'translations': {'indonesian': 'makan'}, 'pos': 'VERB', 'lemma': 'mangan'}

entry_smw = nusanlp.lookup("balong", language="samawa")
print(entry_smw)
# {'word': 'balong', 'language': 'samawa', 'translations': {'indonesian': 'baik / bagus', 'english': 'good'}, 'pos': 'ADJ', 'lemma': 'balong'}

7. Machine Learning & TF-IDF Integration (Scikit-Learn)

NusaNLP seamlessly integrates into TfidfVectorizer and CountVectorizer, providing morphological lemmatization and low-resource stopword removal to prevent vocabulary explosion:

from sklearn.feature_extraction.text import TfidfVectorizer
import nusanlp

corpus = [
    "Tiang lalo mangan leq bale adat Sasak.",
    "Kaji mangan leq bale adat Samawa.",
    "Nahu ngaha di uma rime Mbojo."
]

# Method A: Use NusaNLP custom tokenizer callable directly
vectorizer = TfidfVectorizer(tokenizer=nusanlp.get_tfidf_tokenizer(), lowercase=False)
X = vectorizer.fit_transform(corpus)
print("TF-IDF Features:", vectorizer.get_feature_names_out())

# Method B: Preprocess texts beforehand
clean_docs = [nusanlp.preprocess_for_tfidf(doc) for doc in corpus]
# clean_docs: ['mangan bale adat sasak', 'mangan bale adat samawa', 'ngaha uma rime mbojo']

---

🏛️ Architecture & Project Structure

nusanlp/
│
├── src/
│   ├── nusanlp/
│   │   ├── __init__.py                 # Unified API exports
│   │   ├── core/
│   │   │   ├── pipeline.py             # Unified Pipeline engine
│   │   │   ├── language.py             # BaseLanguage & LanguageRegistry
│   │   │   ├── config.py               # Pipeline & Normalizer configs
│   │   │   └── types.py                # Pydantic models & enums
│   │   │
│   │   ├── languages/
│   │   │   ├── base.py                 # AbstractLanguageHandler
│   │   │   ├── sasak/                  # Sasak modules
│   │   │   ├── samawa/                 # Samawa (Sumbawa) modules
│   │   │   ├── mbojo/                  # Mbojo (Bima) modules
│   │   │   └── indonesian/             # Indonesian language module
│   │   │
│   │   ├── preprocessing/
│   │   │   ├── normalize.py            # Multi-language normalization
│   │   │   └── clean.py                # Text cleaning utilities
│   │   │
│   │   ├── tokenization/
│   │   │   └── tokenizer.py            # RuleBasedTokenizer
│   │   │
│   │   ├── morphology/
│   │   │   └── analyzer.py             # MorphologicalAnalyzer
│   │   │
│   │   ├── lexicon/
│   │   │   ├── dictionary.py           # Multilingual dictionary
│   │   │   └── loader.py               # JSON/CSV loaders
│   │   │
│   │   ├── language_detection/
│   │   │   ├── detector.py             # detect_language API
│   │   │   └── model.py                # N-gram Statistical Classifier
│   │   │
│   │   ├── cli/
│   │   │   └── main.py                 # CLI entry point
│   │   │
│   │   ├── api/
│   │   │   └── app.py                  # FastAPI REST API
│   │   │
│   │   └── utils/
│   │       ├── loaders.py              # Resource discovery
│   │       └── validators.py           # Pydantic data validators
│   │
│   └── sasaknlp/                       # Backward-compatibility layer
│
├── resources/
│   ├── sasak/                          # Balai Bahasa NTB lexicon & stopwords
│   ├── samawa/                         # Sumbawa lexicon & stopwords
│   └── mbojo/                          # Bima/Dompu lexicon & stopwords
│
├── datasets/
├── training/                           # Language detector training scripts
├── tests/                              # 134 automated unit tests
├── examples/                           # Usage examples
└── pyproject.toml

---

💻 Command Line Interface (CLI)

# Language Detection
nusanlp detect "Tiang lalo leq peken"

# Tokenize
nusanlp tokenize --lang sasak "Tiang lalo leq peken"

# Normalize
nusanlp normalize --lang mbojo "NAHU MAI DI KAMPUNG"

# Dictionary Lookup
nusanlp dictionary lookup --lang samawa "bale"

# End-to-end Process
nusanlp process --lang auto "Tiang lalo mangan leq bale" --json

---

🌐 REST API

Run the optional FastAPI server:

uvicorn nusanlp.api.app:app --host 127.0.0.1 --port 8000 --reload

Endpoints available:

• POST /detect-language
• POST /tokenize
• POST /normalize
• POST /process
• GET /dictionary/{language}/{word}
• GET /docs (Interactive Swagger UI)

---

🧪 Testing & Evaluation

Run test suite with pytest:

pytest tests -v

Run unified scientific evaluation benchmark across all 3 languages:

python3 scripts/evaluate_nusanlp_all.py

📊 Scientific Evaluation Highlights

NusaNLP has undergone rigorous empirical validation across 30,000 morphological pairs and 3,000 language detection sentences:

Evaluation Dimension	Dataset	Metric	Score
Sasak Stemmer	Benchmark 10k	Exact Match Accuracy	93.23% (11,279 words/s)
Samawa Stemmer	Benchmark 10k	Exact Match Accuracy	94.65% (43,365 words/s)
Mbojo Stemmer	Benchmark 10k	Exact Match Accuracy	93.22% (42,836 words/s)
Language Identification	Test Set 3,000 sentences	Global Accuracy	99.90% (Macro-F1: 99.90%)
Pipeline Latency	Authentic Sentences	Mean Latency	0.038 - 0.139 ms / sentence
Downstream TF-IDF ML	Multi-dialect Corpus	Vocabulary Compression	9.9% feature reduction (100% Acc)

See the complete evaluation report in docs/laporan_evaluasi_nusanlp.md.

---

📚 Datasets & Resources

• Total Dataset Volume: 959,279 records (Sentence corpora, morphology benchmarks, language detection, and official dictionaries).
• Official Dictionaries (Balai Bahasa NTB): 3,352 Sasak, 659 Samawa, 2,777 Mbojo entries (100% Pydantic compliant).
• Curated Stopwords: 440 functional words (159 Sasak, 147 Samawa, 134 Mbojo).
• See detailed documentation in datasets/README.md and docs/PANDUAN_LENGKAP.md.

---

📖 Citation

If you use NusaNLP in your academic work or research, please cite:

@software{nusanlp2026,
  author = {NusaNLP Contributors},
  title = {NusaNLP: An Open-Source Natural Language Processing Toolkit for Low-Resource Languages of West Nusa Tenggara, Indonesia},
  year = {2026},
  url = {https://github.com/kodetr/nusanlp}
}

📄 License

Distributed under the MIT License. See LICENSE for details.