emakhuwa-tokenizer 0.1.1

Rule-based morphological tokenizer for Emakhuwa (Bantu P.31)

emakhuwa-tokenizer

emakhuwa-tokenizer/ ├── AGENTS.md, BOOTSTRAP.md, HEARTBEAT.md, IDENTITY.md # workspace/agent docs ├── SOUL.md, TASK.md, TOOLS.md, USER.md, README.md ├── LICENSE ├── config.yaml ├── pyproject.toml, pytest.ini # project/build config │ ├── analysis/ # linguistic data │ ├── morpheme_inventory.json │ ├── noun_class_table.json │ └── verb_paradigms.json │ ├── evaluation/ # eval results & gold sets │ ├── error_analysis.md │ ├── gold_errors.json / gold_standard.json │ ├── gold_standard_results.json / results.json │ └── pytest_output.txt │ ├── reference/ │ └── MORPHOLOGY_REFERENCE.md │ ├── rules/ # morphology rule definitions │ ├── extensions.json, imbrication.json │ ├── morphophonology.json, negation.json │ ├── noun_classes.json, spelling_normalization.json │ └── verb_template.json │ ├── scripts/ │ ├── evaluate.py, evaluate_gold.py │ └── phase1_analyze.py │ ├── src/emakhuwa_tokenizer/ # main package │ ├── init.py, cli.py, tokenizer.py │ ├── decision.py, glosser.py, imbrication.py │ ├── lemmatizer.py, lexicon.py, lookup.py │ ├── models.py, morphophonology.py, normalizer.py │ ├── rules.py, segmenter.py, sentence.py │ └── rule_data/ # packaged copies of rules/*.json │ ├── init.py │ └── *.json (extensions, imbrication, morphophonology, ...) │ └── tests/ └── test_tokenizer.py

Rule-based morphological tokenizer for Emakhuwa / Makhuwa (Bantu P.31), focused on morphologically informed segmentation, root identification, and word-class prediction.

The package implements a transparent linguistic pipeline for Emakhuwa morphology: spelling normalization, morphophonological reversal, template-based verb parsing, noun-class analysis, extension handling, imbrication, and lexicon-assisted segmentation.

Linguistic background

Emakhuwa is a Bantu language of Mozambique with rich agglutinative morphology. A useful morphological tokenizer must handle phenomena such as:

• noun-class prefixes: mu-, a-, mi-, ma-, e-, i-, locatives, and agentive patterns
• verbal template slots: NEG-SM-TAM-OP-ROOT-EXT-FV
• subject/object marker allomorphy
• TAM markers such as naa-, ho-, nni-
• derivational extensions: applicative, causative, reciprocal, passive, stative, intensive
• vowel-height harmony in extensions
• imbrication in recent-past forms
• spelling and apostrophe normalization

Installation

pip install emakhuwa-tokenizer

For development:

git clone <repo-url>
cd emakhuwa-tokenizer
pip install -e .[dev]
pytest

Quick start

from emakhuwa_tokenizer import tokenize, tokenize_sentence, lemmatize, load_tokenizer

analysis = tokenize("kinaavaha")
print(analysis.root)
print([(m.form, m.slot, m.gloss) for m in analysis.morphemes])

# Reuse a tokenizer instance for many words
tok = load_tokenizer()
for word in ["kinaavaha", "nkivahale", "kivanje"]:
    print(tok.analyze_word(word).to_dict())

print(lemmatize("kinaavaha"))  # ovaha

results = tokenize_sentence("Mwana anaavaha mwalimu ehuku")
for r in results:
    print(f"{r.surface_form}: {r.root} ({r.word_class})")

Sentence example:

>>> from emakhuwa_tokenizer import tokenize_sentence
>>> results = tokenize_sentence("Mwana anaavaha mwalimu ehuku")
>>> for r in results:
...     print(f"{r.surface_form}: {r.root} ({r.word_class})")
Mwana: ana (noun)
anaavaha: vah (verb)
mwalimu: alimu (noun)
ehuku: huku (noun)

Example output shape:

{
    "surface_form": "kinaavaha",
    "normalized_form": "kinaavaha",
    "root": "vah",
    "word_class": "verb",
    "tam": "PRES/FUT",
    "morphemes": [
        {"form": "ki", "slot": "SM", "gloss": "1SG"},
        {"form": "naa", "slot": "TAM", "gloss": "PRES/FUT"},
        {"form": "vah", "slot": "ROOT", "gloss": "vah"},
        {"form": "a", "slot": "FV", "gloss": "FV.IND/IMP/INF"}
    ]
}

CLI usage

The package installs two equivalent commands:

emakhuwa-tokenize kinaavaha
makhuwa-tokenize kinaavaha

Text output:

Word: kinaavaha
Class: verb
Root: vah
Lemma: ovaha
Morphemes:
  ki-     [SM    ] 1SG
  naa-    [TAM   ] PRES/FUT
  vah-    [ROOT  ] give
  -a      [FV    ] indicative
Confidence: 0.95
Path: rule

Analyze a sentence:

emakhuwa-tokenize "Mwana anaavaha mwalimu ehuku"

Only lemmas:

emakhuwa-tokenize --lemma kinaavaha

JSON output:

emakhuwa-tokenize --json kinaavaha nkivahale

For one word, JSON output is a single object. For multiple words or sentence input, it is an array of objects.

{
  "word": "kinaavaha",
  "word_class": "verb",
  "root": "vah",
  "lemma": "ovaha",
  "morphemes": [
    {"form": "ki", "slot": "SM", "gloss": "1SG"},
    {"form": "naa", "slot": "TAM", "gloss": "PRES/FUT"},
    {"form": "vah", "slot": "ROOT", "gloss": "give"},
    {"form": "a", "slot": "FV", "gloss": "indicative"}
  ],
  "confidence": 0.95,
  "path": "rule"
}

Verbose analysis:

emakhuwa-tokenize --verbose kivanje

Process a file, one word per line:

emakhuwa-tokenize --file words.txt --json

Use a custom dictionary/training data directory:

emakhuwa-tokenize --data-dir data/ kinaavaha

API reference

tokenize(word: str) -> MorphToken

Tokenize a single Emakhuwa word using the packaged rule system.

from emakhuwa_tokenizer import tokenize
analysis = tokenize("kivanje")

load_tokenizer(data_dir: str | None = None) -> Tokenizer

Create a reusable tokenizer. If data_dir is supplied, the tokenizer can build a training lexicon from that directory. If omitted, it uses packaged JSON rules only.

from emakhuwa_tokenizer import load_tokenizer

tok = load_tokenizer()
analysis = tok.analyze_word("nkivahale")

lemmatize(word: str) -> str

Return a citation-form lemma. Verbs are returned as infinitives (o- + root + -a); nouns return the identified bare stem/root.

from emakhuwa_tokenizer import lemmatize
print(lemmatize("kinaavaha"))  # ovaha

tokenize_sentence(text: str) -> list[MorphToken]

Split a sentence on whitespace/punctuation and analyze each word in order.

from emakhuwa_tokenizer import tokenize_sentence
results = tokenize_sentence("Mwana anaavaha mwalimu ehuku")

MorphToken

Dataclass representing one analyzed word.

Important fields:

• surface_form
• normalized_form
• morphemes
• root
• word_class
• noun_class
• tam
• confidence
• gloss
• diagnostics

Use to_dict() for JSON serialization.

Morpheme

Dataclass representing one morpheme.

Important fields:

• form
• slot
• gloss
• position
• underlying
• confidence

Architecture overview

raw word
  ↓
spelling normalization
  ↓
exact dictionary lookup, optional
  ↓
rule-based segmentation
  ├─ noun-class and relational-prefix analysis
  ├─ verb-template parser: NEG-SM-TAM-OP-ROOT-EXT-FV
  ├─ extension and vowel-harmony handling
  ├─ imbrication reversal
  └─ morphophonological candidate handling
  ↓
decision routing and confidence scoring
  ↓
MorphToken output

Rules are externalized as JSON under emakhuwa_tokenizer/rule_data/:

• morphophonology.json
• verb_template.json
• noun_classes.json
• extensions.json
• imbrication.json
• negation.json
• spelling_normalization.json

Evaluation

The current external gold-standard benchmark contains 272 manually annotated items.

Metrics on evaluation/gold_standard.json (v0.1.1, seed=42 training-only lexicon, lookup disabled):

Metric	Score
Boundary precision	0.8164
Boundary recall	0.7508
Boundary F1	0.7822
Lemma/root accuracy	0.7721
Word-class accuracy	0.7390

Gold-standard metrics are near-flat versus v0.1.0, but the v0.1.1 min-root guard has a large effect on real running text: on the corpus_/vmw lemma-frequency sample, tokens ending up with an over-stripped 1–2 character root fell from 14.2% to 1.8% (≈87% reduction), and ~34.8k of 89.1k head-words received a more meaningful root.

Changelog

0.1.1

• Min-root plausibility guard. Reject noun/adjective analyses whose root was over-stripped to a 1–2 character CV residue (the dominant systematic error: high-frequency demonstratives, possessives and connectives such as ani, aya, ale, awo, mwaha were split into a-ni, a-ya, a-le, a-wo, mwa-ha). Such words now fall back to a whole-word analysis so the root stays lexically meaningful. Verb roots (which may legitimately be two characters, e.g. rw, ly) go through the verb path and are unaffected.
• Removed the greedy ett relational prefix, which mis-split class-9 e- + tth-initial stems (etthove → ett-hov); it is now correctly analysed as e-tthove.
• Added emakhuwa_tokenizer.__version__ and regression tests for the guard.

0.1.0

• Initial release.

Test suite:

pytest tests/ -v
9 passed

Development

Run tests:

pytest

Run external gold evaluation:

python scripts/evaluate_gold.py

Build package:

python -m build

 cd C:\Users\t-fali\Pictures\mystuff\Consutorias\MAPA\emakhuwa-tokenizer
 pip install -e .
 emakhuwa-tokenize --json "kinohimeryakani"
 emakhuwa-tokenize "Mwana anaavaha mwalimu ehuku"
 emakhuwa-tokenize --lemma "kinaavaha"
 pytest tests/ -v

Contributing

Contributions are welcome, especially:

• additional manually verified gold data
• dialect-specific spelling normalization rules
• improved finite-state morphology rules
• better handling of lexicalized nouns/adjectives
• documentation of Emakhuwa morpheme inventories

Please keep linguistic rules transparent and prefer JSON-rule updates over opaque model behavior when possible.

License

MIT License. See LICENSE.