Mon Language Detector

Detects Mon (mnw), Burmese (mya) and English (eng), including mixed-script text. Built for filtering scraped corpora.

fastText over character n-grams, with Unicode rules on top: Mon-exclusive characters override the model, other Myanmar-script languages are refused, and every result carries a reliability flag.

Use

from mon_language_detector import LanguageDetector

detector = LanguageDetector()
result = detector.predict("ပ္ဍဲသၞာံ ၁၉၉၀")

if result.reliable and result.label.startswith("mnw"):
    keep(result)

result.basis says where confidence came from — posterior, mon-exclusive, other-myanmar-script, ambiguous-myanmar, too-short, no-script or empty. Threshold on reliable, not on confidence.

Accuracy

Retrained 2026-08-11. make evaluate reproduces these, once a split exists — data/ is not in the repository, so build one first (below).

Corpus filtering keeps the reliable rows and drops the rest, so the first row is the one that describes the workload.

Every remaining error is Mon against Burmese, and it is a length problem: 97.7% of them are in lines of 40 characters or fewer. Lines carrying a Mon-exclusive character had none. MIN_UNAMBIGUOUS_MYANMAR_LEN is set at 30 on that evidence, tuned on the split it is scored on — so 0.9980 is optimistic by an unknown margin.

What it cannot do

Three classes, and the Myanmar script is shared by more. Shan, Khamti, Aiton, Karen and Palaung have nowhere to land, and a Mon scrape collects them.

Text carrying a character exclusive to one of those languages now returns unknown. That covers the ones a character can prove; Shan written without them is still labelled Mon. A fourth class needs Shan data nobody has.

Building a model

uv run datasets --explain                      # what the selection keeps and drops
uv run datasets --corpus-root /path/to/corpus  # link a corpus into datasets/
uv run pipeline --target-mon 150000 --target-mya 150000 --target-eng 150000
uv run train --epoch 20 --dim 128
uv run evaluate

Point --corpus-root at a corpus laid out as a directory per source, each holding .txt files. A directory records where a line came from, not what language it is, so the selection step is where that gets fixed; --explain shows the reasoning.

Targets are 150,000 because Burmese has 46,407 unique lines. More buys repetition.

uv run wrangle cleans raw files first. uv run preview spot-checks a model. make check runs ruff, mypy and the tests.

Install

The model ships inside the wheel, so detection works straight after install with nothing further to download:

pip install mon-language-detector

The wrangle CLI needs the [wrangle] extra on top; detection does not:

pip install "mon-language-detector[wrangle]"

The model's training corpus carries terms the repository's MIT grant does not cover — see LICENSE-MODEL.md before redistributing it.

Deployment

The .ftz works with the standard fastText bindings on Android, iOS and WASM.

Licence

MIT for the code. The shipped model is a derived work of a corpus with mixed and partly unresolved terms, and MIT does not reach it — LICENSE-MODEL.md states which sources are established and which are not.

Documents

• CHANGELOG.md — what changed in each version, with the numbers
• LICENSE-MODEL.md — what the model was trained on, and under what terms