Mon Tokenizer

Unigram tokenizer for Mon (mnw), Burmese and English, with full byte fallback

Mon mixes with Burmese constantly and English routinely, so all three are trained on and measured separately. Anything else on the page — Thai, emoji, IPA, CJK — round-trips through byte fallback rather than being lost.

pip install mon-tokenizer

from mon_tokenizer import MonTokenizer

tokenizer = MonTokenizer()
result = tokenizer.encode("ဂွံအခေါင်အရာမွဲသ္ဂောံဒုင်စသိုင်ကၠာကၠာရ။")

result["pieces"]  # ['▁ဂွံ', 'အခေါင်အရာ', 'မွဲ', 'သ္ဂောံ', 'ဒုင်စသိုင်', 'ကၠာ', 'ကၠာရ။']
result["ids"]  # token ids
result["text"]  # the normalized string that was encoded

assert tokenizer.decode_ids(result["ids"]) == result["text"]

Also on the Hub, producing identical ids:

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("janakhpon/mon_tokenizer")

Measured

Vocabulary 64,256 · trained on the train split of 893,936 lines / 85.8M characters · scored on the whole validation split.

100% of characters in the Mon validation split are single tokens.

Compression describes a corpus, not a tokenizer — quote it with the corpus. Violations count token boundaries inside a Myanmar syllable; the denominator is given because English legitimately has none. Round-trip compares after normalization, which is a deliberate transform.

model_card() returns these machine-readably, and a test fails if they drift from what the artifact does.

Round-trip, and its two known limits

for text in ["🙏 emoji", "ภาษาไทย", "漢字", "Ωπ√∫", "ကျော် page 42 — “quoted” ၏"]:
    r = tokenizer.encode(text)
    assert tokenizer.decode_ids(r["ids"]) == r["text"]

Without byte fallback, characters outside the vocabulary aren't flagged — they're deleted, leaving fluent-looking output with content missing. That matters if you feed OCR output back into a corpus.

Two inputs are known not to round-trip. The loss happens at encode, so no decoder can recover it:

tokenizer.encode(" abc")["ids"] == tokenizer.encode("abc")["ids"]  # True
tokenizer.encode("a b")["ids"] == tokenizer.encode("a▁b")["ids"]  # True

A single leading space is dropped, and a literal U+2581 is read as a space. The table above still holds: the corpus it was measured on is stripped per line and contains no U+2581, so it excludes both cases by construction rather than disproving them. Both are recorded as xfail tests.

Fixing this needs prepend_scheme="never", which changes every token id and so requires a retrain and a new Hub artifact. Deferred on purpose, because id stability is worth more than these two cases. Interior and trailing whitespace, tabs, newlines, NUL and control characters all round-trip.

Normalization

Applied automatically, and stored inside the artifact so it cannot drift from the model: invisible characters stripped, Unicode space separators folded to U+0020, then NFC. Runs of spaces are preserved.

API

The artifact is cached by path, so constructing many instances is cheap.

CLI

pip install 'mon-tokenizer[cli]'

mon-tokenizer "ဂွံအခေါင်အရာမွဲ"          # tokenize
mon-tokenizer -v "ဂွံအခေါင်အရာမွဲ"       # per-token table
mon-tokenizer -d --ids "316,12644,294"   # decode ids

Exit codes: 0 ok · 1 usage · 2 artifact failed to load · 130 interrupted.

Upgrading from 0.2.x

Every token id has changed. Rebuild any embedding matrix built against 0.2.x, or pin mon-tokenizer<1.0 — it stays on PyPI and works.

Requirements

Python 3.11+. Two direct dependencies: tokenizers and regex. The install is 17 packages / ~30MB, because tokenizers requires huggingface-hub.

Development

uv sync --all-extras
uv run ruff format --check . && uv run ruff check . && uv run mypy && uv run pytest

Retraining, releasing and the house rules for tests: docs/how_to_contribute.md.

Design

docs/architecture.md — every decision with the measurement behind it and the condition that would reverse it: algorithm, vocabulary size, why grapheme clusters are the wrong unit for Myanmar, and what was rejected.

Links

• Hugging Face
• MonCorpusCollection — the corpus
• Awesome Mon NLP — the ecosystem

MIT.