Mon Tokenizer

Mon mixes with Burmese constantly and English routinely, so all three are trained on and measured separately. Anything else on the page (Thai, emoji, IPA, CJK) round-trips through byte fallback rather than being lost.

pip install mon-tokenizer

from mon_tokenizer import MonTokenizer

tokenizer = MonTokenizer()
result = tokenizer.encode("ဂွံအခေါင်အရာမွဲသ္ဂောံဒုင်စသိုင်ကၠာကၠာရ။")

result["pieces"]  # ['▁ဂွံ', 'အခေါင်အရာ', 'မွဲ', 'သ္ဂောံ', 'ဒုင်စသိုင်', 'ကၠာ', 'ကၠာရ။']
result["ids"]  # token ids
result["text"]  # the normalized string that was encoded

assert tokenizer.decode_ids(result["ids"]) == result["text"]

Also on the Hub, producing identical ids:

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("janakhpon/mon_tokenizer")

Measured

Vocabulary 64,256 · trained on the train split of 893,936 lines / 85.8M characters · scored on the whole validation split.

98.74% of the distinct characters in the Mon validation split are single tokens — 392 of 397, over all 29,600 lines. The remaining five (a combining diaeresis below, a Greek capital pi, two Sinhala letters and one emoji; seven occurrences in 2.28M characters) round-trip through byte fallback at one to four tokens each.

Compression describes a corpus, not a tokenizer — quote it with the corpus. Violations count token boundaries inside a Myanmar syllable; the denominator is given because English legitimately has none. Round-trip compares after normalization, which is a deliberate transform.

model_card() returns these machine-readably, and a test fails if they drift from what the artifact does.

Round-trip, and its two known limits

for text in ["🙏 emoji", "ภาษาไทย", "漢字", "Ωπ√∫", "ကျော် page 42 — “quoted” ၏"]:
    r = tokenizer.encode(text)
    assert tokenizer.decode_ids(r["ids"]) == r["text"]

Without byte fallback, characters outside the vocabulary aren't flagged — they're deleted, leaving fluent-looking output with content missing. That matters if you feed OCR output back into a corpus.

Two inputs are known not to round-trip. The loss happens at encode, so no decoder can recover it:

tokenizer.encode(" abc")["ids"] == tokenizer.encode("abc")["ids"]  # True
tokenizer.encode("a b")["ids"] == tokenizer.encode("a▁b")["ids"]  # True

A single leading space is dropped, and a literal U+2581 is read as a space. The table above still holds: the corpus it was measured on is stripped per line and contains no U+2581, so it excludes both cases by construction rather than disproving them. Both are recorded as xfail tests.

Fixing this needs prepend_scheme="never", which changes every token id and so requires a retrain and a new Hub artifact. Deferred on purpose, because id stability is worth more than these two cases. Interior and trailing whitespace, tabs, newlines, NUL and control characters all round-trip.

Normalization

Applied automatically, and stored inside the artifact so it cannot drift from the model: invisible characters stripped, Unicode space separators folded to U+0020, then NFC. Runs of spaces are preserved.

API

The artifact is cached by path, so constructing many instances is cheap.

CLI

pip install 'mon-tokenizer[cli]'

mon-tokenizer "ဂွံအခေါင်အရာမွဲ"          # tokenize
mon-tokenizer -v "ဂွံအခေါင်အရာမွဲ"       # per-token table
mon-tokenizer -d --ids "316,12644,294"   # decode ids

Exit codes: 0 ok · 1 usage · 2 artifact failed to load · 130 interrupted.

Upgrading from 0.2.x

Every token id has changed. Rebuild any embedding matrix built against 0.2.x, or pin mon-tokenizer<1.0 — it stays on PyPI and works.

Requirements

Python 3.11+. Two direct dependencies: tokenizers and regex. The install is 17 packages / ~30MB, because tokenizers requires huggingface-hub.

Development

uv sync --all-extras
uv run ruff format --check . && uv run ruff check . && uv run mypy && uv run pytest

Retraining, releasing and the house rules for tests: docs/how_to_contribute.md.

Design

docs/architecture.md — every decision with the measurement behind it and the condition that would reverse it: algorithm, vocabulary size, why grapheme clusters are the wrong unit for Myanmar, and what was rejected.

Links

• Hugging Face
• MonCorpusCollection — the corpus
• Awesome Mon NLP — the ecosystem

MIT.