mongol-norm

English | 中文

English

Status

Both halves of the library — shaping and normalization (MNG / Hudum) — are verified against the same upstream corpora. CI runs the full suite on Python 3.9 – 3.13 on every push.

✅ Shaping

shape() and same_shape() are cross-validated against two upstream TSV suites:

✅ Normalization — corpus-scoped guarantees, machine-checked

For written-unit shapes covered by the bundled normalization table, normalize() / normalize_text() / normalize_written_units() are pure functions of shape with invariants checked in CI over every corpus encoding:

Scope note: normalization is implemented for MNG (Hudum) only — Todo / Sibe / Manchu load shaping rules but have no normalizer yet. The guarantees above cover the checked corpus and any input whose written-unit chains can be encoded by the bundled table. For an uncovered out-of-corpus chain, normalization raises NormalizationFallbackError by default. Pass strict=False explicitly only when returning the original word unchanged is acceptable.

This project was generated with Claude Code (AI-assisted coding). The tests and key parts of the core code have been manually reviewed, and test coverage is extensive (corpus round-trip / shape-canonicity / prefix-stability plus the upstream cross-implementation suites). Treat this as a preview release — it should be fine for normal use; if you hit a problem, please open an issue or PR. Shaping logic is derived from UTN #57 v4 and mongfontbuilder.

Why This Project Exists

Traditional Mongolian script in Unicode has a fundamental problem: the same visible word can be encoded in multiple different Unicode sequences. This happens because:

1. Letters share glyphs — A and E look identical in medial and final positions; O/U and OE/UE share forms; QA and GA share forms depending on vowel harmony.
2. Multiple encoding paths — The same tooth glyph (I) can be encoded as I, YA+FVS1, or even two separate I characters.
3. Redundant FVS usage — Free Variation Selectors (FVS1–FVS4) can create equivalent sequences that render identically.
4. Joining controls — nirugu (U+180A, the visible stem extender) and ZWJ (U+200D) force letters into their joined forms, and inside those joined contexts even more letters collapse to the same glyph (nirugu+o and nirugu+u render identically).
5. Suffix particles after MVS/NNBSP — the same rendered suffix can be spelled with different letters (MVS+a and MVS+e both render the chachlag form; MVS+u and MVS+ue render the same connector).

This means:

• Search fails: Searching for "sain" (one encoding) won't find the same word in another encoding, even though they look identical.
• Deduplication breaks: The same word has multiple Unicode representations.
• Indexing is unreliable: Different encodings of the same word produce different keys.

What This Project Does

This is a shaping-aware normalizer for Traditional Mongolian. It:

1. Shapes the input using the full UTN #57 v4 shaping process (5-step conditional mapping)
2. Compares glyph sequences to detect identical visual forms
3. Normalizes supported shapes to one canonical, FVS-pinned Unicode encoding — same shape ⟹ same Unicode, with an exact shape round-trip

Example: All five of these encode the word "sain" (good) and look identical:

How It Works

The normalizer implements a lightweight Mongolian shaping engine — equivalent to what HarfBuzz does with a font file, but using only the rule data from UTN #57 v4 and the mongfontbuilder project. No font files needed.

Shaping Pipeline (UTN #57 v4 Mongolian-Specific Phase)

1. Chachlag — Suffix forms for A/E after MVS (Mongolian Vowel Separator)
2. Syllabic — Consonant/vowel context: onset, devsger, marked, masculine/feminine harmony, dotless
3. Particle — MVS particle dictionary lookup for specific suffix words
4. Devsger — I after a vowel (vowel_devsger) gets double-tooth form: I → I+I
5. Post-bowed — Vowel forms change after bowed consonants (G, B, K, P, F)

Normalization Strategy

Within the normalization table's supported written-unit domain, normalize is a pure function of shape: any two encodings that shape identically produce the same Unicode output, and the output round-trips — shape(normalize(x)) == shape(x). It is also prefix-stable. When these goals conflict the priority is round-trip > prefix-stable > shortest.

Per word:

1. shape the input into its written-unit sequence. Structural characters — MVS, nirugu, ZWJ — appear verbatim as PascalCase Mvs / Nirugu / Zwj tokens (nirugu renders a visible stem; all three are the evidence for a neighbour's init/medi/fina form). Split the shape at these tokens into chains; the tokens themselves are copied through unchanged.
2. encode each chain (right-to-left, so appending a suffix can't disturb what precedes it):
   1. partition + table lookup — the primary path. At each position take the single unit if the table has it (preferred — clean output), else the longest available multi-unit entry, and look up (position, written-unit) → (letter, FVS) in an FVS-pinned table. Each value renders its unit regardless of neighbours, so the result is a deterministic, O(N), prefix-stable function of the shape.
   2. velar-feminine refinement — a G/Gx velar's forward-coupled vowel (a/o/u) is swapped to its feminine partner (e/oe/ue) for clean output.
   3. verify — reshape the candidate in full context; accept only if it equals the target chain shape.
   4. no search fallback — the table is total over the corpus (with FVS-first selection there are no gap chains left). If an out-of-corpus shape ever misses the table, normalization raises NormalizationFallbackError with the input and uncovered written-unit sequence. Callers may explicitly pass strict=False to return the input unchanged (round-trip preserved, never a mis-encoding). A letter next to a joiner simply looks its unit up at the shifted (joined) position.
3. post-MVS suffix rule — a chain directly after MVS takes its standalone canonical (drop the MVS, normalize, re-attach), so the spelling never depends on MVS. One exception: chachlag Aa after MVS is written the bare letter a. (The isolate-I → i+FVS1 spelling is pinned in the table itself — no post-processing pass exists.)

Prefix-stability means: if word A = word B + a suffix and their shapes share a prefix, the shared region encodes identically except the single boundary unit whose position changes (final in B → medial in A). The per-unit table delivers this for free — each unit's encoding depends only on its own position, never on its neighbours.

How the table is built (the selection method): offline, a context-independence battery fills each (position, written-unit) slot with the (letter, FVS) that renders exactly that unit in every probed neighbour context (the probes include a bowed consonant, so post-bowed effects can't hide). Candidate order is letter-major, FVS-first within the letter — an FVS exists precisely to pin a form against context, so the pinned variant of the right letter always beats its context-sensitive bare form. The result is exported and shipped as JSON; the battery lives in scripts/gen_normalize_table.py.

Note: supported output is FVS-pinned, not bare — each unit carries the selector that fixes its form independent of context. This is what makes "same shape ⟹ same Unicode" and prefix-stability hold inside the table's domain. The per-unit table is exported as language-agnostic JSON (mongol_norm/data/MNG.normalize.json); schema + consuming algorithm are in docs/data-format.md, so ports in other languages can implement normalize with just a JSON parser.

The exact canonical selection policy is frozen as mng-canonical/1. It is available as shaper.canonical_version and embedded in MNG.normalize.json. Applications that persist normalized search/index keys should store this version alongside them and rebuild those keys if a future release changes it.

Installation

mongol-norm is a single self-contained package on PyPI — the shaping/normalize data is bundled, no runtime dependencies:

pip install mongol-norm

Or from source:

git clone https://github.com/Satsrag/mongol-norm.git
cd mongol-norm
pip install .

Usage

from mongol_norm import MongolianShaper, NormalizationFallbackError

shaper = MongolianShaper(locale="MNG")  # Hudum Traditional Mongolian

# Shape: get written-unit sequence
shaper.shape("ᠰᠠᠢᠨ")
# → ['S', 'A', 'I', 'I', 'A']

# Compare: are two encodings visually identical?
shaper.same_shape("ᠰᠠᠢᠨ", "ᠰᠡᠢᠨ")
# → True

shaper.same_shape("ᠰᠠᠢᠨ", "ᠨᠠᠢ᠍ᠮᠠ")
# → False

# Normalize a supported shape to canonical, FVS-pinned Unicode
shaper.normalize("ᠰᠡᠢᠨ")
# → 'ᠰᠠᠢ᠍ᠢ᠍ᠠ᠌'

shaper.normalize("ᠰᠠᠶ᠋ᠢᠨ")
# → 'ᠰᠠᠢ᠍ᠢ᠍ᠠ᠌'

shaper.normalize("ᠰᠠᠶ᠋ᠶ᠋ᠨ")
# → 'ᠰᠠᠢ᠍ᠢ᠍ᠠ᠌'

# Strict is the default. strict=False preserves the input only if fallback occurs.
shaper.normalize("ᠰᠠᠢᠨ", strict=False)

# Normalize an already-shaped written-unit sequence
shaper.normalize_written_units(["B", "Aa"])
# → 'ᠪᠠ᠋'

# Every written unit is PascalCase, including shape() control output
shaper.normalize_written_units(["S", "A", "I", "I", "N", "Mvs", "Aa"])

# Encode authoritative HUD written-unit positions without inferring controls
shaper.normalize_positioned_written_units([
    {"unit": "B", "position": "init"},
    {"unit": "Aa", "position": "fina"},
])
# → 'ᠪᠠ᠋'

# HUD position is not Unicode topology: isolated FA borrows F:init
shaper.normalize_positioned_written_units([
    {"unit": "F", "position": "init"},
])
# → 'ᠹ' (bare U+1839, no ZWJ); F:isol is unsupported

normalize_written_units() accepts an ordered Sequence[str] of shape units, not nominal Unicode. Letter positions are inferred from unit order and structural controls; explicit position records are not accepted by this API. All written-unit names use PascalCase; structural controls are Mvs, Nirugu, and Zwj, exactly as returned by shape(). Old lowercase or all-uppercase control aliases are not accepted. The API never infers or inserts a structural control: ZWJ is present in the output only when Zwj is present in the requested sequence. An empty sequence returns an empty string. A malformed outer input or a non-string item raises TypeError; unknown units and sequences that cannot reshape exactly raise ValueError rather than being guessed or partially encoded.

normalize_positioned_written_units() accepts an ordered sequence of exact built-in {"unit": str, "position": str} dict records. Here position is the position of the written unit in the authoritative HUD inventory, not a Unicode letter's joining topology. It reuses normalize_written_units() rather than a second encoding table. A complete multi-record chain starts with init and ends with fina. An incomplete left or right edge gets an implicit Zwj; for example B:medi, O:medi, G:fina is normalized as Zwj, B, O, G. A single init record is the exception and is normalized without ZWJ, so F:init becomes bare U+1839. A single medi gets ZWJ on both sides and a single fina gets ZWJ on the left. F:isol is absent from the source inventory and fails closed. Mvs and Nirugu use control; explicit Zwj input is rejected. A wrong outer/record/field type raises TypeError; wrong keys, unit, position, chain positions, exact encoding, or more than 1024 records raises ValueError. This API has no CLI subcommand yet.

Full-text normalization

normalize() operates on single words. For sentences, paragraphs, or mixed-script text, use normalize_text() — it normalizes each Mongolian word independently while preserving spaces, punctuation, and non-Mongolian text verbatim.

# Normalize a sentence (each Mongolian word normalized independently)
shaper.normalize_text("ᠰᠡᠢᠨ ᠨᠠᠢ᠍ᠮᠠ")
# → 'ᠰᠠᠢ᠍ᠢ᠍ᠠ᠌ ᠨᠠᠢ᠍ᠮᠠ᠌'

# Mixed script: non-Mongolian text preserved as-is
shaper.normalize_text("Hello ᠰᠡᠢᠨ world")
# → 'Hello ᠰᠠᠢ᠍ᠢ᠍ᠠ᠌ world'

Batch normalization example

words = ["ᠰᠡᠢᠨ", "ᠰᠠᠢᠨ", "ᠰᠨ᠌ᠢᠢᠨ", "ᠰᠠᠶ᠋ᠢᠨ"]
normalized = [shaper.normalize(w) for w in words]
unique = set(normalized)
print(f"{len(words)} inputs → {len(unique)} unique form(s): {unique}")
# 4 inputs → 1 unique form(s): {'ᠰᠠᠢ᠍ᠢ᠍ᠠ᠌'}

Command line

After pip install mongol-norm, the mongol-norm command is on PATH (or run python -m mongol_norm.shaper ... without installing).

# Inline text
mongol-norm shape 'ᠰᠠᠢᠨ'                   # → S+A+I+I+A
mongol-norm normalize 'ᠰᠡᠢᠨ'               # canonical form
mongol-norm normalize --allow-fallback 'ᠰᠡᠢᠨ'  # preserve input if fallback occurs
mongol-norm normalize-written-units 'B+Aa'  # → ᠪᠠ᠋
mongol-norm normalize-written-units 'BZwj'  # compact PascalCase units
mongol-norm normalize-text 'Hello ᠰᠡᠢᠨ'    # mixed script
mongol-norm normalize-text --allow-fallback -i in.txt

# Pipe / stdin (use `-` as the text)
echo 'ᠰᠡᠢᠨ' | mongol-norm normalize -
echo 'B+Aa' | mongol-norm normalize-written-units -
cat doc.txt | mongol-norm normalize-text -

# File in / out
mongol-norm normalize-text -i in.txt -o out.txt

# Batch: one word per line in, one canonical per line out
mongol-norm normalize --batch -i words.txt -o canonical.txt
# Written-unit batch: one compact or '+'-joined sequence per line
mongol-norm normalize-written-units --batch -i units.txt -o canonical.txt

# Visual-identity check (exit 0 if same, 1 if different)
mongol-norm same 'ᠰᠠᠢᠨ' 'ᠰᠡᠢᠨ'

normalize-written-units accepts compact PascalCase or explicit + boundaries. Compact input must have one unique segmentation; ambiguous input fails closed and must be rewritten with +. After parsing, the same exact-shape validation as normalize_written_units() applies, so a syntactically valid unit stream can still be rejected when it has no canonical MNG encoding.

normalize (single-word) skips non-Mongolian characters, so feeding it a multi-line file treats the whole thing as one word. Use --batch for one-word-per-line files, or normalize-text for free-form text.

Running Tests

cd mongol-norm

# Shaping + same_shape + normalize unit tests
python -m unittest tests.test_shaper -v

# Normalize properties: round-trip + shape-canonicity + prefix-stability
python -m unittest tests.test_round_trip

# Normalize-table export (compute == load)
python -m unittest tests.test_normalize_table

# mongfontbuilder core-hud (225) + GB/T 25914-2023 eac-hud (3513, 5 UTN-xfail)
python -m unittest tests.test_core_hud tests.test_eac_hud

# Or all together (auto-discovers every tests/test_*.py)
python -m unittest discover -s tests -p 'test_*.py'

The hand-written suite covers:

Current totals: 214 tests (unit + property + 225 core-hud + 3513 eac-hud corpus runners), all green on Python 3.9 – 3.13.

Use Cases

• Search & Retrieval — Index Mongolian text with unique keys per visual word
• Deduplication — Detect identical words encoded differently
• Spell Checking — Normalize before dictionary lookup
• Corpus Linguistics — Consistent word frequency counts
• OCR Post-processing — Standardize OCR output that may use inconsistent encodings
• Input Method Engines — Validate and normalize user input

Project Structure

mongol-norm/                          # the repo = the package (single, self-contained)
├── .github/workflows/test.yml        # CI: Python 3.9-3.13 on every push
├── pyproject.toml
├── mongol_norm/
│   ├── shaper.py                     # tokenize / assign_positions / shape / normalize
│   ├── rules.py                      # the 5 shaping phases (iii1..iii5) mirroring iii.py
│   ├── _data.py                      # loaders for the bundled JSON
│   └── data/                         # bundled shaping + normalize data
│       ├── MNG.json  TOD.json  SIB.json  MCH.json
│       └── MNG.normalize.json        # per-unit normalize table
├── scripts/                          # dev-only generators (preprocess, gen_normalize_table)
├── docs/data-format.md               # JSON schema, for other-language ports
└── tests/
    ├── test_shaper.py  test_round_trip.py  test_normalize_table.py
    ├── test_written_units_api.py
    ├── test_core_hud.py  test_eac_hud.py
    └── data/{core,eac}-hud.tsv       # vendored from mongfontbuilder

mongol-norm has no runtime dependencies — the shaping/normalize JSON is bundled in mongol_norm/data/. Install with pip install mongol-norm.

Data Sources & Acknowledgments

• UTN #57 v4 — Unicode Technical Note: Encoding and Shaping of the Mongolian Script. The authoritative specification for Mongolian shaping rules.
• mongfontbuilder by Kushim Jiang — Source for the bundled flat variant tables in mongol_norm/data/ (preprocessed from data.variants / data.particles) and for the core-hud.tsv / eac-hud.tsv regression suites we vendor into tests/data/. Both UTN #57 and mongfontbuilder are authored by the same person.
• GB/T 25914—2023 — China national standard for Traditional Mongolian nominal characters; source of the EAC compliance test set.
• Claude Code — This project was developed with AI assistance. The shaping rules are derived from the above sources; Claude Code was used to implement and structure the engine.

Supported Locales

Requirements

• Python 3.7+ (CI-tested on 3.9 / 3.10 / 3.11 / 3.12 / 3.13)
• No runtime dependencies (shaping/normalize data is bundled)

License

MIT License — see LICENSE.

The shaping rules and bundled data are derived from mongfontbuilder (MIT) and UTN #57. Their required notices are retained in NOTICE.

中文

状态

库的两部分 —— 整形与规范化(MNG / Hudum)—— 都对照同一批上游语料验证。CI 在每次 push 上对 Python 3.9 – 3.13 跑完整套件。

✅ Shaping(整形)

shape() 和 same_shape() 对照两套上游 TSV 套件交叉验证:

✅ Normalization(规范化)— 语料域保证,机器验证

对于内置规范化表覆盖的written-unit shape，normalize() / normalize_text() / normalize_written_units() 是 shape 的纯函数,以下不变量在 CI 中对每一条语料编码逐一验证:

范围说明:规范化目前只实现了 MNG(Hudum)—— Todo / 锡伯文 / 满文已加载 shaping 规则,尚无规范化。上述保证覆盖已检查语料及内置表可编码的written-unit chain。语料外chain若未被覆盖，默认抛出NormalizationFallbackError；只有调用者明确接受原样回退时才传入strict=False。

本项目由 Claude Code(AI 辅助编码)生成;测试与部分核心代码经人工审核,测试覆盖比较充分(语料往返 / 同形同码 / 前缀稳定 + 上游跨实现套件)。当前为预览版,正常使用应无问题;遇到问题欢迎提 issue 和 PR。Shaping 逻辑源自 UTN #57 v4 和 mongfontbuilder。

为什么做这个项目

传统蒙古文在 Unicode 中存在一个根本性问题：同一个可见词形可以用多种不同的 Unicode 序列编码。原因是：

1. 字母共享字形 — A 和 E 在中间和尾部位置外形完全相同；O/U、OE/UE 共享形态；QA 和 GA 根据元音和谐共享形态。
2. 多种编码路径 — 同一个齿形字形可以编码为 I、YA+FVS1，甚至两个独立的 I 字符。
3. 冗余的 FVS 使用 — 自由变体选择符（FVS1–FVS4）可以创建渲染结果完全相同的等价序列。
4. 连接控制符 — nirugu(U+180A,可见的连笔延长符)和 ZWJ(U+200D)会强制字母取连接形,而连接语境下更多字母塌缩成同一字形(nirugu+o 和 nirugu+u 渲染完全相同)。
5. MVS/NNBSP 后的后缀词 — 同一个渲染出的后缀可以用不同字母拼写(MVS+a 和 MVS+e 都渲染 chachlag 形;MVS+u 和 MVS+ue 同形)。

这意味着：

• 搜索失效：搜索同一个词的某种编码，找不到另一种编码，尽管它们外形完全一样。
• 去重失败：同一个词有多种 Unicode 表示。
• 索引不可靠：同一个词的不同编码产生不同的索引键。

这个项目做什么

这是一个形态感知的蒙古文规范化器。它：

1. 使用完整的 UTN #57 v4 shaping 过程（5 步条件映射）对输入进行字形化
2. 通过比较字形序列来检测视觉上相同的词形
3. 将受支持的shape规范化为唯一的、FVS 钉死的 canonical Unicode 编码 —— 同 shape ⟹ 同 Unicode,并精确往返还原

示例：以下五种编码都表示 "sain"（好的），外形完全相同：

工作原理

本规范化器实现了一个轻量级蒙古文 shaping 引擎——功能相当于 HarfBuzz 配合字体文件所做的事情，但仅使用 UTN #57 v4 的规则数据和 mongfontbuilder 项目的变体数据。不需要字体文件。

Shaping 管线（UTN #57 v4 蒙古文特定阶段）

规范化策略

在规范化表支持的written-unit域内，normalize 是 shape 的纯函数:任意两个 shape 相同的编码,normalize 输出相同,且往返成立 —— shape(normalize(x)) == shape(x),同时前缀稳定。三者冲突时优先级:往返 > 前缀稳定 > 最短。

逐词:

1. shape 成书写单元序列。结构字符 —— MVS、nirugu、ZWJ —— 原样输出为PascalCase Mvs / Nirugu / Zwj token(nirugu 是可见的连笔字形;三者都是邻居字母 init/medi/fina 形的依据)。按这些 token 切成 chain,token 本身原样拷贝。
2. 逐 chain 编码(从右往左,这样加后缀不影响前面):
   1. 划分 + 查表(主路径):每个位置优先取单单元(输出干净),否则取最长多单元,查 (位置, 书写单元) → (字母, FVS) 的 FVS 钉死表。每个值不依赖邻居就渲染出该单元 → 确定性、O(N)、前缀稳定。
   2. velar 阴性微调:G/Gx 前向耦合的元音(a/o/u)换成阴性(e/oe/ue),输出更干净。
   3. 校验:在完整上下文里重新 shape,只接受与目标 chain shape 一致的结果。
   4. 没有搜索兜底:FVS 优先的选择下,表对全部语料 chain 是完备的(缺口为零)。语料外的 shape 万一查不到表，默认抛出包含原输入与未覆盖written-unit序列的NormalizationFallbackError。只有明确接受原样回退时才传入strict=False(保住往返,绝不错编)。紧邻 joiner 的字母只是按移动后的连接位置查表。
3. MVS 后缀规则:紧跟 MVS 的 chain 用其 standalone canonical(去掉 MVS、归一、再拼回),拼写不依赖 MVS。唯一例外:MVS 后的 chachlag Aa 写裸字母 a。(孤立 I → i+FVS1 的拼写已钉进表本身 —— 不存在后处理。)

前缀稳定的含义:若词 A = 词 B + 后缀,且二者 shape 共享前缀,则共享部分编码完全一致,只有那个位置发生变化的边界单元不同(在 B 里是词尾、在 A 里变词中)。逐单元表天然保证这点 —— 每个单元的编码只取决于它自己的位置,与邻居无关。

表是怎么来的(选择方法):离线跑一个 context 无关性电池 —— 对每个 (位置, 书写单元),挑出在所有探测邻居上下文里都恰好渲染出该单元的 (字母, FVS)(探针含弓形辅音,post-bowed 效应藏不住)。候选顺序是字母优先、字母内 FVS 优先 —— FVS 的意义就是把字形从 context 里隔离出来,所以正确字母的钉死形永远优于其受感染的裸形。结果导出成 JSON 随包发布;电池在 scripts/gen_normalize_table.py。

注意:受支持输出是 FVS 钉死而非 bare —— 每个单元都带着把字形固定住、不受上下文影响的选择符,这正是"同 shape ⟹ 同 Unicode"和前缀稳定在表内成立的原因。逐单元表导出为语言无关的 JSON(mongol_norm/data/MNG.normalize.json),schema 与消费算法见 docs/data-format.md;其他语言只需一个 JSON 解析器即可实现 normalize。

当前精确 canonical 选择策略冻结为 mng-canonical/1。可通过 shaper.canonical_version 读取，并写入 MNG.normalize.json。持久化规范化搜索键/索引键的应用应同时保存该版本；未来版本若发生变化，应重建这些键。

安装

mongol-norm 是单一自包含包,已发布到 PyPI —— shaping/normalize 数据内置,零运行时依赖:

pip install mongol-norm

或从源码安装:

git clone https://github.com/Satsrag/mongol-norm.git
cd mongol-norm
pip install .

使用方法

from mongol_norm import MongolianShaper, NormalizationFallbackError

shaper = MongolianShaper(locale="MNG")  # Hudum 传统蒙文

# 字形化：获取书写单元序列
shaper.shape("ᠰᠠᠢᠨ")
# → ['S', 'A', 'I', 'I', 'A']

# 比较：两个编码视觉上是否相同？
shaper.same_shape("ᠰᠠᠢᠨ", "ᠰᠡᠢᠨ")
# → True

shaper.same_shape("ᠰᠠᠢᠨ", "ᠨᠠᠢ᠍ᠮᠠ")
# → False

# 将受支持shape规范化为唯一的FVS钉死canonical Unicode
shaper.normalize("ᠰᠡᠢᠨ")
# → 'ᠰᠠᠢ᠍ᠢ᠍ᠠ᠌'

shaper.normalize("ᠰᠠᠶ᠋ᠢᠨ")
# → 'ᠰᠠᠢ᠍ᠢ᠍ᠠ᠌'

shaper.normalize("ᠰᠠᠶ᠋ᠶ᠋ᠨ")
# → 'ᠰᠠᠢ᠍ᠢ᠍ᠠ᠌'

# strict是默认行为；strict=False只在触发fallback时原样返回
shaper.normalize("ᠰᠠᠢᠨ", strict=False)

# 从已经shape好的书写单元序列直接生成canonical Unicode
shaper.normalize_written_units(["B", "Aa"])
# → 'ᠪᠠ᠋'

# 所有书写单元均为PascalCase，包括shape()输出的结构control
shaper.normalize_written_units(["S", "A", "I", "I", "N", "Mvs", "Aa"])

# 按权威HUD written-unit position编码，不推断或插入control
shaper.normalize_positioned_written_units([
    {"unit": "B", "position": "init"},
    {"unit": "Aa", "position": "fina"},
])
# → 'ᠪᠠ᠋'

# HUD position不是Unicode topology：FA的isolated variant借用F:init
shaper.normalize_positioned_written_units([
    {"unit": "F", "position": "init"},
])
# → 'ᠹ'（裸U+1839，不含ZWJ）；F:isol不受支持

normalize_written_units()接受由shape unit组成的有序Sequence[str]，而不是 nominal Unicode。字母位置由单元顺序与结构control推导；此API不接受显式 position record。所有written-unit名称统一使用PascalCase；结构control为 Mvs、Nirugu、Zwj，与shape()输出完全一致。旧小写或全大写control别名 不再接受。API绝不自行推断或插入结构control：只有显式包含Zwj时，输出才会 包含ZWJ。空序列返回空字符串。非法外层输入 或非字符串单元抛出TypeError；未知unit或无法精确重新shape的序列抛出 ValueError，不会猜测或返回部分编码结果。

normalize_positioned_written_units()接受由严格内建 {"unit": str, "position": str} dict record组成的有序序列。这里的position 表示权威HUD inventory中的written-unit position，不是Unicode字母在当前序列中的 joining topology。它直接复用normalize_written_units()，不再维护第二套编码表。 完整复合链必须以init开头、以fina结束；左端或右端不完整时自动补Zwj。 例如B:medi, O:medi, G:fina会按Zwj, B, O, G规范化。单个init是特例， 不补ZWJ，因此F:init输出裸U+1839；单个medi前后补ZWJ，单个fina只在 左侧补ZWJ。inventory中不存在的F:isol会fail closed。Mvs与Nirugu使用 control，显式Zwj输入被拒绝。外层/record/字段类型错误抛出TypeError； keys、unit、position、复合链位置、exact encoding错误以及超过1024条record均 抛出ValueError。本API暂不增加CLI命令。

全文规范化

normalize() 作用于单个词。对于句子、段落或混合文字文本，使用 normalize_text() ——它独立规范化每个蒙古文词，同时原样保留空格、标点和非蒙古文文本。

# 规范化句子（每个蒙古文词独立规范化）
shaper.normalize_text("ᠰᠡᠢᠨ ᠨᠠᠢ᠍ᠮᠠ")
# → 'ᠰᠠᠢ᠍ᠢ᠍ᠠ᠌ ᠨᠠᠢ᠍ᠮᠠ᠌'

# 混合文字：非蒙古文文本原样保留
shaper.normalize_text("Hello ᠰᠡᠢᠨ world")
# → 'Hello ᠰᠠᠢ᠍ᠢ᠍ᠠ᠌ world'

批量规范化示例

words = ["ᠰᠡᠢᠨ", "ᠰᠠᠢᠨ", "ᠰᠨ᠌ᠢᠢᠨ", "ᠰᠠᠶ᠋ᠢᠨ"]
normalized = [shaper.normalize(w) for w in words]
unique = set(normalized)
print(f"{len(words)} 个输入 → {len(unique)} 个唯一形态：{unique}")
# 4 个输入 → 1 个唯一形态：{'ᠰᠠᠢ᠍ᠢ᠍ᠠ᠌'}

命令行

pip install mongol-norm 之后,mongol-norm 命令即在 PATH 上(不安装也可用 python -m mongol_norm.shaper ...)。

# 直接传文本
mongol-norm shape 'ᠰᠠᠢᠨ'                   # → S+A+I+I+A
mongol-norm normalize 'ᠰᠡᠢᠨ'               # 输出 canonical
mongol-norm normalize --allow-fallback 'ᠰᠡᠢᠨ'  # 若触发fallback则原样返回
mongol-norm normalize-written-units 'B+Aa'  # → ᠪᠠ᠋
mongol-norm normalize-written-units 'BZwj'  # 紧凑PascalCase单元串
mongol-norm normalize-text 'Hello ᠰᠡᠢᠨ'    # 混合文字
mongol-norm normalize-text --allow-fallback -i in.txt

# 管道 / 标准输入(文本位置写 `-`)
echo 'ᠰᠡᠢᠨ' | mongol-norm normalize -
echo 'B+Aa' | mongol-norm normalize-written-units -
cat doc.txt | mongol-norm normalize-text -

# 文件输入 / 输出
mongol-norm normalize-text -i in.txt -o out.txt

# 批量:一行一词输入,一行一个 canonical 输出
mongol-norm normalize --batch -i words.txt -o canonical.txt
# 书写单元批量:每行一个紧凑或`+`分隔序列
mongol-norm normalize-written-units --batch -i units.txt -o canonical.txt

# 视觉等价检查(相同退出码 0,不同退出码 1)
mongol-norm same 'ᠰᠠᠢᠨ' 'ᠰᠡᠢᠨ'

normalize-written-units接受紧凑PascalCase或显式+边界。紧凑输入必须只有 一种合法切分；存在歧义时fail closed，调用方须改用+。解析后继续执行与 normalize_written_units()相同的exact-shape校验，因此语法合法的unit stream 若没有canonical MNG编码仍会被拒绝。

normalize(单词模式)会跳过非蒙古文字符,多行文件直接喂给它会被当成一整个词;一行一词的文件请用 --batch,自由文本请用 normalize-text。

运行测试

cd mongol-norm

# 手写 shaper / same_shape / normalize 测试
python -m unittest tests.test_shaper -v

# normalize 性质:往返 + 同 shape 同输出 + 前缀稳定
python -m unittest tests.test_round_trip

# normalize 表导出(compute == load)
python -m unittest tests.test_normalize_table

# mongfontbuilder core-hud(225)+ GB/T 25914-2023 eac-hud(3513,5 个 UTN-xfail)
python -m unittest tests.test_core_hud tests.test_eac_hud

# 或一次跑全部(自动发现所有 tests/test_*.py)
python -m unittest discover -s tests -p 'test_*.py'

手写套件覆盖范围:

当前总数: 214 个测试(单元 + 性质 + 225 core-hud + 3513 eac-hud 语料跑批), 在 Python 3.9 – 3.13 上全绿。

应用场景

• 搜索与检索 — 为每个可见词形建立唯一索引键
• 文本去重 — 检测编码不同但外形相同的词
• 拼写检查 — 规范化后再查词典
• 语料库语言学 — 一致的词频统计
• OCR 后处理 — 标准化可能使用不一致编码的 OCR 输出
• 输入法引擎 — 验证和规范化用户输入

项目结构

mongol-norm/                          # 仓库 = 包(单一自包含)
├── .github/workflows/test.yml        # CI: 每次 push 跑 Python 3.9-3.13
├── pyproject.toml
├── mongol_norm/
│   ├── shaper.py                     # tokenize / assign_positions / shape / normalize
│   ├── rules.py                      # 5 步 shaping 阶段 (iii1..iii5) 镜像 iii.py
│   ├── _data.py                      # 内置 JSON 的加载器
│   └── data/                         # 内置 shaping + normalize 数据
│       ├── MNG.json  TOD.json  SIB.json  MCH.json
│       └── MNG.normalize.json        # 逐单元 normalize 表
├── scripts/                          # 仅开发用的生成脚本 (preprocess, gen_normalize_table)
├── docs/data-format.md               # JSON schema, 供其他语言移植
└── tests/
    ├── test_shaper.py  test_round_trip.py  test_normalize_table.py
    ├── test_written_units_api.py
    ├── test_core_hud.py  test_eac_hud.py
    └── data/{core,eac}-hud.tsv       # 来自 mongfontbuilder

mongol-norm 没有运行时依赖 —— shaping/normalize JSON 内置在 mongol_norm/data/。安装:pip install mongol-norm。

数据来源与致谢

• UTN #57 v4 — Unicode 技术注释：蒙古文编码与字形化。蒙古文 shaping 规则的权威规范。
• mongfontbuilder(Kushim Jiang)— mongol_norm/data/ 内置扁平变体表的来源(从 data.variants / data.particles 预处理而来), 同时也是我们 vendor 进 tests/data/ 的 core-hud.tsv / eac-hud.tsv 回归套件的来源。UTN #57 和 mongfontbuilder 的作者是同一人。
• GB/T 25914—2023 — 中国国家标准：传统蒙古文名义字符、表现字符和控制字符使用规则; EAC 一致性测试集的来源。
• Claude Code — 本项目使用 AI 辅助开发。shaping 规则来源于上述数据；Claude Code 用于实现和组织引擎代码。

支持的语种

环境要求

• Python 3.7+(CI 实测矩阵: 3.9 / 3.10 / 3.11 / 3.12 / 3.13)
• 无运行时依赖(shaping/normalize 数据已内置)

许可证

MIT License —— 见 LICENSE。

整形规则与内置数据派生自 mongfontbuilder(MIT)和 UTN #57,其许可证要求的署名保留在 NOTICE 中。