mongol-norm 0.0.1

Shape-aware normalizer for Traditional Mongolian script. Implements the UTN #57 v4 shaping pipeline in pure Python.

mongol-norm

English | 中文

---

English

Status

Both halves of the library — shaping and normalization (MNG / Hudum) — are verified against the same upstream corpora. CI runs the full suite on Python 3.9 – 3.13 on every push.

✅ Shaping

shape() and same_shape() are cross-validated against two upstream TSV suites:

Suite	Cases	Pass	Notes
mongfontbuilder/core-hud.tsv	225	100%	curated regression set
mongfontbuilder/eac-hud.tsv (GB/T 25914-2023)	3513	100%	5 cases excluded as UTN ↔ EAC xfail, matching mongfontbuilder's own pytest.mark.xfail set
Hand-written unit tests	—	100%	shape / same_shape / joiner tokens (nirugu, ZWJ)

✅ Normalization — same guarantees, machine-checked

normalize() / normalize_text() are a pure function of shape with invariants checked in CI over every corpus encoding:

Property	Result
Round-trip — shape(normalize(x)) == shape(x)	3757 / 3757 corpus encodings (100%)
Shape-canonicity — same shape ⟹ same Unicode output	1993 / 1993 shape groups (100%)
Prefix-stability — word and word+suffix share their prefix encoding	99.87% of real corpus pairs

Scope note: normalization is implemented for MNG (Hudum) only — Todo / Sibe / Manchu load shaping rules but have no normalizer yet.

This project was generated with Claude Code (AI-assisted coding). The tests and key parts of the core code have been manually reviewed, and test coverage is extensive (corpus round-trip / shape-canonicity / prefix-stability plus the upstream cross-implementation suites). Treat this as a preview release — it should be fine for normal use; if you hit a problem, please open an issue or PR. Shaping logic is derived from UTN #57 v4 and mongfontbuilder.

---

Why This Project Exists

Traditional Mongolian script in Unicode has a fundamental problem: the same visible word can be encoded in multiple different Unicode sequences. This happens because:

1. Letters share glyphs — A and E look identical in medial and final positions; O and U share forms; QA and GA share forms depending on vowel harmony.
2. Multiple encoding paths — The same tooth glyph (I) can be encoded as I, YA+FVS1, or even two separate I characters.

3. Redundant FVS usage — Free Variation Selectors (FVS1–FVS4) can create equivalent sequences that render identically.

This means:

• Search fails: Searching for "sain" (one encoding) won't find the same word in another encoding, even though they look identical.
• Deduplication breaks: The same word has multiple Unicode representations.
• Indexing is unreliable: Different encodings of the same word produce different keys.

What This Project Does

This is a shaping-aware normalizer for Traditional Mongolian. It:

1. Shapes the input using the full UTN #57 v4 shaping process (5-step conditional mapping)
2. Compares glyph sequences to detect identical visual forms
3. Normalizes to a canonical, human-readable, bare Unicode encoding

Example: All five of these encode the word "sain" (good) and look identical:

How It Works

The normalizer implements a lightweight Mongolian shaping engine — equivalent to what HarfBuzz does with a font file, but using only the rule data from UTN #57 v4 and the mongfontbuilder project. No font files needed.

Shaping Pipeline (UTN #57 v4 Mongolian-Specific Phase)

1. Chachlag — Suffix forms for A/E after MVS (Mongolian Vowel Separator)
2. Syllabic — Consonant/vowel context: onset, devsger, marked, masculine/feminine harmony, dotless
3. Particle — MVS particle dictionary lookup for specific suffix words
4. Devsger — I after a vowel (vowel_devsger) gets double-tooth form: I → I+I
5. Post-bowed — Vowel forms change after bowed consonants (G, B, K, P, F)

Normalization Strategy

normalize is a pure function of shape: any two encodings that shape identically produce the same Unicode output, and the output always round-trips — shape(normalize(x)) == shape(x). It is also prefix-stable. When these goals conflict the priority is round-trip > prefix-stable > shortest.

Per word:

1. shape the input into its written-unit sequence. Structural characters — MVS, nirugu, ZWJ — appear verbatim as mvs / nirugu / zwj tokens (nirugu renders a visible stem; all three are the evidence for a neighbour's init/medi/fina form). Split the shape at these tokens into chains; the tokens themselves are copied through unchanged.
2. encode each chain (right-to-left, so appending a suffix can't disturb what precedes it):
   1. partition + table lookup — the primary path. At each position take the single unit if the table has it (preferred — clean output), else the longest available multi-unit entry, and look up (position, written-unit) → (letter, FVS) in an FVS-pinned table. Each value renders its unit regardless of neighbours, so the result is a deterministic, O(N), prefix-stable function of the shape.
   2. velar-feminine refinement — a G/Gx velar's forward-coupled vowel (a/o/u) is swapped to its feminine partner (e/oe/ue) for clean output.
   3. verify — reshape the candidate in full context; accept only if it equals the target chain shape.
   4. gap chains — a handful of chains can't be expressed by table entries alone (bowed-consonant + following-vowel forms like B+A; Sh+I clusters). These fall back to an exhaustive search over letter partitions and FVS combinations (~13 corpus chains). A letter next to a joiner simply looks its unit up at the shifted (joined) position — no wrapping tricks needed.
3. post-MVS suffix rule — a chain directly after MVS takes its standalone canonical (drop the MVS, normalize, re-attach), so the spelling never depends on MVS. One exception: chachlag Aa after MVS is written the bare letter a. (The isolate-I → i+FVS1 spelling is pinned in the table itself — no post-processing pass exists.)

Prefix-stability means: if word A = word B + a suffix and their shapes share a prefix, the shared region encodes identically except the single boundary unit whose position changes (final in B → medial in A). The per-unit table delivers this for free — each unit's encoding depends only on its own position, never on its neighbours.

How the table is built (the selection method): offline, a context-independence battery fills each (position, written-unit) slot with the (letter, FVS) — tried masculine-first, bare-first — that renders exactly that unit in every probed neighbour context. The result is exported and shipped as JSON; see MongolianShaper.compute_normalize_tables().

Note: output is FVS-pinned, not bare — each unit carries the selector that fixes its form independent of context. This is what makes "same shape ⟹ same Unicode" and prefix-stability hold. The per-unit table is exported as language-agnostic JSON (mongol_norm/data/MNG.normalize.json); schema + consuming algorithm are in docs/data-format.md, so ports in other languages can implement normalize with just a JSON parser.

Installation

mongol-norm is a single self-contained package — the shaping/normalize data is bundled, no separate data package or runtime dependency. Not on PyPI yet — install from source:

git clone https://github.com/Satsrag/mongol-norm.git
cd mongol-norm
pip install .

Usage

from mongol_norm import MongolianShaper

shaper = MongolianShaper(locale="MNG")  # Hudum Traditional Mongolian

# Shape: get written-unit sequence
shaper.shape("ᠰᠠᠢᠨ")
# → ['S', 'A', 'I', 'I', 'A']

# Compare: are two encodings visually identical?
shaper.same_shape("ᠰᠠᠢᠨ", "ᠰᠡᠢᠨ")
# → True

shaper.same_shape("ᠰᠠᠢᠨ", "ᠨᠠᠢᠮᠠ")
# → False

# Normalize: canonical bare Unicode
shaper.normalize("ᠰᠡᠢᠨ")
# → 'ᠰᠠᠢᠨ'

shaper.normalize("ᠰᠠᠶ᠋ᠢᠨ")
# → 'ᠰᠠᠢᠨ'

shaper.normalize("ᠰᠠᠶ᠋ᠶ᠋ᠨ")
# → 'ᠰᠠᠢᠨ'

Full-text normalization

normalize() operates on single words. For sentences, paragraphs, or mixed-script text, use normalize_text() — it normalizes each Mongolian word independently while preserving spaces, punctuation, and non-Mongolian text verbatim.

# Normalize a sentence (each Mongolian word normalized independently)
shaper.normalize_text("ᠰᠡᠢᠨ ᠨᠠᠢᠮᠠ")
# → 'ᠰᠠᠢᠨ ᠨᠠᠢᠮᠠ'

# Mixed script: non-Mongolian text preserved as-is
shaper.normalize_text("Hello ᠰᠡᠢᠨ world")
# → 'Hello ᠰᠠᠢᠨ world'

Batch normalization example

words = ["ᠰᠡᠢᠨ", "ᠰᠠᠢᠨ", "ᠰᠨ᠌ᠢᠢᠨ", "ᠰᠠᠶ᠋ᠢᠨ"]
normalized = [shaper.normalize(w) for w in words]
unique = set(normalized)
print(f"{len(words)} inputs → {len(unique)} unique form(s): {unique}")
# 4 inputs → 1 unique form(s): {'ᠰᠠᠢᠨ'}

Command line

After pip install ., the mongol-norm command is on PATH (or run python -m mongol_norm.shaper ... without installing).

# Inline text
mongol-norm shape 'ᠰᠠᠢᠨ'                   # → S+A+I+I+A
mongol-norm normalize 'ᠰᠡᠢᠨ'               # canonical form
mongol-norm normalize-text 'Hello ᠰᠡᠢᠨ'    # mixed-script

# Pipe / stdin (use `-` as the text)
echo 'ᠰᠡᠢᠨ' | mongol-norm normalize -
cat doc.txt | mongol-norm normalize-text -

# File in / out
mongol-norm normalize-text -i in.txt -o out.txt

# Batch: one word per line in, one canonical per line out
mongol-norm normalize --batch -i words.txt -o canonical.txt

# Visual-identity check (exit 0 if same, 1 if different)
mongol-norm same 'ᠰᠠᠢᠨ' 'ᠰᠡᠢᠨ'

normalize (single-word) skips non-Mongolian characters, so feeding it a multi-line file treats the whole thing as one word. Use --batch for one-word-per-line files, or normalize-text for free-form text.

Running Tests

cd mongol-norm

# Shaping + same_shape + normalize unit tests
python -m unittest tests.test_shaper -v

# Normalize properties: round-trip + shape-canonicity + prefix-stability
python -m unittest tests.test_round_trip

# Normalize-table export (compute == load)
python -m unittest tests.test_normalize_table

# mongfontbuilder core-hud (225) + GB/T 25914-2023 eac-hud (3513, 5 UTN-xfail)
python -m unittest tests.test_core_hud tests.test_eac_hud

# Or all together (auto-discovers every tests/test_*.py)
python -m unittest discover -s tests -p 'test_*.py'

The hand-written suite covers:

Test class	What it checks
TestShape	shape() returns correct written-unit sequence (sain variants, the 5 shaping phases step-by-step, UTN-vs-EAC divergences)
TestSameShape	same_shape() correctly identifies visually identical vs. distinct encodings
TestNormalize	normalize() produces canonical output; idempotency; normalized result matches original visually
TestNormalizeText	normalize_text() handles multi-word, mixed-script, punctuation, empty input; idempotency; word independence
TestNNBSP	NNBSP ↔ MVS equivalence (UTN model)

Current totals: 142 tests (unit + property + 225 core-hud + 3513 eac-hud corpus runners), all green on Python 3.9 – 3.13.

Use Cases

• Search & Retrieval — Index Mongolian text with unique keys per visual word
• Deduplication — Detect identical words encoded differently
• Spell Checking — Normalize before dictionary lookup
• Corpus Linguistics — Consistent word frequency counts
• OCR Post-processing — Standardize OCR output that may use inconsistent encodings
• Input Method Engines — Validate and normalize user input

Project Structure

mongol-norm/                          # the repo = the package (single, self-contained)
├── .github/workflows/test.yml        # CI: Python 3.9-3.13 on every push
├── pyproject.toml
├── mongol_norm/
│   ├── shaper.py                     # tokenize / assign_positions / shape / normalize
│   ├── rules.py                      # the 5 shaping phases (iii1..iii5) mirroring iii.py
│   ├── _data.py                      # loaders for the bundled JSON
│   └── data/                         # bundled shaping + normalize data
│       ├── MNG.json  TOD.json  SIB.json  MCH.json
│       └── MNG.normalize.json        # per-unit normalize table
├── scripts/                          # dev-only generators (preprocess, gen_normalize_table)
├── docs/data-format.md               # JSON schema, for other-language ports
└── tests/
    ├── test_shaper.py  test_round_trip.py  test_normalize_table.py
    ├── test_core_hud.py  test_eac_hud.py
    └── data/{core,eac}-hud.tsv       # vendored from mongfontbuilder

mongol-norm has no runtime dependencies — the shaping/normalize JSON is bundled in mongol_norm/data/. Not on PyPI yet — install from source.

Data Sources & Acknowledgments

• UTN #57 v4 — Unicode Technical Note: Encoding and Shaping of the Mongolian Script. The authoritative specification for Mongolian shaping rules.
• mongfontbuilder by Kushim Jiang — Source for the bundled flat variant tables in mongol_norm/data/ (preprocessed from data.variants / data.particles) and for the core-hud.tsv / eac-hud.tsv regression suites we vendor into tests/data/. Both UTN #57 and mongfontbuilder are authored by the same person.
• GB/T 25914—2023 — China national standard for Traditional Mongolian nominal characters; source of the EAC compliance test set.
• Claude Code — This project was developed with AI assistance. The shaping rules are derived from the above sources; Claude Code was used to implement and structure the engine.

Supported Locales

Locale	Script	Status
MNG	Hudum (Traditional Mongolian)	✅ Full shaping + normalization
TOD	Todo	⬜ Shaping rules generated, normalization WIP
SIB	Sibe	⬜ Shaping rules generated, normalization WIP
MCH	Manchu	⬜ Shaping rules generated, normalization WIP

Requirements

• Python 3.6+ (CI-tested on 3.9 / 3.10 / 3.11 / 3.12 / 3.13)
• No runtime dependencies (shaping/normalize data is bundled)

License

SIL Open Font License 1.1 (OFL-1.1) — consistent with upstream mongfontbuilder and UTN #57 sources.

---

中文

状态

库的两部分 —— 整形与规范化(MNG / Hudum)—— 都对照同一批上游语料验证。CI 在每次 push 上对 Python 3.9 – 3.13 跑完整套件。

✅ Shaping(整形)

shape() 和 same_shape() 对照两套上游 TSV 套件交叉验证:

套件	用例数	通过	说明
mongfontbuilder/core-hud.tsv	225	100%	精选回归集
mongfontbuilder/eac-hud.tsv (GB/T 25914-2023)	3513	100%	5 个 UTN ↔ EAC 分歧 case 跳过(跟 mongfontbuilder 自己的 pytest.mark.xfail 列表一致)
手写单元测试	—	100%	shape / same_shape / joiner token(nirugu、ZWJ)

✅ Normalization(规范化)— 与整形同级保障,机器验证

normalize() / normalize_text() 是 shape 的纯函数,以下不变量在 CI 中对每一条语料编码逐一验证:

性质	结果
往返 —— shape(normalize(x)) == shape(x)	3757 / 3757 语料编码(100%)
同形同码 —— shape 相同 ⟹ 输出 Unicode 相同	1993 / 1993 shape 组(100%)
前缀稳定 —— 词与词+后缀共享前缀编码	99.87% 真实语料词对

范围说明:规范化目前只实现了 MNG(Hudum)—— Todo / 锡伯文 / 满文已加载 shaping 规则,尚无规范化。

本项目由 Claude Code(AI 辅助编码)生成;测试与部分核心代码经人工审核,测试覆盖比较充分(语料往返 / 同形同码 / 前缀稳定 + 上游跨实现套件)。当前为预览版,正常使用应无问题;遇到问题欢迎提 issue 和 PR。Shaping 逻辑源自 UTN #57 v4 和 mongfontbuilder。

---

为什么做这个项目

传统蒙古文在 Unicode 中存在一个根本性问题：同一个可见词形可以用多种不同的 Unicode 序列编码。原因是：

1. 字母共享字形 — A 和 E 在中间和尾部位置外形完全相同；O 和 U 共享形态；QA 和 GA 根据元音和谐共享形态。
2. 多种编码路径 — 同一个齿形字形可以编码为 I、YA+FVS1，甚至两个独立的 I 字符。

3. 冗余的 FVS 使用 — 自由变体选择符（FVS1–FVS4）可以创建渲染结果完全相同的等价序列。

这意味着：

• 搜索失效：搜索同一个词的某种编码，找不到另一种编码，尽管它们外形完全一样。
• 去重失败：同一个词有多种 Unicode 表示。
• 索引不可靠：同一个词的不同编码产生不同的索引键。

这个项目做什么

这是一个形态感知的蒙古文规范化器。它：

1. 使用完整的 UTN #57 v4 shaping 过程（5 步条件映射）对输入进行字形化
2. 通过比较字形序列来检测视觉上相同的词形
3. 规范化为唯一的、人类可读的、bare Unicode 编码

示例：以下五种编码都表示 "sain"（好的），外形完全相同：

工作原理

本规范化器实现了一个轻量级蒙古文 shaping 引擎——功能相当于 HarfBuzz 配合字体文件所做的事情，但仅使用 UTN #57 v4 的规则数据和 mongfontbuilder 项目的变体数据。不需要字体文件。

Shaping 管线（UTN #57 v4 蒙古文特定阶段）

步骤	名称	说明
1	Chachlag	MVS（蒙古文元音分隔符）后的 a/e 后缀形态
2	Syllabic	辅音/元音上下文：onset/devsger/marked/阴阳和谐/dotless
3	Particle	MVS 小品词词典查找
4	Devsger	元音后的 i 获得双齿形态：I → I+I（vowel_devsger）
5	Post-bowed	弓形辅音(G/B/K/P/F)后的元音形态变化

规范化策略

normalize 是 shape 的纯函数:任意两个 shape 相同的编码,normalize 输出相同,且始终往返成立 —— shape(normalize(x)) == shape(x),同时前缀稳定。三者冲突时优先级:往返 > 前缀稳定 > 最短。

逐词:

1. shape 成书写单元序列。结构字符 —— MVS、nirugu、ZWJ —— 原样输出为 mvs / nirugu / zwj token(nirugu 是可见的连笔字形;三者都是邻居字母 init/medi/fina 形的依据)。按这些 token 切成 chain,token 本身原样拷贝。
2. 逐 chain 编码(从右往左,这样加后缀不影响前面):
   1. 划分 + 查表(主路径):每个位置优先取单单元(输出干净),否则取最长多单元,查 (位置, 书写单元) → (字母, FVS) 的 FVS 钉死表。每个值不依赖邻居就渲染出该单元 → 确定性、O(N)、前缀稳定。
   2. velar 阴性微调:G/Gx 前向耦合的元音(a/o/u)换成阴性(e/oe/ue),输出更干净。
   3. 校验:在完整上下文里重新 shape,只接受与目标 chain shape 一致的结果。
   4. 缺口 chain:少数 chain 仅靠表条目表达不了(弓形辅音+后随元音形如 B+A;Sh+I 簇),回退到对字母划分×FVS 组合的穷举搜索(语料中约 13 个)。紧邻 joiner 的字母只是按移动后的连接位置查表 —— 不再需要任何包裹技巧。
3. MVS 后缀规则:紧跟 MVS 的 chain 用其 standalone canonical(去掉 MVS、归一、再拼回),拼写不依赖 MVS。唯一例外:MVS 后的 chachlag Aa 写裸字母 a。(孤立 I → i+FVS1 的拼写已钉进表本身 —— 不存在后处理。)

前缀稳定的含义:若词 A = 词 B + 后缀,且二者 shape 共享前缀,则共享部分编码完全一致,只有那个位置发生变化的边界单元不同(在 B 里是词尾、在 A 里变词中)。逐单元表天然保证这点 —— 每个单元的编码只取决于它自己的位置,与邻居无关。

表是怎么来的(选择方法):离线跑一个 context 无关性电池 —— 对每个 (位置, 书写单元),按"阳性优先、裸形优先"挑出那个在所有探测邻居上下文里都恰好渲染出该单元的 (字母, FVS)。结果导出成 JSON 随包发布;见 MongolianShaper.compute_normalize_tables()。

注意:输出是 FVS 钉死而非 bare —— 每个单元都带着把字形固定住、不受上下文影响的选择符,这正是"同 shape ⟹ 同 Unicode"和前缀稳定成立的原因。逐单元表导出为语言无关的 JSON(mongol_norm/data/MNG.normalize.json),schema 与消费算法见 docs/data-format.md;其他语言只需一个 JSON 解析器即可实现 normalize。

安装

mongol-norm 是单一自包含包 —— shaping/normalize 数据已内置,没有独立数据包,也没有运行时依赖。还没发布到 PyPI,从源码本地安装:

git clone https://github.com/Satsrag/mongol-norm.git
cd mongol-norm
pip install .

使用方法

from mongol_norm import MongolianShaper

shaper = MongolianShaper(locale="MNG")  # Hudum 传统蒙文

# 字形化：获取书写单元序列
shaper.shape("ᠰᠠᠢᠨ")
# → ['S', 'A', 'I', 'I', 'A']

# 比较：两个编码视觉上是否相同？
shaper.same_shape("ᠰᠠᠢᠨ", "ᠰᠡᠢᠨ")
# → True

shaper.same_shape("ᠰᠠᠢᠨ", "ᠨᠠᠢᠮᠠ")
# → False

# 规范化：输出唯一的 bare Unicode
shaper.normalize("ᠰᠡᠢᠨ")
# → 'ᠰᠠᠢᠨ'

shaper.normalize("ᠰᠠᠶ᠋ᠢᠨ")
# → 'ᠰᠠᠢᠨ'

shaper.normalize("ᠰᠠᠶ᠋ᠶ᠋ᠨ")
# → 'ᠰᠠᠢᠨ'

全文规范化

normalize() 作用于单个词。对于句子、段落或混合文字文本，使用 normalize_text() ——它独立规范化每个蒙古文词，同时原样保留空格、标点和非蒙古文文本。

# 规范化句子（每个蒙古文词独立规范化）
shaper.normalize_text("ᠰᠡᠢᠨ ᠨᠠᠢᠮᠠ")
# → 'ᠰᠠᠢᠨ ᠨᠠᠢᠮᠠ'

# 混合文字：非蒙古文文本原样保留
shaper.normalize_text("Hello ᠰᠡᠢᠨ world")
# → 'Hello ᠰᠠᠢᠨ world'

批量规范化示例

words = ["ᠰᠡᠢᠨ", "ᠰᠠᠢᠨ", "ᠰᠨ᠌ᠢᠢᠨ", "ᠰᠠᠶ᠋ᠢᠨ"]
normalized = [shaper.normalize(w) for w in words]
unique = set(normalized)
print(f"{len(words)} 个输入 → {len(unique)} 个唯一形态：{unique}")
# 4 个输入 → 1 个唯一形态：{'ᠰᠠᠢᠨ'}

运行测试

cd mongol-norm

# 手写 shaper / same_shape / normalize 测试
python -m unittest tests.test_shaper -v

# normalize 性质:往返 + 同 shape 同输出 + 前缀稳定
python -m unittest tests.test_round_trip

# normalize 表导出(compute == load)
python -m unittest tests.test_normalize_table

# mongfontbuilder core-hud(225)+ GB/T 25914-2023 eac-hud(3513,5 个 UTN-xfail)
python -m unittest tests.test_core_hud tests.test_eac_hud

# 或一次跑全部(自动发现所有 tests/test_*.py)
python -m unittest discover -s tests -p 'test_*.py'

手写套件覆盖范围:

测试类	测试内容
TestShape	shape() 输出正确的书写单元序列(sain 变体、5 步 shaping 分步测试、UTN-vs-EAC 分歧)
TestSameShape	same_shape() 正确识别外形相同 vs 不同的编码
TestNormalize	normalize() 输出规范结果; 幂等性; 规范化后与原始词形视觉相同
TestNormalizeText	normalize_text() 处理多词、混合文字、标点、空输入; 幂等性; 词独立性
TestNNBSP	NNBSP ↔ MVS 等价性(UTN 模型)

当前总数: 142 个测试(单元 + 性质 + 225 core-hud + 3513 eac-hud 语料跑批), 在 Python 3.9 – 3.13 上全绿。

应用场景

• 搜索与检索 — 为每个可见词形建立唯一索引键
• 文本去重 — 检测编码不同但外形相同的词
• 拼写检查 — 规范化后再查词典
• 语料库语言学 — 一致的词频统计
• OCR 后处理 — 标准化可能使用不一致编码的 OCR 输出
• 输入法引擎 — 验证和规范化用户输入

项目结构

mongol-norm/                          # 仓库 = 包(单一自包含)
├── .github/workflows/test.yml        # CI: 每次 push 跑 Python 3.9-3.13
├── pyproject.toml
├── mongol_norm/
│   ├── shaper.py                     # tokenize / assign_positions / shape / normalize
│   ├── rules.py                      # 5 步 shaping 阶段 (iii1..iii5) 镜像 iii.py
│   ├── _data.py                      # 内置 JSON 的加载器
│   └── data/                         # 内置 shaping + normalize 数据
│       ├── MNG.json  TOD.json  SIB.json  MCH.json
│       └── MNG.normalize.json        # 逐单元 normalize 表
├── scripts/                          # 仅开发用的生成脚本 (preprocess, gen_normalize_table)
├── docs/data-format.md               # JSON schema, 供其他语言移植
└── tests/
    ├── test_shaper.py  test_round_trip.py  test_normalize_table.py
    ├── test_core_hud.py  test_eac_hud.py
    └── data/{core,eac}-hud.tsv       # 来自 mongfontbuilder

mongol-norm 没有运行时依赖 —— shaping/normalize JSON 内置在 mongol_norm/data/。还没上 PyPI, 从源码安装。

数据来源与致谢

• UTN #57 v4 — Unicode 技术注释：蒙古文编码与字形化。蒙古文 shaping 规则的权威规范。
• mongfontbuilder(Kushim Jiang)— mongol_norm/data/ 内置扁平变体表的来源(从 data.variants / data.particles 预处理而来), 同时也是我们 vendor 进 tests/data/ 的 core-hud.tsv / eac-hud.tsv 回归套件的来源。UTN #57 和 mongfontbuilder 的作者是同一人。
• GB/T 25914—2023 — 中国国家标准：传统蒙古文名义字符、表现字符和控制字符使用规则; EAC 一致性测试集的来源。
• Claude Code — 本项目使用 AI 辅助开发。shaping 规则来源于上述数据；Claude Code 用于实现和组织引擎代码。

支持的语种

Locale	文字	状态
MNG	Hudum（传统蒙文）	✅ 完整 shaping + 规范化
TOD	Todo（托忒文）	⬜ shaping 规则已生成，规范化开发中
SIB	Sibe（锡伯文）	⬜ shaping 规则已生成，规范化开发中
MCH	Manchu（满文）	⬜ shaping 规则已生成，规范化开发中

环境要求

• Python 3.6+(CI 实测矩阵: 3.9 / 3.10 / 3.11 / 3.12 / 3.13)
• 无运行时依赖(shaping/normalize 数据已内置)

许可证

SIL Open Font License 1.1 (OFL-1.1) — 跟上游 mongfontbuilder 和 UTN #57 一致。