zomi-syl 0.1.0

A modular, dialect-aware Zomi syllabification library with rule-based, CRF, FST, and neural backends.

⭐ README.md (v0.1.0 — Pre‑Release)

---

zomi‑syl

A modular, dialect‑aware Zomi syllabification library with rule‑based and CRF backends.

zomi-syl provides a production‑ready syllabifier for Zomi, supporting multiple dialects, multiple backends, and a clean, extensible architecture. It includes:

• A fast rule‑based syllabifier
• A statistical CRF syllabifier
• A unified API
• A full CLI
• A backend registry
• Benchmarking tools
• Dialect profiles
• A clean, documented developer workflow

---

🚀 Features

• Multiple backends: rule‑based, CRF, transformer‑ready
• Dialect‑aware syllabification (csy [Siyin], ctd [Tedim] , gnb [Gangte], kmm [Kom Rem], pck [Paite], vap [Vaiphei], smt [Simte], tcz [Thado/Thadou], zom [Zo/Zou], [Mate], [Thangkhal], Zolai Standard, Myanmar Zomi, India Zomi)
• Unified API (zs.syllabify(), zs.analyze())
• Full CLI (zomi-syl syllabify, zomi-syl models benchmark, zomi-syl models compare)
• Benchmarking & evaluation tools
• Extensible backend architecture
• Clean developer documentation

---

📦 Installation

pip install zomi-syl

---

🧠 Quick Start

Syllabify a word

zomi-syl syllabify itna

Analyze a word

zomi-syl analyze itna --json

Batch syllabify

zomi-syl batch words.txt --output out.txt

---

🧰 Python API

import zomi_syl as zs

zs.syllabify("itna")
zs.analyze("itna")

---

🧩 Backends

zomi-syl supports multiple backends through a unified registry:

• rule — deterministic rule‑based syllabifier
• crf — statistical CRF syllabifier
• transformer — placeholder for future transformer models

List available backends:

zomi-syl models list

Show backend metadata:

zomi-syl models info crf

---

📊 Benchmarking

Single backend

zomi-syl models benchmark crf

Compare multiple backends

zomi-syl models compare rule crf

Compare all backends

zomi-syl models compare --all

---

🩺 Diagnostics

Run a full backend self‑test:

zomi-syl models doctor

This checks:

• registry integrity
• model metadata
• backend loadability
• single prediction
• batch prediction

---

🌏 Dialect Profiles

Profiles live under:

src/zomi_syl/profiles/

Supported dialects:

• Gangte | Not Yet
• Kom | Not Yet
• Mate | Not Yet
• Paite | Yes
• Simte | Not Yet
• Siyin | Not Yet
• Tedim | Yes
• Thangkhaal | Not Yet
• Thado/Thadou | Not Yet
• Vaiphei | Not Yet
• Zo/Zou | Not Yet
• India Zomi | Not Yet
• Myanmar Zomi | Not Yet
• Zolai Standard | Not Yet

Eventhough some dialects are not yet supportted, zomi-syl will give higher 90% accurarcy for all the dialects.

List profiles:

zomi-syl profiles list

Show profile info:

zomi-syl profiles info tedim

---

🧪 Testing

Run all tests:

pytest

Golden CRF regression data:

tests/golden/crf_golden.tsv

---

🗂 Project Structure

src/zomi_syl/
    api.py
    cli.py
    backends/
    profiles/
    models/
    evaluation/
    rule_based/
    utils/
    ...
scripts/
docs/
tests/
training/

---

🛠 Development

Developer documentation lives in:

docs/Developer/

Key guides:

• Adding new backends
• Unified Metadata Schema (UMS)
• CRF training
• Backend loader
• Test templates

---

📄 Changelog

The changelog is generated automatically:

make changelog

Template:

docs/Developer/CHANGELOG_template.md

---

📦 Release Checklist

See:

docs/RELEASE_CHECKLIST_v0.1.0.md

---

📜 License

MIT License — see LICENSE.

---

🙌 Contributing

See:

CONTRIBUTING.md

---

🔗 Command Reference

Full CLI command tree:

zomi-syl
│
├── syllabify
├── analyze
├── batch
├── benchmark
│
├── profiles list|info|validate
├── datasets list|download|validate
│
├── config show|path|validate|set
├── cache info|clear|remove
│
├── validate
├── download
├── version
│
└── models
    ├── list
    ├── info
    ├── benchmark
    ├── compare
    └── doctor