hmaraniam 🇲z

High-precision, zero-dependency language identification library for Hmar.

"Hmar a ni am?" — "Is it Hmar?"

hmaraniam is a lightweight Python library designed to accurately distinguish Hmar text from English and other Kuki-Chin / Zo languages (Mizo, Kuki, Paite, Vaiphei).

Key Features

• Microsecond Speed: $O(1)$ dictionary lookups with no heavy ML dependencies (PyTorch/TensorFlow free).
• Dual-Lens Diacritic Engine: Reports both casual_hmar_ratio (ASCII-normalized for standard QWERTY typing) and formal_hmar_ratio (exact diacritic matching for formal literary text).
• Dual Continuous Confidence Metrics: Disambiguates language metrics into hmar_confidence (permanent metric answering "How confident are we that this text is Hmar?") and detected_language_confidence (confidence in the overall classification choice).
• Standardized Schema: Guarantees an immutable JSON output structure across all calls, including non_hmar_words_count and diacritic_words_count.
• Extensible & Customizable: Allows developers to supply custom_unigrams, extra_unigrams, custom_stopwords, or disable default stopwords.
• Dual Offline/CDN Architecture: Automatically syncs with the live hmar-heritage-org/hmaraniam unigram dataset via jsDelivr CDN, with automatic local disk caching and bundled fallback.

Installation

pip install hmaraniam

Standard Output Schema

{
  "language": "hmar",
  "hmar_confidence": 0.9842,
  "detected_language_confidence": 0.9842,
  "mode": "basic",
  "scores": {
    "casual_hmar_ratio": 0.9524,
    "formal_hmar_ratio": 0.8095,
    "english_stopword_ratio": 0.0000,
    "sibling_zo_stopword_ratio": 0.0000,
    "unknown_words_ratio": 0.0476,
    "total_words": 21,
    "hmar_words_count": 20,
    "non_hmar_words_count": 1,
    "unknown_words_count": 1,
    "english_stopwords_count": 0,
    "sibling_zo_stopwords_count": 0,
    "hmar_diacritic_words_count": 17,
    "non_hmar_diacritic_words_count": 0,
    "total_diacritic_words_count": 17
  }
}

Usage

Quick Start

import hmaraniam

# Authentic text from L. Keivom archive
sample_text = "Khawvel fe dan phung ei en chun, ram le hnam damna thuruk chu lien lema intel le insung khawm a nih."

result = hmaraniam.detect(sample_text)

print(result)

Custom Unigrams & Stopwords

from hmaraniam import Detector

# Provide custom unigrams or extra domain vocabulary
detector = Detector(
    mode="basic",
    extra_unigrams=["customworda", "customwordb"],
    custom_stopwords=["and", "the", "with"],
    disable_default_stopwords=False
)

result = detector.detect("Khawvel fe dan phung...")

Modes & Advanced Options

from hmaraniam import Detector

# Basic Mode (Active default ~30k core unigrams)
basic_detector = Detector(mode="basic")

# High Mode (Scans data/shards/ and loads all available unigram shards, falling back seamlessly to basic)
high_detector = Detector(mode="high")

# Offline-only mode (uses cached/bundled dataset without network calls)
offline_detector = Detector(offline_only=True)

Empirical Benchmarks & Performance

hmaraniam has been empirically validated across both controlled parallel corpus datasets (Parallel Zo Bibles across multiple literary genres) and unfiltered real-world web archives (~1,300 scraped articles and raw HTML pages across 5 major publishers).

1. Controlled Parallel Zo Bible Benchmark

Evaluated across parallel chapters (Genesis 1, Exodus 20, Matthew 5, Luke 2, Romans 8, Revelation 21) across 10 parallel Bible translations in 8 Zo languages + English:

Cognate Resolution: Closely related sibling languages like Mizo share up to 78% unigram overlap with Hmar. hmaraniam cleanly resolves sibling Zo languages without requiring massive full dictionaries by combining vocabulary completeness (unknown_words_ratio $\le 0.18$) with curated structural markers (sibling_zo_stopwords).

2. Real-World Web Archive Benchmark (~1,300 Scraped Documents)

Evaluated on real-world scraped web archives across 5 major Hmar/Zo web publishers:

• Literary Archives: Archives like L. Keivom and Inpui Journal feature rich Hmar prose and opinion articles, achieving high Hmar classification rates ($42.6% - 66.3%$) and high token ratios ($63.7% - 79.3%$).
• Community News & Recruitment Notices: Community portals like Hmarram and Virthli publish predominantly in English (recruitment notices, exam guidelines, press statements). hmaraniam accurately tags these as "english" or "other" without false-positive over-classification.

Error Handling

hmaraniam provides clear, descriptive error messages:

import hmaraniam

# Raises ValueError for unsupported modes
try:
    hmaraniam.detect("Text", mode="ultra")
except ValueError as e:
    print(e)

# Raises TypeError for non-string input
try:
    hmaraniam.detect(12345)
except TypeError as e:
    print(e)

License

Published under the MIT License by the Hmar Heritage Project.