hmaraniam 🇲z

Zero-dependency language identification library for Hmar.

"Hmar a ni am?" — "Is it Hmar?"

hmaraniam is a lightweight Python library that identifies Hmar text and distinguishes it from English and related Kuki-Chin / Zo languages (such as Mizo, Kuki, Paite, and Vaiphei).

Features

• Fast dictionary lookups: Uses $O(1)$ set matching with no machine learning dependencies (PyTorch and TensorFlow free).
• Dual diacritic scoring: Reports casual_hmar_ratio (ASCII-normalized for standard QWERTY typing) and formal_hmar_ratio (exact diacritic matches for formal text).
• Separate confidence scores: Separates overall classification confidence (detected_language_confidence) from Hmar-specific confidence (hmar_confidence).
• Consistent JSON output: Returns the same dictionary structure for every call, including word counts and diacritic breakdowns.
• Custom unigrams & stopwords: Pass custom unigram sets, extra domain vocabulary, or custom stopword lists.
• Offline & CDN dataset loading: Syncs unigram sets via jsDelivr CDN with local disk caching and bundled offline fallbacks.

Design Principles

• Language ID vs. Spell Correction: hmaraniam measures vocabulary identity ("Is this text Hmar?"). It is not a spell checker and does not modify typos, character variants (acute á, grave à, circumflex â), or mobile keyboard codepoints (ṭ vs ţ).
• ASCII Normalization (casual_hmar_ratio): Mobile keyboards produce varying accent codepoints. Stripping diacritics (strip_diacritics) allows consistent vocabulary evaluation across devices.
• 1-Token-Per-Row Boundaries: To evaluate hyphenated (mithiem-hai), spaced (mithiem hai), or compound (mithiemhai) terms directly, hmaraniam accepts 1-token-per-row inputs (JSON, CSV, TXT, Python lists) without re-tokenizing.
• Vocabulary Identity vs. Grammar: hmaraniam measures dictionary presence and token overlap, not syntax or semantics. A random sequence of valid Hmar words yields a high vocabulary score regardless of grammatical structure.

Installation

pip install hmaraniam

Output Schema

{
  "language": "hmar",
  "hmar_confidence": 0.9842,
  "detected_language_confidence": 0.9842,
  "mode": "basic",
  "scores": {
    "casual_hmar_ratio": 0.9524,
    "formal_hmar_ratio": 0.8095,
    "english_stopword_ratio": 0.0000,
    "sibling_zo_stopword_ratio": 0.0000,
    "unknown_words_ratio": 0.0476,
    "total_words": 21,
    "hmar_words_count": 20,
    "non_hmar_words_count": 1,
    "unknown_words_count": 1,
    "english_stopwords_count": 0,
    "sibling_zo_stopwords_count": 0,
    "hmar_diacritic_words_count": 17,
    "non_hmar_diacritic_words_count": 0,
    "total_diacritic_words_count": 17
  }
}

Usage

Quick Start

import hmaraniam

# Authentic text quote from L. Keivom archive (Coleman Factor, 2002)
sample_text = "Khawvel fe dan phung ei en chun, ram le hnam damna thuruk chu lien lema intel le insung khawm, zai khat le trong khata luong khawm a nih."

result = hmaraniam.detect(sample_text)
print(result)

1-Token-Per-Row Inputs

When token boundaries are pre-defined (such as distinguishing "mithiem-hai" vs "mithiem hai" vs "mithiemhai"), hmaraniam evaluates 1-token-per-row inputs without internal re-tokenization:

Expected File Formats & Code Examples

1. JSON Array File (tokens.json):

   [
     "khawvel",
     "fe",
     "dan",
     "mithiem-hai",
     "pathien",
     "hnenah"
   ]
   

   Usage: hmaraniam.detect("tokens.json") or CLI hmaraniam tokens.json

2. CSV File (tokens.csv):

   token
   khawvel
   fe
   dan
   mithiem-hai
   pathien
   hnenah
   

   Usage: hmaraniam.detect("tokens.csv") or CLI hmaraniam tokens.csv

3. Line-Delimited TXT File (tokens.txt - 1 word per line):

   khawvel
   fe
   dan
   mithiem-hai
   pathien
   hnenah
   

   Usage: hmaraniam.detect("tokens.txt") or CLI hmaraniam tokens.txt

4. Python List (List[str]):

   tokens = ["mithiem-hai", "pathien", "hnenah", "khawvel"]
   result = hmaraniam.detect(tokens)
   

Un-tokenized Raw Text Documents

For raw text files or strings (article.txt, raw text string, or stdin pipe), hmaraniam extracts word tokens using word-boundary regex matching:

# Raw text string evaluation
result = hmaraniam.detect("Khawvel fe dan phung ei en chun, ram le hnam damna thuruk...")

# Raw text article file evaluation
result = hmaraniam.detect("path/to/article.txt")

Custom Unigrams & Stopwords

from hmaraniam import Detector

# Provide custom unigrams or extra domain vocabulary
detector = Detector(
    mode="basic",
    extra_unigrams=["customworda", "customwordb"],
    custom_stopwords=["and", "the", "with"],
    disable_default_stopwords=False
)

result = detector.detect("Khawvel fe dan phung...")

Modes & Advanced Options

from hmaraniam import Detector

# Basic Mode (Default ~30k core unigrams)
basic_detector = Detector(mode="basic")

# High Mode (Loads extended unigram shards, falling back to basic if unavailable)
high_detector = Detector(mode="high")

# Offline-only mode (uses cached or bundled dataset without network calls)
offline_detector = Detector(offline_only=True)

Benchmarks & Evaluation

1. Parallel Zo Bible Passages

Evaluated across parallel chapters (Genesis 1, Exodus 20, Matthew 5, Luke 2, Romans 8, Revelation 21) across 10 Bible translations in 8 Zo languages + English:

Cognate Resolution: Related Zo languages like Mizo share up to 78% vocabulary overlap with Hmar. hmaraniam distinguishes sibling Zo languages by combining vocabulary coverage (unknown_words_ratio $\le 0.18$) with structural markers (sibling_zo_stopwords).

2. Web Archive Evaluation (~1,300 Documents)

Evaluated on scraped web documents from 5 Hmar/Zo web publishers:

• Literary Archives: Sites like L. Keivom and Inpui Journal contain Hmar prose, resulting in higher Hmar classification rates (42%–66%).
• Community News & Job Alerts: Portals like Hmarram and Virthli publish mostly recruitment notices and exam guidelines in English, which classify as "english" or "other".

Error Handling

hmaraniam raises standard Python exceptions:

import hmaraniam

# Raises ValueError for unsupported modes
try:
    hmaraniam.detect("Text", mode="ultra")
except ValueError as e:
    print(e)

# Raises TypeError for non-string input
try:
    hmaraniam.detect(12345)
except TypeError as e:
    print(e)

License

Published under the MIT License by the Hmar Heritage Foundation.