vi-textmatch

Vietnamese text normalization and type-aware comparison. Zero dependencies, standard library only.

Built for document verification — checking that a drafted contract matches the source record — where "close enough" is the right answer for a person's name and a catastrophically wrong answer for an ID number.

The problem

Two strings that look identical on screen are often not equal in memory:

"Huệ" == "Huệ"   # False

One is NFC (ệ = 1 codepoint), the other NFD (e + combining dot below + combining circumflex = 3). macOS filenames are NFD; most other sources are NFC.

And Vietnamese tone marks have two accepted positions in open syllables:

"hoà" == "hòa"   # False — same word, different bytes

Meanwhile, applying a naive oà → òa rewrite corrupts closed syllables: hoàn → hòan, suýt → súyt, huỳnh → hùynh — none of which are Vietnamese words. Worse, the corruption creates false matches between different words.

Install

pip install vi-textmatch

Python 3.11+. No third-party dependencies.

Usage

from vi_textmatch import normalize, compare, Kind

normalize("Nguyễn Thị  Hoà ")     # 'nguyễn thị hòa'
normalize("hoàn")                  # 'hoàn'  — closed syllable, untouched
normalize("suýt")                  # 'suýt'  — closed syllable, untouched

compare("Nguyễn Thị Hòa", "nguyễn thị hoà", kind=Kind.TEXT)
# Verdict(match=True,  level='IDENTICAL')

compare("Nguyễn Thị Hòa", "Nguyễn Thị Hoa",  kind=Kind.TEXT)
# Verdict(match=True,  level='TYPO')        — one character off, likely OCR/typing

compare("Nguyễn Thị Hòa", "Trần Văn Bình",   kind=Kind.TEXT)
# Verdict(match=False, level='DIFFERENT')   — different person

compare("031234567890", "031234567891", kind=Kind.ID)
# Verdict(match=False)   — one digit apart is a different person, not a typo

compare("2.500.000.000", "2,500,000,000", kind=Kind.MONEY)   # match
compare("2.500.000.000", "2.600.000.000", kind=Kind.MONEY)   # no match

kind is keyword-only and has no default. Calling compare(a, b) raises TypeError rather than silently falling back to fuzzy text matching.

Comparison kinds

TEXT grades the difference so you can act on it:

Similarity is meaningless for identifiers and amounts, so ID, MONEY and DATE never use it. MONEY and DATE raise on input they cannot parse; they never silently degrade to string comparison.

Normalization pipeline

normalize() applies four steps, in this order:

1. NFC — compose combining marks. Must come first: step 2 matches single codepoints.
2. Tone placement — rewrite hoà → hòa only in open syllables.
3. Whitespace — strip nbsp/zero-width, collapse runs, trim.
4. casefold — must come last: step 2 needs the original case to preserve it.

Each step is also exported individually.

Tone placement rule

The tone mark sits on the syllable nucleus. Two written forms exist only for oa, oe, uy in an open syllable (nothing follows the pair). In a closed syllable there is exactly one correct form and it must never be rewritten:

This library deliberately avoids the labels "old style" and "new style": published sources use them in opposite directions. Positions are named explicitly instead.

See research/QUY-TAC.md for the derivation and sources.

Scope

This library knows orthography, not vocabulary.

It does: normalize spelling variants, compare by data type, grade differences.

It does not: know whether a string is a real Vietnamese word, find names inside text (that is NER — see underthesea), correct spelling, or understand meaning.

Development

PYTHONPATH=src python3 -m unittest discover tests

License

MIT