moine 0.2.2

Lattice Path Edit Distance and romanization-aware string comparison

mòine

mòine is a Python and Rust library for romanization-aware string comparison.

It implements Lattice Path Edit Distance (LPED; Kaji, 2023), a distance metric that compares strings through possible reading paths rather than only through visible surface characters.

This is useful when romanized input and written Japanese or Chinese look far apart as strings, but stay close in reading space.

With the package installed and language artifacts downloaded:

>>> import moine

>>> moine.distance("moine", "モイニャ", lang="ja")
2

>>> moine.distance("もいにゃ", "モイニャ", lang="ja")
0

>>> moine.distance("weishiji", "威士忌", lang="zh")
0

>>> moine.distance("布納哈奔", "布納哈本", lang="zh")
0

Name

The project name comes from Moine, a peated malt from Bunnahabhain and one of the developer's favorite Scotch whiskies. In Japanese, the name has several plausible katakana renderings, such as モイニャ, モーイン, and モアンヌ, which makes it a fitting name for a project about readings, spelling variation, and ambiguity in input sequences.

Features

• Japanese comparison with UniDic-CWJ-derived reading artifacts and separate SudachiDict-derived artifacts.
• Chinese comparison with CC-CEDICT-derived no-tone pinyin artifacts.
• Plain string Levenshtein-compatible distance helpers.
• Lattice-aware Damerau-Levenshtein distance for adjacent transpositions.
• Combined min(surface Damerau-Levenshtein, LPED) scorer for candidate reranking.
• Normalized similarity / ratio helpers in 0.0..=1.0.
• RapidFuzz-inspired APIs such as cdist and partial matching helpers.

Installation

Install the Python package:

pip install moine
uv pip install moine

Install the Rust command-line tool:

cargo install moine

The packages do not bundle dictionary data. Download the language artifacts you need explicitly:

# Default Japanese artifact: UniDic-CWJ
uv run python -m moine download ja

# Explicit Japanese sources
uv run python -m moine download ja-unidic
uv run python -m moine download ja-sudachi

# Chinese artifact: CC-CEDICT
uv run python -m moine download zh

# Same selectors are available from the Rust CLI.
moine download ja
moine download ja-unidic
moine download ja-sudachi
moine download zh

ja is the short default selector for the current Japanese artifact, which is UniDic-CWJ. Use ja-unidic or ja-sudachi when the dictionary source should be explicit.

Quick Start

Use the top-level Python API when you want mòine to load the default dictionary for a language:

import moine

print(moine.distance("もいにゃ", "モイニャ", lang="ja"))  # 0
print(moine.ratio("ピィート", "ピート", lang="ja"))  # 0.7142857142857143
print(moine.partial_ratio("ウイスキー", "ういすきーをのんでいます", lang="ja"))  # 1.0
print(moine.distance("weishiji", "威士忌", lang="zh"))  # 0

Load a dictionary explicitly when you want to control startup cost or artifact location:

import moine

dictionary = moine.load_dict(lang="ja")
moine.set_default_dictionary(dictionary)

print(moine.distance("もいにゃ", "モイニャ", lang="ja"))  # 0

Use cdist for query-by-choice matrices:

import moine

scores = moine.cdist(
    ["もいにゃ", "ぴーと", "ピィート"],
    ["モイニャ", "ピート", "ピーと", "ピィート"],
    lang="ja",
    metric="damerau_distance",
    score_cutoff=1,
)

For search or entity matching, generate candidates with your existing system and use mòine as a reading-aware reranker:

import moine

query = "moine"
candidates = ["モイニャ", "モーイン", "モアンヌ", "ストイーシャ"]

scores = moine.cdist(
    [query],
    candidates,
    lang="ja",
    metric="distance",
)[0]

ranked = sorted(zip(candidates, scores), key=lambda item: item[1])
print(ranked)
# [('モイニャ', 2), ('モーイン', 2), ('モアンヌ', 3), ('ストイーシャ', 7)]

Score interpretation is intentionally simple: distance=0 means the best reading paths are identical, combined_distance is the minimum of surface Damerau-Levenshtein and LPED, distance metrics are smaller-is-better, ratio and normalized_similarity are in 0.0..=1.0 and larger-is-better, and score_cutoff filters in the RapidFuzz style.

Dictionary reading expansion is bounded by controls such as max_paths, max_readings_per_segment, and longest_only. When those controls prune candidates, a score is exact for the retained reading paths but may differ from an exhaustive score over every dictionary reading. Candidate retention follows deterministic expansion order; it is not a frequency or probability ranking.

Command Line

Most users only need the public runtime commands:

moine download ja
moine download zh
moine list
moine where
moine compare --left "もいにゃ" --right "モイニャ" \
  --artifact-metadata /path/to/moine-unidic-cwj-202512/metadata.yaml
moine chinese-compare --left weishiji --right 威士忌 \
  --artifact-metadata /path/to/moine-cedict-20260520/metadata.yaml

Use moine download ja-unidic for explicit UniDic-CWJ and moine download ja-sudachi for SudachiDict-full.

The artifact bundle, verification, archive, and raw-dictionary inspection commands are maintainer-facing tools for producing and checking release assets. They are documented in docs/development.md and docs/release_process.md.

The public comparison commands can also emit lattice graphs: moine compare --romaji-lattice <PATH> --output-format <dot|svg|png> for Japanese romaji lattices, and moine chinese-compare --pinyin-lattice <PATH> --output-format <dot|svg|png> for Chinese pinyin lattices. Writing SVG or PNG graphs requires the Graphviz dot command to be available in PATH; DOT output does not require that runtime dependency. See CLI usage for rendered examples.

Documentation

• Project documentation
• Installation
• Python usage
• CLI usage
• API reference
• Rust usage
• Dictionary artifacts
• Browser demo
• Rust docs

Developer and maintainer notes live under docs/, starting with docs/development.md and docs/release_process.md. See CONTRIBUTING.md before opening pull requests.

How It Differs From RapidFuzz

RapidFuzz is the better fit when both inputs should be compared directly as surface strings and you need a broad set of highly optimized fuzzy-matching scorers. mòine focuses on a narrower problem: comparing strings through possible reading paths before edit distance is computed.

Benchmark

The scoring table was recorded on 2026-06-28. It reports scoring time only; dictionary loading is shown separately below.

[!IMPORTANT] RapidFuzz measures surface Levenshtein distance, so it is expected to be much faster. Treat this as a reference for mòine's dictionary-backed reading edit distance, not a same-task speed comparison.

uv run python -m moine download ja
uv run --python python3.14 --with rapidfuzz \
  python scripts/benchmark_distances.py \
  --loops 10000

This is a quick local benchmark command. The release-wheel command used for the recorded table lives in development notes.

Method	mean (±std)	relative
RapidFuzz Levenshtein	0.15 ± 0.01 us/call	1.00x
mòine ja distance	26.08 ± 33.82 us/call	177x

Fresh dictionary loads from the standard installed artifacts, recorded on 2026-06-28 and measured over 100 loads:

Dictionary	mean (±std)
UniDic-CWJ	476.82 ms ± 14.09 ms
SudachiDict-full	2002.83 ms ± 42.17 ms
CC-CEDICT	176.68 ms ± 30.45 ms

Limitations

• mòine does not reproduce the original paper's private search-query-log evaluation.
• Dictionary-backed comparison requires separately distributed dictionary artifacts.
• UniDic matching intentionally does not use MeCab/Viterbi costs.
• Chinese support is Mandarin pinyin only; it does not model Cantonese/Jyutping or non-Mandarin readings.
• processor, score_hint, NumPy dtype options, and worker parallelism are not part of the current cdist API.

Reference

[!CAUTION] This project is not the official implementation by the paper author.

@inproceedings{kaji-2023-lattice,
    title = "Lattice Path Edit Distance: A {R}omanization-aware Edit Distance for Extracting Misspelling-Correction Pairs from {J}apanese Search Query Logs",
    author = "Kaji, Nobuhiro",
    editor = "Wang, Mingxuan  and
      Zitouni, Imed",
    booktitle = "Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: Industry Track",
    month = dec,
    year = "2023",
    address = "Singapore",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2023.emnlp-industry.24/",
    doi = "10.18653/v1/2023.emnlp-industry.24",
    pages = "233--242",
}

License

mòine source code is licensed under either MIT or Apache-2.0. See LICENSE-MIT and LICENSE-APACHE.

Dictionary data is separate. UniDic-derived, SudachiDict-derived, and CC-CEDICT-derived artifacts carry their own license and attribution metadata, and should keep dictionary license information separate from the mòine source-code license. See THIRD_PARTY_NOTICES.md.