Skip to main content

Haqumei 🌅

Haqumei is a Japanese Grapheme-to-Phoneme (G2P) library implemented in Rust.

English | 日本語

Crates.io docs.rs License: Apache-2.0

PyPI version Python version Push CI PyPI CI

Table of Contents

Features

Word-Phoneme Mapping APIs Provides mapping information between words ($\approx$ surface forms / dictionary entries) and phonemes, which was previously difficult to obtain directly. Enables retrieval of detailed analysis results with minimal loss of information from the input text, including unknown-word information. (See Advanced Features)
IPA Transcription Converts the full phoneme sequence to typed broad-IPA phones, grouped by word. g2ipa_prosody also retains pitch accent and prosodic boundaries (g2ipa, g2ipa_prosody).
Prosody Information Retrieval Provides phoneme sequences annotated with prosodic symbols, along with a word-to-phoneme mapping carrying structured prosody information (g2p_prosody, g2p_mapping_prosody). (For more details, see Prosody Features.)
More Detailed Phoneme Labels Through allophone resolution for moraic nasals (撥音) and geminate consonants (促音), you can choose from several options for the allophones introduced as dedicated phoneme labels. (See here for details.)
Performance Enables fast processing through a native Rust implementation. (See Benchmark)
Accuracy Successive dictionary and logic improvements reach 0.83% PER on jsut-label and 0.78% CER on ROHAN. Further changes build on the dictionary and the accuracy techniques of pyopenjtalk-plus. (See Accuracy)
Unknown Word Fallbacks Reading estimation for English words that would otherwise be unknown via haqumei-kanalizer, an on'yomi fallback for kanji that match no dictionary entry, and accent correction for words written entirely in katakana.
Concurrency Enables concurrent G2P processing across multiple threads using the *_batch methods.
Diverse Options Using HaqumeiOptions, you can flexibly customize allophone phoneme label introduction, Unicode normalization, and reading behavior.

Examples can be found in haqumei/examples.

Install

Rust

During the initial build of haqumei, the dictionary is downloaded and embedded into the binary due to the file size limits on crates.io. For custom dictionaries, or for environments where network access is unavailable during the build, please refer to here.

cargo add haqumei

Python

pip install haqumei

Supported Platforms

Pre-built wheels are available for the following platforms:

OS Architecture
Linux x86_64, aarch64
macOS aarch64 (e.g., Apple Silicon M1/M2/M3)
Windows x86_64

Pre-built wheels bundle the embedded dictionary and require no network access during installation.

If a wheel is unavailable for your platform, installation falls back to building from source, which requires a Rust toolchain. In that case, the dictionary is downloaded and embedded during the build (same as the Rust crate build process).

Command-Line Tool

We also provide haqumei-cli, a command-line interface for text processing from the terminal. For detailed usage, including pipeline processing and JSON output, please see haqumei-cli/README.md

cargo install haqumei-cli

Usage

Rust

use haqumei::Haqumei;

fn main() -> Result<(), Box<dyn std::error::Error>> {
  let mut haqumei = Haqumei::new()?;

  let text = "こんにちは、世界!";

  // Convert to phoneme list
  let phonemes = haqumei.g2p(text)?;
  assert_eq!(phonemes, ["k", "o", "N", "n", "i", "ch", "i", "w", "a", "pau", "s", "e", "k", "a", "i"]);

  // Get phoneme list with prosodic symbols
  let phones = haqumei.g2p_prosody(text)?.join(" ");
  assert_eq!(phones, "^ k o [ N n i ch i w a _ s e ] k a i ! $");

  // Convert to katakana reading
  let kana = haqumei.g2k(text)?;
  assert_eq!(kana, "コンニチワ、セカイ!");

  // Enable allophone resolution
  haqumei.options.use_allophones = true;

  let text = "執筆";

  // Get Word-Phoneme mapping with prosody information
  let mapping = haqumei.g2p_mapping_prosody(text)?;
  let shippitsu = &mapping[0];
  assert_eq!(shippitsu.word, "執筆");
  assert_eq!(shippitsu.pos, "名詞");
  assert_eq!(shippitsu.accent_nucleus, 0); // Heiban (flat) type

  println!("{:?}", shippitsu.phonemes);
  // Output:
  // [Phoneme {
  //     phoneme: Sh,
  //     pitch: Some(Low)
  // },
  // Phoneme {
  //     phoneme: I,
  //     pitch: Some(Low)
  // },
  // Phoneme {
  //     phoneme: ClP, // Allophone of the geminate consonant /cl/ (Phoneme::Cl): voiceless bilabial stop
  //     pitch: Some(High)
  // },
  // Phoneme {
  //     phoneme: P,
  //     pitch: Some(High)
  // },
  // Phoneme {
  //     phoneme: UnvoicedI,
  //     pitch: Some(High)
  // }, ...]

  Ok(())
}

Python

from haqumei import Haqumei

# Initialize Haqumei (the dictionary will be automatically set up)
haqumei = Haqumei()

text = "こんにちは、世界!"

# Convert to a phoneme list
phonemes = haqumei.g2p(text)
print(f"Phonemes: {phonemes}")
# -> Phonemes: ["k", "o", "N", "n", "i", "ch", "i", "w", "a", "pau", "s", "e", "k", "a", "i"]

# Get phoneme list with prosodic symbols
phones = " ".join(haqumei.g2p_prosody(text))
print(f"Prosody-annotated phonemes: {phones}")
# -> Prosody-annotated phonemes: ^ k o [ N n i ch i w a _ s e ] k a i ! $

# Convert to katakana reading
kana = haqumei.g2k(text)
print(f"Katakana reading: {kana}")
# -> Katakana reading: コンニチワ、セカイ!

Advanced Features

Word-Phoneme Mapping APIs

In Open JTalk (pyopenjtalk), unknown words are treated as pau (pauses), and Haqumei's standard g2p function follows this behavior.
However, by using G2P functions whose names contain mapping, detailed, or prosody, you can detect unknown words and spaces themselves as unk and sp respectively.

  • Known words: Regular phoneme sequence (punctuation marks become pau).
  • Unknown words: unk
  • Spaces, etc.: sp (Space)

Using g2p_mapping, you can obtain the phoneme-to-word mapping along with flags indicating whether a word is unknown (is_unknown) and whether it would normally be ignored in the original pipeline (is_ignored). In addition, using g2p_mapping_detailed allows you to retrieve not only the mapping but also part-of-speech information and accent details.

To obtain words and phonemes together with prosody information, g2p_mapping_prosody is useful. See here for details. That said, keep in mind that WordPhonemeProsody, the list type returned by g2p_mapping_prosody, is essentially a superset of WordPhonemeDetail (returned by g2p_mapping_detailed), aside from Mecab's features.

In short, the amount of information provided by these APIs can be roughly ordered as: g2p_mapping < g2p_mapping_detailed < g2p_mapping_prosody

use haqumei::Haqumei;

fn main() -> Result<(), Box<dyn std::error::Error>> {
  let mut haqumei = Haqumei::new()?;

  println!("{:?}", haqumei.g2p_mapping("𰻞𰻞麺 お冷を頼んだ")?);
  // [WordPhonemeMap {
  //     word: "𰻞𰻞",
  //     phonemes: ["unk"],
  //     is_unknown: true,
  //     is_ignored: false,
  //     char_span: 0..2,
  // },
  // WordPhonemeMap {
  //     word: "麺",
  //     phonemes: ["m", "e", "N"],
  //     is_unknown: false,
  //     is_ignored: false,
  //     char_span: 2..3,
  // },
  // WordPhonemeMap {
  //     word: "\u{3000}",
  //     phonemes: ["sp"],
  //     is_unknown: false,
  //     is_ignored: true,
  //     char_span: 3..4,
  // },
  // WordPhonemeMap {
  //     word: "お冷",
  //     phonemes: ["o", "h", "i", "y", "a"],
  //     is_unknown: false,
  //     is_ignored: false,
  //     char_span: 4..6,
  // }, ... ]

  println!("{:?}", haqumei.g2p_mapping_detailed("薄明")?);
  // [WordPhonemeDetail {
  //    word: "薄明",
  //    phonemes: ["h","a","k","u","m","e","e"],
  //    features: [
  //        "薄明",
  //        "名詞",
  //        "一般",
  //        "*",
  //        "*",
  //        "*",
  //        "*",
  //        "薄明",
  //        "ハクメイ",
  //        "ハクメー",
  //        "0/4",
  //        "C2",
  //    ],
  //    pos: "名詞",
  //    pos_group1: "一般",
  //    pos_group2: "*",
  //    pos_group3: "*",
  //    ctype: "*",
  //    cform: "*",
  //    orig: "薄明",
  //    read: "ハクメイ",
  //    pron: "ハクメー",
  //    accent_nucleus: 0,
  //    mora_count: 4,
  //    chain_rule: "C2",
  //    chain_flag: -1,
  //    is_unknown: false,
  //    is_ignored: false,
  //    char_span: 0..2,
  // }]

  Ok(())
}

IPA Transcription (g2ipa)

g2ipa converts Japanese text to broad IPA transcription. It returns a WordIpaMap for each word, containing its surface text, character span, and transcription. Use IpaToken::as_str() to get the string representation of each token.

use haqumei::{Haqumei, IpaToken};

fn main() -> Result<(), Box<dyn std::error::Error>> {
  let mut haqumei = Haqumei::new()?;
  for text in ["学校", "検査", "関与", "新票"] {
    let mapping = haqumei.g2ipa(text)?;
    let ipa: String = mapping.iter()
      .flat_map(|word| word.tokens.iter())
      .map(IpaToken::as_str)
      .collect();
    println!("{text}: {ipa}");
  }
  Ok(())
}
学校: ɡakːoː
検査: ke{N:s}sa
関与: ka{N:y}jo
新票: ɕi{N:hy}çoː

In "学校", kː and oː represent the geminate consonant and long vowel. For geminate affricates, the length mark applies to the closure: "ッズ" in "グッズ" is transcribed as dːz.

Labels such as {N:s} identify sounds whose realization cannot be assigned a single IPA symbol. {N:s} denotes a moraic nasal before /s/ without specifying the location or presence of oral closure. Braces distinguish these labels from IPA symbols; their definitions are in SpecialPhone. IpaToken represents IPA phones as Phone, dedicated labels as Special, and unknown input as Unknown. The string representation of unknown input is {unk}.

The output is a broad transcription derived from text using pronunciation rules. It does not describe fine differences in individual speakers' pronunciation. Allophone options such as use_allophones do not change the output.

g2ipa_prosody returns WordIpaProsody with pitch accent and prosodic boundaries. IpaTokenProsody also records pitch changes within a long vowel. Use g2ipa_batch and g2ipa_prosody_batch to convert multiple texts at once.

Python's Haqumei class provides the same four methods. Each token's type is available through token.kind, and its string representation through str(token) or token.symbol. The CLI outputs transcription with --mode ipa, or transcription with pitch and boundaries with --mode ipa-prosody.

Getting Reading Candidates (g2p_candidates)

g2p_mapping commits to a single reading, but g2p_candidates leaves the branch open when a downstream model should decide between them, as in forced alignment against audio.

Candidates are limited to readings the dictionary already holds. A surface form carrying more than one entry with a different pronunciation is a branch point, and each candidate re-runs the analysis with one reading picked per branch. Every candidate is built from a single fixed morpheme sequence, so its words have the same shape as what g2p_mapping returns.

The first entry of Candidates::candidates is exactly what g2p_mapping returns.

Segmentation differences are candidates too: 彼の splits into 彼 + の (カレノ) or into the single 連体詞 彼の (アノ).

use haqumei::Haqumei;

fn main() -> Result<(), Box<dyn std::error::Error>> {
  let mut haqumei = Haqumei::new()?;
  let got = haqumei.g2p_candidates("彼の話を聞いた。")?;

  for branch in &got.branches {
    println!(
      "{:?} {} {:?}",
      branch.char_span, branch.surface,
      branch.alternatives.iter().map(|a| (a.pron(), a.nodes.len())).collect::<Vec<_>>()
    );
  }
  // 0..2 彼の [("カレノ", 2), ("アノ", 1)]

  for cand in &got.candidates {
    println!(
      "{} {:?}",
      cand.delta,
      cand.words.iter().flat_map(|w| w.phonemes.iter()).collect::<Vec<_>>()
    );
  }
  // 0    [k, a, r, e, n, o, h, a, n, a, sh, i, o, k, i, i, t, a, pau]
  // 1529 [a, n, o, h, a, n, a, sh, i, o, k, i, i, t, a, pau]

  Ok(())
}

The words of a candidate are WordPhonemeMap for g2p_candidates, WordPhonemeDetail for g2p_candidates_detailed, and WordPhonemeProsody for g2p_candidates_prosody. CandidateOptions controls the cost cutoff and the number of candidates.

Candidates::branches lists the branch points themselves and is not subject to max_candidates. Build the product yourself from branches when the cap would leave out variants you need, and join it to the candidates on char_span.

Four things keep a reading out of the candidate list.

  • A word with a single dictionary entry: it yields one reading, however many it could have in principle
  • Unknown-word nodes: they carry no pronunciation of their own and are left out by default (branch_on_unknown_words)
  • Places where a correction decides the reading: the 何 predictor, the context-reading rules and njd_set_digit overwrite whatever the lattice chose
  • Candidates that end up with the same phoneme sequence: only the one with the smallest cost difference is kept

0 and 何 branch in the lattice but rarely yield a second candidate.

Combinations are assembled from the smallest sum upward, max_candidates of them, so Candidates::candidates comes out sorted by Candidate::delta. MeCab's costs are there to decide segmentation and part of speech rather than to measure how likely a reading is, so the value cannot serve as an arc weight.

Modifying Output with G2P Options

You can customize the behavior of Haqumei by using Haqumei::with_options. For details on the default behavior and available options, please refer to HaqumeiOptions.

In the following example, normalize_unicode (which is disabled by default) is enabled to apply Unicode NFC normalization to the input text.

use haqumei::{Haqumei, HaqumeiOptions, UnicodeNormalization};

fn main() -> Result<(), Box<dyn std::error::Error>> {
  let mut haqumei = Haqumei::with_options(HaqumeiOptions {
    normalize_unicode: UnicodeNormalization::Nfc,
    ..Default::default()
  })?;

  let text = &[
    "\u{304B}\u{3099}", // か + ゙ (が)
    "\u{306F}\u{309A}", // は + ゚ (ぱ)
    "\u{30B3}\u{3099}", // コ + ゙ (ゴ)
  ];

  println!("{:?}", haqumei.g2p_detailed_batch(text)?);
  // Output: [["g", "a"], ["p", "a"], ["g", "o"]]

  Ok(())
}

Prosody Features (g2p_prosody / g2p_mapping_prosody)

Specification of g2p_prosody_with_options

Converts the input text into a phoneme list annotated with prosodic symbols based on the ProsodyFormat setting. (The g2p_prosody method behaves identically to specifying ProsodyFormat::Default.)

The output commonly includes the following prosodic symbols:

Symbol Meaning Position
^ Beginning of utterance (BOS) Sentence-initial
$ End of utterance (EOS) Sentence-final
? End of interrogative (?) Sentence-medial
! End of exclamation (Custom extension) Sentence-medial
_ Pause / Comma (、) Sentence-medial
# Accent phrase boundary Sentence-medial
{...} Unknown word Sentence-medial

For more information on Japanese accents, please refer to the tdmelodic User Manual / Preliminary Knowledge (Japanese).

ProsodyFormat::Default

In addition to the above, the output includes the following prosodic symbols:

Symbol Meaning Position
[ Pitch rise (Phrase head) Near the beginning of a phrase
] Pitch fall (Accent nucleus) Right after the nuclear mora

The symbols [ and ] are based on the accent notation commonly used in tdmelodic and similar tools. They correspond to ^ and ! in the algorithm described by Kurihara et al. (2021) in "Prosodic Features Control by Symbols as Input of Sequence-to-Sequence Acoustic Modeling for Neural TTS".

ProsodyFormat::Prefix

Instead of using pitch rise/fall symbols ([ and ]), pitch high/low is attached as a prefix to each phoneme:

  • H_ : High pitch
  • L_ : Low pitch

The pitch is explicitly indicated for each phoneme.

Example: "青い空" -> ["^", "L_a", "H_o", "L_i", "#", "H_s", "H_o", "L_r", "L_a", "$"]

ProsodyFormat::Numeric

Pitch high/low is attached as a suffix to each phoneme as a numeric value:

  • :1 : High pitch
  • :0 : Low pitch

Example: "青い空" -> ["^", "a:0", "o:1", "i:0", "#", "s:1", "o:1", "r:0", "a:0", "$"]

Example

use haqumei::Haqumei;

fn main() -> Result<(), Box<dyn std::error::Error>> {
  let mut haqumei = Haqumei::new()?;

  let phones = haqumei.g2p_prosody("こんにちは、世界!")?;
  assert_eq!(phones.join(" "), "^ k o [ N n i ch i w a _ s e ] k a i ! $");

  let phones = haqumei.g2p_prosody("青い空、広がる。")?;
  assert_eq!(phones.join(" "), "^ a [ o ] i # s o ] r a _ h i [ r o g a r u _ $");

  Ok(())
}

Specification of g2p_mapping_prosody

On the other hand, g2p_mapping_prosody analyzes the input text and retrieves an alignment between detailed linguistic information for each morpheme (word) and phonemes with prosodic symbols.

While [Haqumei::g2p_prosody] and [Haqumei::g2p_prosody_with_options] return a flat list of strings (Vec<String>), this function returns structured data (Vec<WordPhonemeProsody>) annotated with part-of-speech, accent type, reading, and pitch information.

This is suitable for speech synthesis frontend processing when you want to maintain the correspondence between morphemes and phonemes, individually retrieve and manipulate pitch high/low ([PitchAccent]), or handle unknown words.

Information included in WordPhonemeProsody

The following information is included as data for each morpheme:

Field Description Example
word Word, a substring of the input text "空"
phonemes List consisting of phonemes, pitch information, and prosodic symbols (see below) [ProsodicPhoneme::Exclamatory]
pos, pos_group1~3 Part-of-speech and its subdivisions "名詞", "一般"
orig, read, pron Original form, reading, pronunciation form "空", "ソラ", "ソラ"
accent_nucleus Accent nucleus position (0: Heiban type, 1~: n-th mora) 1
mora_count Number of moras 2
is_unknown Whether it was judged as an unknown word by MeCab false
is_ignored Whether no phoneme was assigned false

Prosodic Phoneme (ProsodicPhoneme)

The phonemes field contains a list of the following elements:

Variant Meaning Output symbol in g2p_prosody, etc.
Phoneme Phoneme and its pitch (High / Low) a, a:0, H_a, etc.
AccentPhraseBoundary Accent phrase boundary #
Pause Regular pause / comma _
Interrogative End of interrogative / Pause ?
Exclamatory End of exclamation / Pause !

Example

use haqumei::{Haqumei, PitchAccent, ProsodicPhoneme};

fn main() -> Result<(), Box<dyn std::error::Error>> {
  let mut haqumei = Haqumei::new()?;

  // Retrieve text as structured data per morpheme
  let mapping = haqumei.g2p_mapping_prosody("青い空が、好きだ!")?;

  // Morpheme information for "青い"
  let aoi = &mapping[0];
  assert_eq!(aoi.word, "青い");
  assert_eq!(aoi.pos, "形容詞");
  assert_eq!(aoi.read, "アオイ");
  assert_eq!(aoi.accent_nucleus, 2); // 中高型

  // Phoneme and pitch information for "青い" (a: Low, o: High, i: Low)
  assert!(matches!(
      aoi.phonemes[0],
      ProsodicPhoneme::Phoneme { pitch: Some(PitchAccent::Low), .. }
  ));

  let da = mapping.last().unwrap();
  assert_eq!(da.word, "!");
  assert!(da.phonemes.contains(&ProsodicPhoneme::Exclamatory));

  Ok(())
}

Accuracy

Measured with japanese-g2p-benchmark.

The figures are the phoneme error rate (PER) on prj-beatrice/jsut-label, a fork of jsut-label that annotates the basic5000 subset of the JSUT corpus, and the katakana error rate (KER) on ROHAN.

G2P jsut-label (PER) ROHAN (KER)
pyopenjtalk 0.4.1 1.31% 5.02% *
pyopenjtalk-plus 0.4.1.post9 1.09% 1.60%
haqumei 0.12.0 0.83% 0.78%

* vanilla pyopenjtalk has no way to write long vowels and yotsugana in their original spelling, so its output cannot be brought to ROHAN's notation. Most of that gap is notation rather than misreading.

For each G2P, the options are chosen from the ones it offers to match the annotation convention of the corpus.
Every combination is listed under "All option combinations" below.

All option combinations
G2P options jsut-label (PER) ROHAN (KER)
pyopenjtalk - 1.31% 5.02%
pyopenjtalk_plus use_sudachi_kanji_yomi=True, use_tsqyomi=True, revert_long_vowels=True, revert_yotsugana=True - 1.60%
pyopenjtalk_plus use_sudachi_kanji_yomi=True, use_tsqyomi=True, revert_long_vowels=False, revert_yotsugana=False 1.10% 4.63%
pyopenjtalk_plus use_sudachi_kanji_yomi=False, use_tsqyomi=True, revert_long_vowels=True, revert_yotsugana=True - 1.60%
pyopenjtalk_plus use_sudachi_kanji_yomi=False, use_tsqyomi=True, revert_long_vowels=False, revert_yotsugana=False 1.10% 4.63%
pyopenjtalk_plus use_sudachi_kanji_yomi=True, use_tsqyomi=False, revert_long_vowels=True, revert_yotsugana=True - 1.62%
pyopenjtalk_plus use_sudachi_kanji_yomi=True, use_tsqyomi=False, revert_long_vowels=False, revert_yotsugana=False 1.09% 4.65%
pyopenjtalk_plus use_sudachi_kanji_yomi=False, use_tsqyomi=False, revert_long_vowels=True, revert_yotsugana=True - 1.64%
pyopenjtalk_plus use_sudachi_kanji_yomi=False, use_tsqyomi=False, revert_long_vowels=False, revert_yotsugana=False 1.11% 4.66%
haqumei normalize_iu=none, revert_long_vowels=True, revert_yotsugana=True - 0.78%
haqumei normalize_iu=none, revert_long_vowels=False, revert_yotsugana=False 0.96% 3.79%
haqumei normalize_iu=yuu, revert_long_vowels=True, revert_yotsugana=True - 0.83%
haqumei normalize_iu=yuu, revert_long_vowels=False, revert_yotsugana=False 0.92% 3.84%
haqumei normalize_iu=yuu-base, revert_long_vowels=True, revert_yotsugana=True - 0.80%
haqumei normalize_iu=yuu-base, revert_long_vowels=False, revert_yotsugana=False 0.83% 3.81%

- means it was not measured. The options that restore the original spelling (revert_long_vowels / revert_yotsugana) only mean something for ROHAN, which avoids the long vowel mark, so jsut-label is not run on the rows that enable them.

Reproducing

git clone https://github.com/o24s/japanese-g2p-benchmark
cd japanese-g2p-benchmark
uv run init.py
uv run python run_all.py --datasets phoneme,no_lvs --sources jsut-label,rohan4600

jsut-label

Phoneme Error Rate (S+D+I / N_expected): 0.83% (Substitute=1459, Delete=434, Insert=580, N=297843)

HaqumeiOptions:

HaqumeiOptions {
  normalize_iu: Some(IuPronunciation::YuuBase),
  ..Default::default()
}

ROHAN

Katakana Error Rate (S+D+I / N_expected): 0.78% (Substitute=757, Delete=164, Insert=250, N=150637)

HaqumeiOptions:

HaqumeiOptions {
  revert_long_vowels: true,
  revert_yotsugana: true,
  ..Default::default()
}

Benchmark

The following benchmark compares pyopenjtalk (Baseline) with haqumei, using approximately 318,000 characters of Japanese text. Initialization time is excluded.

Input data: I Am a Cat (吾輩は猫である) 318,407 chars / 8,451 lines (Average 37 chars/line) (Ruby characters have been removed)

Execution Mode Execution Time (Mean) Throughput Speedup
pyopenjtalk (Baseline) 2.358 s 135k chars/s 1.00x
haqumei (Default) 0.680 s 468k chars/s 3.47x
haqumei (g2p_batch, Default) 0.048 s 6.62M chars/s 49.00x

The detailed benchmark code can be found in haqumei-bench/pyopenjtalk.

Additionally, Rust-layer benchmarks for Haqumei using Criterion.rs can be run via cargo bench in the haqumei-bench crate. The comparison benchmark with pyopenjtalk-plus is located in haqumei-bench/pyopenjtalk-plus.

Performance Notes

  • Throughput Variation by Input Structure:
    Especially in the *_batch APIs, throughput (chars/s) tends to increase as the number of characters per line grows (up to approximately 4KB), compared with pyopenjtalk. When processing large volumes of text, it is most efficient to pass content in substantial chunks rather than splitting it into excessively short lines.
  • "Default" in the table:
    The configuration using Haqumei::new as is.

Building with a Custom Embedded Dictionary

By default, haqumei downloads the dictionary at build time and embeds it into the binary. This allows the crate to be published to crates.io while still producing a self-contained binary.

If you want to build with your own dictionary embedded in the binary, you can change the configuration as follows.

Change the Cargo Features

Disable the default download-dictionary feature and enable build-dictionary.

[dependencies]
haqumei = { version = "x.y.z", features = ["embed-dictionary", "build-dictionary"], default-features = false }

Prepare the Dictionary Source and Set the Environment Variable

Prepare a dictionary source directory containing .csv and .def files to be compiled at build time, then set its path to the HAQUMEI_DICT_SRC environment variable before running the build.

On Unix-like systems:

HAQUMEI_DICT_SRC="/path/to/your/dictionary" cargo build --release

On Windows (PowerShell):

& { $env:HAQUMEI_DICT_SRC="C:\path\to\your\dictionary"; cargo build --release }

Note: If the environment variable is not set, the build script falls back to dictionary, relative to the crate root.

Setting HAQUMEI_DICT_ARCHIVE to a prebuilt .tar.zst dictionary skips dictionary downloading and compilation. The archive must contain system.bin, char.bin, and matrix.bin at its root. Haqumei's distributed archive also contains COPYING with the dictionary license notices.

system.bin is a vibrato-rkyv dictionary converted for Haqumei to preserve MeCab-compatible analysis results. It is neither a renamed sys.dic nor interchangeable with a standard Vibrato dictionary. Convert a UTF-8 MeCab-compatible dictionary directory containing sys.dic, unk.dic, char.bin, and matrix.bin with cargo run -p haqumei-dict-tool -- --convert-mecab /path/to/compiled. The command writes system.bin into the same directory.

At runtime, the system lexicon is memory-mapped from system.bin. Adding or changing MeCab-compatible user dictionaries builds only the user lexicon in memory.

Dictionary

Haqumei uses a modified form of the dictionary included in pyopenjtalk-plus.

License

Haqumei, excluding haqumei-jpreprocess*, haqumei-jlabel, and haqumei-kanalizer, is distributed under the terms of the Apache License 2.0.

Haqumei's logic includes an implementation ported from tsukumijima/pyopenjtalk-plus, and it likewise bundles the file stating the license of r9y9/pyopenjtalk that is placed in tsukumijima/pyopenjtalk-plus. Bundling it does not state the license of the code newly added on top of tsukumijima/pyopenjtalk-plus's upstream.

Licenses and Origins of Bundled Software

Haqumei includes Rust ports and dictionary data derived from the following projects.

  • Adapted haqumei-jpreprocess* crates

    • Origin: jpreprocess/jpreprocess, revision 54cf9bc2d40a5d6f25144333e9cd03fd3258a126. The NJD and JPCommon implementations include compatibility changes for tsukumijima/open_jtalk.
    • License: BSD-3-Clause. Each crate includes LICENSE and NOTICE, preserving the jpreprocess and Open JTalk notices.
  • Dictionary Reading and Compilation

    • Origin: The Rust dictionary reader and compiler are based on the MeCab and Darts code included in Open JTalk.
    • License: BSD-3-Clause. Copyright notices and license terms are provided in NOTICE and LICENSE-MeCab.
  • Bundled Dictionary Data

    • Origin: The dictionary data contained in the haqumei/dictionary directory is based on tsukumijima/pyopenjtalk-plus, a modified fork of r9y9/pyopenjtalk.
    • License: The dictionary data is covered by the license notices in haqumei/dictionary/COPYING.
  • Bundled Kanji Reading Fallback Data

    • Origin: The data in haqumei/data/unihan is generated from the kJapanese field of the Unihan Database (Unicode 18.0.0). It provides a per-character reading fallback, used to keep kanji missing from the dictionary from appearing verbatim in the kana output (see HaqumeiOptions::read_unknown_kanji).
    • License: UNICODE LICENSE V3. This license applies only to the data located in haqumei/data/unihan, and does not apply to the rest of this project. In accordance with redistribution requirements, the full text is included in haqumei/data/unihan/LICENSE.
  • Bundled Kanji Variant Data

    • Origin: haqumei/data/cjkvi is generated from the Japanese character-form tables in the CJKVI Variants Database. The pinned revision and extraction rules are recorded in its README.
    • License: MIT. The copyright notice and full license text are included in haqumei/data/cjkvi/LICENSE.
  • Bundled model for predicting the reading of 「何」

    • Origin: the ONNX models in haqumei/yomi_model are a conversion, by tsukumijima/pyopenjtalk-plus, of the 「何」 reading-prediction logic implemented in n5-suzuki/pyopenjtalk. They are embedded into the binary with include_bytes!.
    • License: the root of n5-suzuki/pyopenjtalk carries the MIT license notice of r9y9/pyopenjtalk, the repository it was forked from. Meanwhile, no license statement covering the logic newly added on top of pyopenjtalk, or the converted models, could be found. The LICENSE-pyopenjtalk bundled with haqumei does not state the license of this model.
  • Bundled haqumei-jlabel Source Code

    • Origin: The code contained in the haqumei-jlabel directory is based on the jpreprocess/jlabel repository.
    • License: The bundled haqumei-jlabel source code is licensed under the BSD 3-Clause License. This license applies only to the code located in haqumei-jlabel, and does not apply to the rest of this project. In accordance with redistribution requirements, the full text of the BSD 3-Clause License is included in haqumei-jlabel/LICENSE.
  • Bundled haqumei-kanalizer Crate

Acknowledgements

The fundamental design and API of haqumei are inspired by pyopenjtalk and its highly improved fork, pyopenjtalk-plus. In addition, some implementations are based on jlabel and kanalizer to improve usability and accuracy.

  • pyopenjtalk: Copyright (c) 2018 Ryuichi Yamamoto
  • pyopenjtalk-plus: Copyright (c) 2023 tsukumijima
  • jpreprocess: Copyright (c) 2024 JPreprocess Team
  • jlabel: Copyright (c) 2024 JPreprocess Team
  • kanalizer: Copyright (c) 2025 VOICEVOX

We are deeply grateful to the authors and contributors of these foundational projects.

Metadata

Release files for haqumei 0.12.2

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for haqumei 0.12.2
File Size Uploaded
haqumei-0.12.2.tar.gz 5.9 MB Details

Built distributions (wheels)

Table of built distributions (wheels) for haqumei 0.12.2
File
haqumei-0.12.2-cp39-abi3-win_amd64.whl CPython 3.9 abi3 Windows x86-64 Details
haqumei-0.12.2-cp39-abi3-manylinux_2_28_x86_64.whl CPython 3.9 abi3 Linux glibc 2.28+ x86-64 Details
haqumei-0.12.2-cp39-abi3-manylinux_2_28_aarch64.whl CPython 3.9 abi3 Linux glibc 2.28+ ARM64 Details
haqumei-0.12.2-cp39-abi3-macosx_11_0_arm64.whl CPython 3.9 abi3 macOS 11.0+ ARM64 Details

Total release size: 138.4 MB

Release files / haqumei-0.12.2.tar.gz

Download URL haqumei-0.12.2.tar.gz
Size 5.9 MB
Tags Source
SHA-256 checksum
How to use checksums
02a583da1d34941749001e044289775e06ef81751631cf04a8cc5a4e62ced117
BLAKE2b-256 checksum
How to use checksums
e3f5b78a352b09c07ea188b27ddb0ec4fbfad330240df9a799d95fa4d19d4cbd
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.14

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Oct 11, 2026.

Transparency log

Release files / haqumei-0.12.2-cp39-abi3-win_amd64.whl

Download URL haqumei-0.12.2-cp39-abi3-win_amd64.whl
Size 31.1 MB
Tags CPython 3.9 Windows x86-64 abi3
SHA-256 checksum
How to use checksums
92b232b6cfc935aacf2dfeb55c7f098a956d6c3d06e7453ee53fb36b4f01284e
BLAKE2b-256 checksum
How to use checksums
22382f1116921335db71ea0f23e18f9ba7df5fbff6b2216bbfaf20177dc58be6
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.14

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Oct 11, 2026.

Transparency log

Release files / haqumei-0.12.2-cp39-abi3-manylinux_2_28_x86_64.whl

Download URL haqumei-0.12.2-cp39-abi3-manylinux_2_28_x86_64.whl
Size 33.8 MB
Tags CPython 3.9 Linux glibc 2.28+ x86-64 abi3
SHA-256 checksum
How to use checksums
8b75ee78002a413e4e32db980fcbba86190194f352705c5dcb042a38150116d7
BLAKE2b-256 checksum
How to use checksums
23221778764dc33638f099b275102559d713bc786bbb9469e14a1b974854b507
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.14

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Oct 11, 2026.

Transparency log

Release files / haqumei-0.12.2-cp39-abi3-manylinux_2_28_aarch64.whl

Download URL haqumei-0.12.2-cp39-abi3-manylinux_2_28_aarch64.whl
Size 35.1 MB
Tags CPython 3.9 Linux glibc 2.28+ ARM64 abi3
SHA-256 checksum
How to use checksums
b69df9d078dd323accda1513428d42e3fbf3fc76fc4823f0b207c360b53a1d17
BLAKE2b-256 checksum
How to use checksums
a84e86bc6eea2ff8058f39faae9d3b9ff0b47ed1e4f58f6c18b3180bdc0ed44d
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.14

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Oct 11, 2026.

Transparency log

Release files / haqumei-0.12.2-cp39-abi3-macosx_11_0_arm64.whl

Download URL haqumei-0.12.2-cp39-abi3-macosx_11_0_arm64.whl
Size 32.5 MB
Tags CPython 3.9 abi3 macOS 11.0+ ARM64
SHA-256 checksum
How to use checksums
149d5b41b35c361f47fb3b931ee0aef41fb6483239e19cf7f94f37bcde79f974
BLAKE2b-256 checksum
How to use checksums
9ed4fc5800ffeb962cf8f96788c0f3edcc54d0f348a29914c855136db3149b9c
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.14

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Oct 11, 2026.

Transparency log

Release history Release notifications | RSS feed

This release

0.12.2 This release

5 release files

0.12.1

5 release files

0.10.0

5 release files

0.9.2

5 release files

0.9.1

5 release files

0.9.0

5 release files

0.8.5

5 release files

0.8.4

5 release files

0.8.2

5 release files

0.8.1

5 release files

0.8.0

5 release files

0.7.2

5 release files

0.7.1

5 release files

0.7.0

5 release files

0.6.11

5 release files

0.6.9

5 release files

0.6.8

5 release files

0.6.7

5 release files

0.6.6

5 release files

0.6.5

5 release files

0.6.4

5 release files

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page