Skip to main content

Japanese kanji disambiguation

Project description


A robot reading a book

Yomikata uses context to resolve ambiguous words in Japanese. Check out the interactive demo!

Yomikata supports 130 ambiguous forms and reaches a global accuracy of 94%. See the demo page for detailed performance information.

Yomikata follows the approach of Sato et al. 2022 by fine-tuning the Tohoku group's Japanese BERT transformer to classify words into different readings based on the sentence context. A similar approach was used in English by Nicolis et al. 2021.

Yomikata recognizes ~50% more heteronyms than Sato et al. by adding support for words which are not in the original BERT vocabulary, and it expands the original Aozora Bunko and NDL titles training data to include the core BCCWJ corpus and the KWDLC corpus.


from yomikata.dbert import dBert
reader = dBert()
# => そして、畳の{表/おもて}は、すでに幾年前に換えられたのか分らなかった。

This example sentence, from the short story When I Was looking for a Room to Let (1923) by Mimei Ogawa, contains the very common heteronym 表, which admits the readings omote (surface) and hyō (table). Yomikata's dBert (disambiguation BERT) correctly determines that in this sentence it refers to the surface of a tatami mat and should be read omote.

The furigana function outputs the sentence with the heteronym annotated. Readings for the other words can be obtained with a simple dictionary lookup.

from yomikata.dictionary import Dictionary
dictreader = Dictionary() # defaults to unidic.
# => そして、{畳/たたみ}の{表/おもて}は、すでに{幾年/いくねん}{前/まえ}に{換/か}えられたのか{分/わ}らなかった。

Without Yomikata, the dictionary outputs the wrong reading for the heteronym.


pip install yomikata
python -m yomikata download

The second command is necessary to download the model weights, which are too large to host PyPI.

Inference should work fine on CPU.

For details on data processing and training, see the main notebook.

Project details

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

yomikata-0.0.4.tar.gz (25.0 kB view hashes)

Uploaded source

Built Distribution

yomikata-0.0.4-py3-none-any.whl (25.1 kB view hashes)

Uploaded py3

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page