bunkai 1.1.1

Sentence boundary disambiguation tool for Japanese texts

Bunkai

Bunkai is a sentence boundary (SB) disambiguation tool for Japanese texts.

Quick Start

$ pip install -U bunkai
$ echo -e '宿を予約しました♪!まだ2ヶ月も先だけど。早すぎかな(笑)楽しみです★\n2文書目の先頭行です。▁改行はU+2581で表現します。' \
    | bunkai
宿を予約しました♪!│まだ2ヶ月も先だけど。│早すぎかな(笑)│楽しみです★
2文書目の先頭行です。▁│改行はU+2581で表現します。

Feed a document as one line by using ▁ (U+2581) for line breaks. The output shows sentence boundaries with │ (U+2502).

If you want to disambiguate sentence boundaries for line breaks, please add a --model option with the path to the model. First time, please setup a model.

$ bunkai --model bunkai-model-directory --setup

Then, please designate the directory.

$ echo -e "文の途中で改行を▁入れる文章ってありますよね▁それも対象です。" | bunkai --model bunkai-model-directory
文の途中で改行を▁入れる文章ってありますよね▁│それも対象です。

For more information, see examples or documents.

References

• Yuta Hayashibe and Kensuke Mitsuzawa. Sentence Boundary Detection on Line Breaks in Japanese. Proceedings of The 6th Workshop on Noisy User-generated Text (W-NUT 2020), pp.71-75. November 2020. [PDF] [bib]

License

Apache License 2.0