Bunkai
Bunkai is a sentence boundary (SB) disambiguation tool for Japanese texts.
Quick Start
$ pip install bunkai
$ echo -e '宿を予約しました♪!まだ2ヶ月も先だけど。早すぎかな(笑)楽しみです★\n2文書目の先頭行です。▁改行はU+2581で表現します。' \
| bunkai
│宿を予約しました♪!│まだ2ヶ月も先だけど。│早すぎかな(笑)│楽しみです★
│2文書目の先頭行です。│▁改行はU+2581で表現します。
Feed a document as one line by using ▁ (U+2581) for line breaks.
The output shows sentence boundaries with │ (U+2502).
If you want to disambiguate sentence boundaries for line breaks, please add a --model option with the path to the model.
Trained models are available here.
$ echo -e "文の途中で改行を▁入れる文章ってありますよね▁それも対象です。" | bunkai --model /path/to/model
文の途中で改行を▁入れる文章ってありますよね▁│それも対象です。
For more information, see examples or documents.
References
- Yuta Hayashibe and Kensuke Mitsuzawa. Sentence Boundary Detection on Line Breaks in Japanese. Proceedings of The 6th Workshop on Noisy User-generated Text (W-NUT 2020), pp.71-75. November 2020. [PDF] [bib]
License
Apache License 2.0
1 maintainer
Metadata
1 maintainer
Release files for bunkai 1.0.1
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| bunkai-1.0.1.tar.gz | 40.8 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| bunkai-1.0.1-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 99.2 kB
Release files / bunkai-1.0.1.tar.gz
| Download URL | bunkai-1.0.1.tar.gz |
|---|---|
| Size | 40.8 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
1d4c344a5516b423f949ed9eeb9cafbe061c125b86aa8dddf2eb8c3d9f8e9f2f
|
|
BLAKE2b-256 checksum How to use checksums |
599f2e420f557df24984c1d7a35362cafa5dc89bb3313a67f3d91637a0f1a49f
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
poetry/1.1.5 CPython/3.9.2 Linux/4.19.0-16-cloud-amd64
|
Release files / bunkai-1.0.1-py3-none-any.whl
| Download URL | bunkai-1.0.1-py3-none-any.whl |
|---|---|
| Size | 58.3 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
f610ea61d7150ffe3faade1648bc6d526b3b202e1f5e659702fae7bc46eb724b
|
|
BLAKE2b-256 checksum How to use checksums |
3f0ffc173ebe8ed297cd3f9883d07f4b63faaa7a874762528f4eba2e7563ae64
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
poetry/1.1.5 CPython/3.9.2 Linux/4.19.0-16-cloud-amd64
|
