Bunkai
Bunkai is a sentence boundary (SB) disambiguation tool for Japanese texts.
Bunkaiは日本語文境界判定器です．
Quick Start
Install
$ pip install -U bunkai
Disambiguation without Models
$ echo -e '宿を予約しました♪!まだ2ヶ月も先だけど。早すぎかな(笑)楽しみです★\n2文書目の先頭行です。▁改行はU+2581で表現します。' \
| bunkai
宿を予約しました♪!│まだ2ヶ月も先だけど。│早すぎかな(笑)│楽しみです★
2文書目の先頭行です。▁│改行はU+2581で表現します。
- Feed a document as one line by using
▁(U+2581) for line breaks.
1行は1つの文書を表します．文書中の改行は▁(U+2581) で与えてください． - The output shows sentence boundaries with
│(U+2502).
出力では文境界は│(U+2502) で表示されます．
Disambiguation for Line Breaks with a Model
If you want to disambiguate sentence boundaries for line breaks, please add a --model option with the path to the model.
改行記号に対しても文境界判定を行いたい場合は，--modelオプションを与える必要があります．
First time, please setup a model. It will take some time.
はじめにモデルをセットアップする必要があります．セットアップには少々時間がかかります．
$ bunkai --model bunkai-model-directory --setup
Then, please designate the directory.
そしてモデルを指定して動かしてください．
$ echo -e "文の途中で改行を▁入れる文章ってありますよね▁それも対象です。" | bunkai --model bunkai-model-directory
文の途中で改行を▁入れる文章ってありますよね▁│それも対象です。
Python Library
You can also use Bunkai as Python library.
BunkaiはPythonライブラリとしても使えます．
from bunkai import Bunkai
bunkai = Bunkai()
for sentence in bunkai("はい。このようにpythonライブラリとしても使えます！"):
print(sentence)
For more information, see examples.
ほかの例はexamplesをご覧ください．
Documents
References
- Yuta Hayashibe and Kensuke Mitsuzawa. Sentence Boundary Detection on Line Breaks in Japanese. Proceedings of The 6th Workshop on Noisy User-generated Text (W-NUT 2020), pp.71-75. November 2020. [PDF] [bib]
License
Apache License 2.0
1 maintainer
Metadata
1 maintainer
Release files for bunkai 1.2.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| bunkai-1.2.0.tar.gz | 43.3 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| bunkai-1.2.0-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 104.1 kB
Release files / bunkai-1.2.0.tar.gz
| Download URL | bunkai-1.2.0.tar.gz |
|---|---|
| Size | 43.3 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
88c73ea4b17aed594bbdca61d3c32bb77808cc2792f2e2d83f8cb363a07960eb
|
|
BLAKE2b-256 checksum How to use checksums |
c949e83092c1674b36e9fc4acfff86f4223b9fae76771ea8aa8847b6835cbae3
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
poetry/1.1.6 CPython/3.9.5 Linux/4.19.0-16-cloud-amd64
|
Release files / bunkai-1.2.0-py3-none-any.whl
| Download URL | bunkai-1.2.0-py3-none-any.whl |
|---|---|
| Size | 60.9 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
7b491fcadd21d300f1b65b3875306a2fde96c44d9c30df403a0754d933540a48
|
|
BLAKE2b-256 checksum How to use checksums |
e10df0f674a11ed3f3aa8b67fc7d2d271f165b85b856f456ca597dccc3eb7392
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
poetry/1.1.6 CPython/3.9.5 Linux/4.19.0-16-cloud-amd64
|
