sengiri
Yet another sentence-level tokenizer for the Japanese text
DEPENDENCIES
MeCab
emoji
INSTALLATION
$ pip install sengiri
USAGE
import sengiri
print(sengiri.tokenize('うーん🤔🤔🤔どうしよう'))
#=>['うーん🤔🤔🤔', 'どうしよう']
print(sengiri.tokenize('モー娘。のコンサートに行った。'))
#=>['モー娘。のコンサートに行った。']
print(sengiri.tokenize('ありがとう＾＾ 助かります。'))
#=>['ありがとう＾＾', '助かります。']
print(sengiri.tokenize('顔文字テスト(*´ω｀*)うまくいくかな？'))
#=>['顔文字テスト(*´ω｀*)うまくいくかな？']
# I recommend using the NEologd dictionary.
print(sengiri.tokenize('顔文字テスト(*´ω｀*)うまくいくかな？', mecab_args='-d /usr/local/lib/mecab/dic/mecab-ipadic-neologd'))
#=>['顔文字テスト(*´ω｀*)', 'うまくいくかな？']
print(sengiri.tokenize('子供が大変なことになった。'
'（後で聞いたのだが、脅されたらしい）'
'（脅迫はやめてほしいと言っているのに）'))
#=>['子供が大変なことになった。', '（後で聞いたのだが、脅されたらしい）', '（脅迫はやめてほしいと言っているのに）']
print(sengiri.tokenize('楽しかったw また遊ぼwww'))
#=>['楽しかったw', 'また遊ぼwww']
print(sengiri.tokenize('http://www.inpaku.go.jp/'))
#=>['http://www.inpaku.go.jp/']CHANGES
0.2.3 (2025-11-28)
Enhanced URL handling stability using MeCab’s partial parsing and regex
Resolve problem about the dependency module (emoji)
Support Python 3.9-3.14
Drop supporting Python 3.4-3.8
0.2.2 (2019-10-15)
In tokenize() method, emoji_threshold parameter is available
Bugfix
0.2.1 (2019-10-12)
Works well with also a text including emoticon and www (Laughing expression)
Always treat emoji to delimiter regardless MeCab’s POS
0.1.1 (2019-10-05)
First release
1 maintainer
Metadata
1 maintainer
Release files for sengiri 0.2.3
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| sengiri-0.2.3.tar.gz | 6.3 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| sengiri-0.2.3-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 12.5 kB
Release files / sengiri-0.2.3.tar.gz
| Download URL | sengiri-0.2.3.tar.gz |
|---|---|
| Size | 6.3 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
5003b3cd1e237890502ae957e78642d5a310755cc8771b568c1795e030d25ff2
|
|
BLAKE2b-256 checksum How to use checksums |
c054ee8c0a26530f605b002ab043a9df25a8283dc226ec3da7fdc09d7c5ab0e3
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.1.0 CPython/3.10.16
|
Release files / sengiri-0.2.3-py3-none-any.whl
| Download URL | sengiri-0.2.3-py3-none-any.whl |
|---|---|
| Size | 6.2 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
e9db681da9bbaf82b60a42c4356a2ee70164a4665b8a6d68b5a688f9ab9dba7a
|
|
BLAKE2b-256 checksum How to use checksums |
876ed47f33f18c119f798680a2b0ab3ec37b045242f43c10c2560bdad7cf45c3
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.1.0 CPython/3.10.16
|
