neologdn
neologdn is a Japanese text normalizer for mecab-neologd.
The normalization is based on the neologd’s rules: https://github.com/neologd/mecab-ipadic-neologd/wiki/Regexp.ja
Contributions are welcome!
NOTE: Installing this module requires C++11 compiler.
Installation
$ pip install neologdn
Usage
import neologdn
neologdn.normalize("ﾊﾝｶｸｶﾅ")
# => 'ハンカクカナ'
neologdn.normalize("全角記号！？＠＃")
# => '全角記号!?@#'
neologdn.normalize("全角記号例外「・」")
# => '全角記号例外「・」'
neologdn.normalize("長音短縮ウェーーーーイ")
# => '長音短縮ウェーイ'
neologdn.normalize("チルダ削除ウェ~∼∾〜〰～イ")
# => 'チルダ削除ウェイ'
neologdn.normalize("いろんなハイフン˗֊‐‑‒–⁃⁻₋−")
# => 'いろんなハイフン-'
neologdn.normalize(" ＰＲＭＬ 副 読 本 ")
# => 'PRML副読本'
neologdn.normalize(" Natural Language Processing ")
# => 'Natural Language Processing'Benchmark
# Sample code from
# https://github.com/neologd/mecab-ipadic-neologd/wiki/Regexp.ja#python-written-by-hideaki-t--overlast
import normalize_neologd
%timeit normalize(normalize_neologd.normalize_neologd)
# => 1000 loops, best of 3: 1.18 ms per loop
import neologdn
%timeit normalize(neologdn.normalize)
# => 10000 loops, best of 3: 140 µs per loopneologdn is about x10 faster than sample code.
details are described as the below notebook: https://github.com/ikegami-yukino/neologdn/blob/master/benchmark/benchmark.ipynb
License
Apache Software License.
CHANGES
0.1.2 (2016-03-29)
Fix installation bug
0.1.1.1 (2016-03-19)
Support Windows
Explicitly specify to -std=c++11 in build (This bug is reported by id774. Many thanks)
0.1.1 (2015-10-10)
Initial release.
1 maintainer
Release files for neologdn 0.1.2
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| neologdn-0.1.2.tar.gz | 42.8 kB | Details |
Release files / neologdn-0.1.2.tar.gz
| Download URL | neologdn-0.1.2.tar.gz |
|---|---|
| Size | 42.8 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
071e4f8cdc3f3466d58f8285b241d0d310278b608d654be66b2089e381ddf27f
|
|
BLAKE2b-256 checksum How to use checksums |
8f97fed71dc2b82a21288f9771c96055a737a8023f8e1ff948a8bb9b8ddead75
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
