neologdn

pyversion

neologdn is a Japanese text normalizer for mecab-neologd.

The normalization is based on the neologd’s rules: https://github.com/neologd/mecab-ipadic-neologd/wiki/Regexp.ja

Contributions are welcome!

NOTE: Installing this module requires C++11 compiler.

Installation

$ pip install neologdn

Usage

import neologdn
neologdn.normalize("ﾊﾝｶｸｶﾅ")
# => 'ハンカクカナ'
neologdn.normalize("全角記号！？＠＃")
# => '全角記号!?@#'
neologdn.normalize("全角記号例外「・」")
# => '全角記号例外「・」'
neologdn.normalize("長音短縮ウェーーーーイ")
# => '長音短縮ウェーイ'
neologdn.normalize("チルダ削除ウェ~∼∾〜〰～イ")
# => 'チルダ削除ウェイ'
neologdn.normalize("いろんなハイフン˗֊‐‑‒–⁃⁻₋−")
# => 'いろんなハイフン-'
neologdn.normalize("　　　ＰＲＭＬ　　副　読　本　　　")
# => 'PRML副読本'
neologdn.normalize(" Natural Language Processing ")
# => 'Natural Language Processing'

Benchmark

# Sample code from
# https://github.com/neologd/mecab-ipadic-neologd/wiki/Regexp.ja#python-written-by-hideaki-t--overlast
import normalize_neologd

%timeit normalize(normalize_neologd.normalize_neologd)
# => 1000 loops, best of 3: 1.18 ms per loop


import neologdn
%timeit normalize(neologdn.normalize)
# => 10000 loops, best of 3: 140 µs per loop

neologdn is about x10 faster than sample code.

details are described as the below notebook: https://github.com/ikegami-yukino/neologdn/blob/master/benchmark/benchmark.ipynb

License

Apache Software License.

CHANGES

0.1.2 (2016-03-29)

Fix installation bug

0.1.1.1 (2016-03-19)

Support Windows
Explicitly specify to -std=c++11 in build (This bug is reported by id774. Many thanks)

0.1.1 (2015-10-10)

Initial release.

Metadata

Release files for neologdn 0.1.2

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for neologdn 0.1.2
File	Size	Uploaded	Provenance
neologdn-0.1.2.tar.gz	42.8 kB	Mar 29, 2016	Without provenance	Details

Release files / neologdn-0.1.2.tar.gz

Download URL	neologdn-0.1.2.tar.gz
Size	42.8 kB
Tags	Source
SHA-256 checksum How to use checksums	`071e4f8cdc3f3466d58f8285b241d0d310278b608d654be66b2089e381ddf27f`
BLAKE2b-256 checksum How to use checksums	`8f97fed71dc2b82a21288f9771c96055a737a8023f8e1ff948a8bb9b8ddead75`
Upload date	Mar 29, 2016
Uploaded using Trusted Publishing? What is trusted publishing?	No

Security for neologdn 0.1.2

Report project as malware

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.