tokenizers-collection

中文分词器集合

https://img.shields.io/pypi/v/chinese_tokenzier_iterator.svg

https://img.shields.io/travis/howl-anderson/chinese_tokenzier_iterator.svg

一些中文分词器的简单封装和集合

Free software: MIT license
Documentation: https://chinese-tokenzier-iterator.readthedocs.io.

Features

TODO

使用

from tokenizers_collection.config import tokenizer_registry
for name, tokenizer in tokenizer_registry:
    print("Tokenizer: {}".format(name))
    tokenizer('input_file.txt', 'output_file.txt')

安装

pip install tokenizers_collection

更新许可文件与下载模型

因为其中有些模型需要更新许可文件（比如：pynlpir）或者需要下载模型文件（比如：pyltp），因此安装后需要执行特定的命令完成操作，这里已经将所有的操作封装成了一个函数，只需要执行类似如下的指令即可

python -m tokenizers_collection.helper

Credits

This package was created with Cookiecutter and the audreyr/cookiecutter-pypackage project template.

History

0.1.0 (2018-08-28)

First release on PyPI.

Metadata

Release files for tokenizers-collection 0.1.2

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for tokenizers-collection 0.1.2
File	Size	Uploaded
tokenizers_collection-0.1.2.tar.gz	11.2 kB	Aug 28, 2018	Details

Built distribution (wheel)

Table of built distributions (wheels) for tokenizers-collection 0.1.2
File	Interpreter	ABI	Platform	Reset
tokenizers_collection-0.1.2-py2.py3-none-any.whl 7.7 kB Aug 28, 2018	Python 2, Python 3	none	any	Details

Total release size: 18.9 kB

Release files / tokenizers_collection-0.1.2.tar.gz

Download URL	tokenizers_collection-0.1.2.tar.gz
Size	11.2 kB
Tags	Source
SHA-256 checksum How to use checksums	`b43e162333bf43e2ae80d567bc9dff48ed9ecfdffbfbf2fafc62035c52b39f9e`
BLAKE2b-256 checksum How to use checksums	`5f13524a0fae90c6254b9ccc62ec416ef5236e892b7e8455a8043c3cfc94e961`
Upload date	Aug 28, 2018
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/1.10.0 pkginfo/1.4.2 requests/2.19.1 setuptools/39.1.0 requests-toolbelt/0.8.0 tqdm/4.25.0 CPython/3.6.5`

Release files / tokenizers_collection-0.1.2-py2.py3-none-any.whl

Download URL	tokenizers_collection-0.1.2-py2.py3-none-any.whl
Size	7.7 kB
Tags	Python 2 Python 3
SHA-256 checksum How to use checksums	`40bf32f5352f0542a8c92c7bab73451f21c873676568ed9709acdfb88601dc03`
BLAKE2b-256 checksum How to use checksums	`f20a2058b20bbaf939b8cfad4c7205af0853fbd6d75ff057c6e194f25f591dda`
Upload date	Aug 28, 2018
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/1.10.0 pkginfo/1.4.2 requests/2.19.1 setuptools/39.1.0 requests-toolbelt/0.8.0 tqdm/4.25.0 CPython/3.6.5`