Skip to main content

Tools for tokenizer develope and evaluation

Project description

Tokenizer Tools

https://img.shields.io/pypi/v/tokenizer_tools.svg https://img.shields.io/travis/howlandersonn/tokenizer_tools.svg Documentation Status Updates

Tools/Utils for NLP (including dataset reading, tagset encoding & decoding, metrics computing) | NLP 工具集(包含数据集读取、tagset 编码和解码、指标的计算等)

Features

  • 常见数据集格式的读取

  • 多种 Tagset 的编码和解码

  • 指标的计算

TODO

  • 改变项目的名字,tokenizer_tools 已经无法正确描述现在项目的功能

Credits

This package was created with Cookiecutter and the audreyr/cookiecutter-pypackage project template.

History

0.1.0 (2018-09-05)

  • First release on PyPI.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

tokenizer_tools-0.14.0.tar.gz (25.4 kB view hashes)

Uploaded Source

Built Distribution

tokenizer_tools-0.14.0-py2.py3-none-any.whl (28.6 kB view hashes)

Uploaded Python 2 Python 3

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page