easy-tokenizer 0.0.1

tokenizer tool

Easy-Tokenizer

Description

Try with the following script: easy_tokenizer input_text

Status

todo

Requirements

Python 3.6+

Installation

pip install easy-tokenizer

Usage

todo

Development

To install package and its dependencies, run the following from project root directory:

python setup.py install

To work the code and develop the package, run the following from project root directory:

python setup.py develop

To run unit tests, execute the following from the project root directory:

python setup.py test

0.0.1 (2019-10-22)

• Add the first version of the tokenizer