phrase-tokenizer 0.1.1

Tokenize an English sentence to phrases

Phrase Tokenizer

Tokenize an English sentence to phrases

Installation

pip install phrase-tokenizer
# pip install phrase-tokenizer -U to update

Or clone the repo https://github.com/ffreemt/phrase-tokenizer.git:

git clone https://github.com/ffreemt/phrase-tokenizer.git
cd phrase-tokenizer
pip install logzero benepar tensorflow

Or use poetry, e.g.

git clone https://github.com/ffreemt/phrase-tokenizer.git
cd phrase-tokenizer
poetry install

Usage

from phrase_tokenizer import phrase_tok

res = phrase_tok("Short cuts make long delay.")
print(res)
# ['Short cuts', 'make long delay']

# verbose=True turns on verbose to see the tokenizing process
res = phrase_tok("Short cuts make long delay", verbose=True)
# ',..Short.cuts,.make..long.delay..'

Consult the source code for details.

For Developers

git clone https://github.com/ffreemt/phrase-tokenizer.git
cd phrase-tokenizer
pip install -r requirements-dev.txt

In jupyter notebook, plot_tree is able to draw a nice tree to aid the development, e.g.,

from phrase_tokenizer.phrase_tok import plot_tree

plot_tree("Short cuts make long delay.")