fast-sentence-tokenize 0.1.15

Fast and Efficient Sentence Tokenization

Fast Sentence Tokenizer (fast-sentence-tokenize)

Best in class tokenizer

Usage

Import

from fast_sentence_tokenize import fast_sentence_tokenize

Call Tokenizer

results = fast_sentence_tokenize("isn't a test great!!?")

Results

[
   "isn't",
   "a",
   "test",
   "great",
   "!",
   "!",
   "?"
]

Note that whitespace is not preserved in the output by default.

This generally results in a more accurate parse from downstream components, but may make the reassembly of the original sentence more challenging.

Preserve Whitespace

results = fast_sentence_tokenize("isn't a test great!!?", eliminate_whitespace=False)

Results

[
   "isn't ",
   "a ",
   "test ",
   "great",
   "!",
   "!",
   "?"
]

This option preserves whitespace.

This is useful if you want to re-assemble the tokens using the pre-existing spacing

assert ''.join(tokens) == input_text