Fast Sentence Tokenizer (fast-sentence-tokenize)
Best in class tokenizer
Usage
Import
from fast_sentence_tokenize import fast_sentence_tokenize
Call Tokenizer
results = fast_sentence_tokenize("isn't a test great!!?")
Results
[
"isn't",
"a",
"test",
"great",
"!",
"!",
"?"
]
Note that whitespace is not preserved in the output by default.
This generally results in a more accurate parse from downstream components, but may make the reassembly of the original sentence more challenging.
Preserve Whitespace
results = fast_sentence_tokenize("isn't a test great!!?", eliminate_whitespace=False)
Results
[
"isn't ",
"a ",
"test ",
"great",
"!",
"!",
"?"
]
This option preserves whitespace.
This is useful if you want to re-assemble the tokens using the pre-existing spacing
assert ''.join(tokens) == input_text
1 maintainer
Metadata
1 maintainer
Release files for fast-sentence-tokenize 0.1.15
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| fast_sentence_tokenize-0.1.15.tar.gz | 9.3 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| fast_sentence_tokenize-0.1.15-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 23.2 kB
Release files / fast_sentence_tokenize-0.1.15.tar.gz
| Download URL | fast_sentence_tokenize-0.1.15.tar.gz |
|---|---|
| Size | 9.3 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
0f5d8f5691f8dc41e321eac720ddaf1cb59fd33259e5482f78992e26162ac294
|
|
BLAKE2b-256 checksum How to use checksums |
36591c68d48388ab9d7e6e77a2d6029d94317159bd1d6dadb6a533facd99cdf1
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
poetry/1.5.1 CPython/3.11.4 Darwin/22.5.0
|
Release files / fast_sentence_tokenize-0.1.15-py3-none-any.whl
| Download URL | fast_sentence_tokenize-0.1.15-py3-none-any.whl |
|---|---|
| Size | 13.9 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
85eed0ba762a6f919c7628b8c6951c5a09abf8f0544bfcf5add033c0e59e0b8d
|
|
BLAKE2b-256 checksum How to use checksums |
0ebc4f5de44e36700aff3303c1def32e7d155146cd9070d7f92ca8904e9983c2
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
poetry/1.5.1 CPython/3.11.4 Darwin/22.5.0
|
