Penn Treebank tokenizer
This is a simple fork of the famous Penn Treebank tokenizer. It is forked from DetectorMorse via NLTK.
- It is appropriate for English, but not other languages.
- It is appropriate when applied one sentence at a time, but should not be applied to paragraphs or documents.
Unlike the NLTK equivalent, it has no (library or data) dependencies except the
built-in re. Unlike the NLTK
equivalent, it is not hostilely
polymorphic.
1 maintainer
Metadata
1 maintainer
Release files for ptbtok 0.1
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| ptbtok-0.1.tar.gz | 3.1 kB | Details |
Release files / ptbtok-0.1.tar.gz
| Download URL | ptbtok-0.1.tar.gz |
|---|---|
| Size | 3.1 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
33a7f1447dba8edad9a9fa89921dea44f0d2ccf3634dbc6a55b2383a2f69da6f
|
|
BLAKE2b-256 checksum How to use checksums |
c34cf7bfe412c409a2fb74b0726a6a7d540704c70c625eca1f5b7bcec663f414
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/3.4.1 importlib_metadata/3.7.3 pkginfo/1.7.0 requests/2.24.0 requests-toolbelt/0.9.1 tqdm/4.59.0 CPython/3.8.8
|
