Package to detokenize the stream of tokens
Project description
De-Tokenize
What is it?
De-Tokenize is a Python package which provides fast, accurate structuring of tokens back to original sentence form.
Contributor
Littin Rajan
Main Features
- Light-weight package
- No more dependencies
- Powerful, flexible
Where to get it?
The source code is currently hosted on GitHub at: https://github.com/littinrajan/detokenize
Binary installers for the latest released version are available at the Python Package Index (PyPI) and on Conda.
# conda
conda install detokenize
# or PyPI
pip install detokenize
Usage
from detokenize.detokenizer import detokenize
sample_tokens = ['These', 'are', 'some', 'tokens', '.']
sentence = detokenize(sample_tokens)
License
Contributing to De-Tokenize
All contributions, bug reports, bug fixes, documentation improvements, enhancements, and ideas are welcome.
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
detokenize-0.0.4.tar.gz
(3.0 kB
view details)
Built Distribution
File details
Details for the file detokenize-0.0.4.tar.gz.
File metadata
- Download URL: detokenize-0.0.4.tar.gz
- Upload date:
- Size: 3.0 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/4.0.2 CPython/3.7.9
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | 22c8db6a9051ff2fa10aad9befe30d2c1cb7ce2208ad7d9ddc88e89003bebf69 |
|
| MD5 | 158a28d76a915c90acf583363d0b5569 |
|
| BLAKE2b-256 | fcde9c0bc033c21dbe52dea2298f3f5862ffea5895198546720b44b881dbfda0 |
File details
Details for the file detokenize-0.0.4-py3-none-any.whl.
File metadata
- Download URL: detokenize-0.0.4-py3-none-any.whl
- Upload date:
- Size: 3.5 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/4.0.2 CPython/3.7.9
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | e21886a1fa1636c47d5c7b80a1ba39f139cb144cc08eebad4b96d4d548e86a44 |
|
| MD5 | 856ba9ae02b50596c46191f350752aab |
|
| BLAKE2b-256 | b3c6091cd35835dc377afab31556db914e71c88ac79a3bd9114b8ee504a7185c |
