detokenize 0.0.4

Package to detokenize the stream of tokens

De-Tokenize

---

What is it?

De-Tokenize is a Python package which provides fast, accurate structuring of tokens back to original sentence form.

Contributor

Littin Rajan

Main Features

• Light-weight package
• No more dependencies
• Powerful, flexible

Where to get it?

The source code is currently hosted on GitHub at: https://github.com/littinrajan/detokenize

Binary installers for the latest released version are available at the Python Package Index (PyPI) and on Conda.

# conda
conda install detokenize

# or PyPI
pip install detokenize

Usage

from detokenize.detokenizer import detokenize

sample_tokens = ['These', 'are', 'some', 'tokens', '.']
sentence = detokenize(sample_tokens)

License

MIT

Contributing to De-Tokenize

All contributions, bug reports, bug fixes, documentation improvements, enhancements, and ideas are welcome.