DicTok
A dictionary-based tokenizer. It tokenizes a text based on known tokens.
Installation
pip install dictok
Usage
- Create your dic-file with a list of tokens e.g.
tokens.dic:
super
man
note
book
store
...
- Import
dictokand pass it the dictionary file as main parameter:
>>> import dictok
>>> dt = dictok.DicTok('tokens.dic')
- You are ready to use it:
>>> sent = "Superman bought a notebook in the bookstore."
>>> dt.tokenize(sent)
['Super', 'man', 'bought', 'a', 'note', 'book', 'in', 'the', 'book', 'store', '.']
Options
You can also ignore single characters or unknown tokens:
>>> dt.tokenize(sent, include_unknown = False, include_single_chars = False)
['Super', 'man', 'note', 'book', 'book', 'store']
If you want to rewrite words or, for example, recognize and correct words with typing errors, you can do so by specifying them as pair in the dictionary:
super
man
bought,buy
note
book
buok,book
store
stohre,store
...
>>> dt = dictok.DicTok('tokens.dic')
>>> sent = "Superman bought a notebuok in the bookstohre."
>>> dt.tokenize(sent, include_unknown = False, include_single_chars = False)
['Super', 'man', 'buy', 'note', 'book', 'book', 'store']
Metadata
Release files for dictok 0.0.5
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| dictok-0.0.5.tar.gz | 3.5 kB | Without provenance | Details for dictok-0.0.5.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| dictok-0.0.5-py3-none-any.whl | Python 3 | none | any | Details for dictok-0.0.5-py3-none-any.whl |
Total release size: 7.3 kB
Release files / dictok-0.0.5.tar.gz
| Download URL | dictok-0.0.5.tar.gz |
|---|---|
| Size | 3.5 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
34364ec7edd0ba66291f47e32c0d8a776ff1dab8368d757f35bf7a14a49fd89c
|
|
BLAKE2b-256 checksum How to use checksums |
4eca69299906ca025b23cd6ecce7953f165ee6070741ff648a405d26fbf7fc0a
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/4.0.1 CPython/3.8.10
|
Release files / dictok-0.0.5-py3-none-any.whl
| Download URL | dictok-0.0.5-py3-none-any.whl |
|---|---|
| Size | 3.8 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
9b9a66c089d382ace195633566c7bb5bdcd2bd1e55ea3b0261d1d1f2bcd2289e
|
|
BLAKE2b-256 checksum How to use checksums |
57f96c90b15346921abbc569ddf17943e26329641f05ebf31abda65dbaabf940
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/4.0.1 CPython/3.8.10
|
Security for dictok 0.0.5
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.