Skip to main content

DicTok

A dictionary-based tokenizer. It tokenizes a text based on known tokens.

Installation

pip install dictok

Usage

  1. Create your dic-file with a list of tokens e.g. tokens.dic:
super
man
note
book
store
...
  1. Import dictok and pass it the dictionary file as main parameter:
>>> import dictok
>>> dt = dictok.DicTok('tokens.dic')
  1. You are ready to use it:
>>> sent = "Superman bought a notebook in the bookstore."
>>> dt.tokenize(sent)
['Super', 'man', 'bought', 'a', 'note', 'book', 'in', 'the', 'book', 'store', '.']

Options

You can also ignore single characters or unknown tokens:

>>> dt.tokenize(sent, include_unknown = False, include_single_chars = False)
['Super', 'man', 'note', 'book', 'book', 'store']

If you want to rewrite words or, for example, recognize and correct words with typing errors, you can do so by specifying them as pair in the dictionary:

super
man
bought,buy
note
book
buok,book
store
stohre,store
...
>>> dt = dictok.DicTok('tokens.dic')
>>> sent = "Superman bought a notebuok in the bookstohre."
>>> dt.tokenize(sent, include_unknown = False, include_single_chars = False)
['Super', 'man', 'buy', 'note', 'book', 'book', 'store']

Metadata

Release files for dictok 0.0.5

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for dictok 0.0.5
File Size Uploaded Provenance Actions
dictok-0.0.5.tar.gz 3.5 kB Without provenance Details for dictok-0.0.5.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for dictok 0.0.5
File Interpreter ABI Platform Actions
dictok-0.0.5-py3-none-any.whl Python 3 none any Details for dictok-0.0.5-py3-none-any.whl

Total release size: 7.3 kB

Release files / dictok-0.0.5.tar.gz

Download URL dictok-0.0.5.tar.gz
Size 3.5 kB
Tags Source
SHA-256 checksum
How to use checksums
34364ec7edd0ba66291f47e32c0d8a776ff1dab8368d757f35bf7a14a49fd89c
BLAKE2b-256 checksum
How to use checksums
4eca69299906ca025b23cd6ecce7953f165ee6070741ff648a405d26fbf7fc0a
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/4.0.1 CPython/3.8.10

Release files / dictok-0.0.5-py3-none-any.whl

Download URL dictok-0.0.5-py3-none-any.whl
Size 3.8 kB
Tags Python 3
SHA-256 checksum
How to use checksums
9b9a66c089d382ace195633566c7bb5bdcd2bd1e55ea3b0261d1d1f2bcd2289e
BLAKE2b-256 checksum
How to use checksums
57f96c90b15346921abbc569ddf17943e26329641f05ebf31abda65dbaabf940
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/4.0.1 CPython/3.8.10

Security for dictok 0.0.5

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page