A streaming tokenizer.
Project description
pyTokenizer
A streaming tokenizer.
Contributors
- Patrick Lehmann (Maintainer)
License
This Python package (source code) is licensed under Apache License 2.0.
SPDX-License-Identifier: Apache-2.0
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
pyTokenizer-1.1.4.tar.gz
(5.2 kB
view details)
Built Distribution
File details
Details for the file pyTokenizer-1.1.4.tar.gz.
File metadata
- Download URL: pyTokenizer-1.1.4.tar.gz
- Upload date:
- Size: 5.2 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.24.0 setuptools/47.1.0 requests-toolbelt/0.9.1 tqdm/4.48.2 CPython/3.8.5
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | d31e6e01730faae89383f32a0a4e497fb29955b291b941c556d92d43c110fd6c |
|
| MD5 | 7baf37af5630aebe731140a26960bf40 |
|
| BLAKE2b-256 | c0a631c06100242c4bbc1e6e68248e1df76d9eab16524b48d9b8550405054ddf |
File details
Details for the file pyTokenizer-1.1.4-py3-none-any.whl.
File metadata
- Download URL: pyTokenizer-1.1.4-py3-none-any.whl
- Upload date:
- Size: 9.0 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.24.0 setuptools/47.1.0 requests-toolbelt/0.9.1 tqdm/4.48.2 CPython/3.8.5
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | 117de4342add39128cd38202f4af098ef6ee1938e6a7fd749f27093e6d23daca |
|
| MD5 | 2bd4c84e30a98597b5d87ead2128075b |
|
| BLAKE2b-256 | 97588b6d3189bd4c10d1092763f53ed42bdba1bfea6374f5e66dd376d21cf7ed |
