A Python tokenizer trained on modern web corpus
Project description
BTok
A Python multilingual tokenizer trained on modern web corpus with SentencePiece.
Install
pip install btok --upgrade
Usage
Run example:
python example.py
See: example.py
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
btok-0.1.tar.gz
(2.1 kB
view details)
Built Distribution
Filter files by name, interpreter, ABI, and platform.
If you're not sure about the file name format, learn more about wheel file names.
Copy a direct link to the current filters
btok-0.1-py3-none-any.whl
(2.3 kB
view details)
File details
Details for the file btok-0.1.tar.gz.
File metadata
- Download URL: btok-0.1.tar.gz
- Upload date:
- Size: 2.1 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/6.1.0 CPython/3.9.22
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
a4be6ab2ae11185995e085559f64a0c4aac0eaaef01b2684ea106741203512cf
|
|
| MD5 |
b8844d4b1779cd15ab4ec9b36b386847
|
|
| BLAKE2b-256 |
c48786c32cb9e3977162b64ca13f363a7c04d9f26866da2a99163d2abc4501ca
|
File details
Details for the file btok-0.1-py3-none-any.whl.
File metadata
- Download URL: btok-0.1-py3-none-any.whl
- Upload date:
- Size: 2.3 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/6.1.0 CPython/3.9.22
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
2038cb1f0cf57ba9c2f1d12d5bf25e70df49b389d1a7e69e7464e96558113f75
|
|
| MD5 |
1e3d7d4ef85bd279a661e25e19cfc4e7
|
|
| BLAKE2b-256 |
f829863b96c3f83bdfa73e7cb1a45500d9653a02935856fcea52c57eb67f5f05
|
