Text tokenizers.
Project description
totokenizers
A model-agnostic library to encode text into tokens and couting them using different tokenizers.
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
totokenizers-1.0.0.tar.gz
(7.1 kB
view details)
Built Distribution
File details
Details for the file totokenizers-1.0.0.tar.gz.
File metadata
- Download URL: totokenizers-1.0.0.tar.gz
- Upload date:
- Size: 7.1 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/4.0.2 CPython/3.11.4
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | f3ddd984af10fd36ee715d58908d7c361067d3c6c40123bba20731b387b51468 |
|
| MD5 | 8485872b9d2c09e1b2cf9bce90968b38 |
|
| BLAKE2b-256 | f9a860477cd0bb3dbfd6fc5d5bb3d4b3e992c29448b3c4d8a1e00692745a19da |
Provenance
File details
Details for the file totokenizers-1.0.0-py3-none-any.whl.
File metadata
- Download URL: totokenizers-1.0.0-py3-none-any.whl
- Upload date:
- Size: 7.5 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/4.0.2 CPython/3.11.4
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | 680d9229ca382d2a359c268e5594a986074321fd8c40715a350336ba7b7b4858 |
|
| MD5 | d870ef25ba2a6c9c9f1d1f1d439b0b25 |
|
| BLAKE2b-256 | 069226a4126242559408b6d173705562823eea9686e2e26026ac99a4837a7634 |
