Text tokenizers.
Project description
totokenizers
A model-agnostic library to encode text into tokens and couting them using different tokenizers.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
totokenizers-1.0.0.tar.gz
(7.1 kB
view hashes)
Built Distribution
Close
Hashes for totokenizers-1.0.0-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 680d9229ca382d2a359c268e5594a986074321fd8c40715a350336ba7b7b4858 |
|
MD5 | d870ef25ba2a6c9c9f1d1f1d439b0b25 |
|
BLAKE2b-256 | 069226a4126242559408b6d173705562823eea9686e2e26026ac99a4837a7634 |