llm-tokenizer
Tokenizer utilities for LLM inference
Status
This package is under development as part of the Shepherd project.
Installation
pip install llm-tokenizer
Coming Soon
This package will provide Python bindings (via PyO3) for the Rust implementation.
Metadata
Release files for llm-tokenizer 0.0.1
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | |
|---|---|---|---|---|
| llm_tokenizer-0.0.1.tar.gz | 1.4 kB | Without provenance | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| llm_tokenizer-0.0.1-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 3.1 kB
Release files / llm_tokenizer-0.0.1.tar.gz
| Download URL | llm_tokenizer-0.0.1.tar.gz |
|---|---|
| Size | 1.4 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
e9160dfa49b87064e4128ed84cfd265a6aed808876ec0f7d19b08402eb6e3cec
|
|
BLAKE2b-256 checksum How to use checksums |
659c45d80c3da261b0940588ca963342a2eb19943f55dc65495f82726ac8a15a
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.2.0 CPython/3.9.6
|
Release files / llm_tokenizer-0.0.1-py3-none-any.whl
| Download URL | llm_tokenizer-0.0.1-py3-none-any.whl |
|---|---|
| Size | 1.7 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
245075c82f718e1d5651fb6ebf397859540441c0fcc78bbab9546cb5a93fc8ef
|
|
BLAKE2b-256 checksum How to use checksums |
8f868bb38c4d0f5d508ea4b0b5ce1ea84420f8709e2c7b35dcab11152b16eb1b
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.2.0 CPython/3.9.6
|
Security for llm-tokenizer 0.0.1
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.