Tokenizer utilities for LLM inference
Project description
llm-tokenizer
Tokenizer utilities for LLM inference
Status
This package is under development as part of the Shepherd project.
Installation
pip install llm-tokenizer
Coming Soon
This package will provide Python bindings (via PyO3) for the Rust implementation.
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
llm_tokenizer-0.0.1.tar.gz
(1.4 kB
view details)
Built Distribution
Filter files by name, interpreter, ABI, and platform.
If you're not sure about the file name format, learn more about wheel file names.
Copy a direct link to the current filters
File details
Details for the file llm_tokenizer-0.0.1.tar.gz.
File metadata
- Download URL: llm_tokenizer-0.0.1.tar.gz
- Upload date:
- Size: 1.4 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/6.2.0 CPython/3.9.6
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
e9160dfa49b87064e4128ed84cfd265a6aed808876ec0f7d19b08402eb6e3cec
|
|
| MD5 |
a89184e31b1f072c8d2585131dd32842
|
|
| BLAKE2b-256 |
659c45d80c3da261b0940588ca963342a2eb19943f55dc65495f82726ac8a15a
|
File details
Details for the file llm_tokenizer-0.0.1-py3-none-any.whl.
File metadata
- Download URL: llm_tokenizer-0.0.1-py3-none-any.whl
- Upload date:
- Size: 1.7 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/6.2.0 CPython/3.9.6
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
245075c82f718e1d5651fb6ebf397859540441c0fcc78bbab9546cb5a93fc8ef
|
|
| MD5 |
6425073cd25ba8ee6cdf0e63e5b0fa62
|
|
| BLAKE2b-256 |
8f868bb38c4d0f5d508ea4b0b5ce1ea84420f8709e2c7b35dcab11152b16eb1b
|
