Skip to main content

llm-tokenizer

Tokenizer utilities for LLM inference

Status

This package is under development as part of the Shepherd project.

Installation

pip install llm-tokenizer

Coming Soon

This package will provide Python bindings (via PyO3) for the Rust implementation.

Metadata

Release files for llm-tokenizer 0.0.1

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for llm-tokenizer 0.0.1
File Size Uploaded Provenance
llm_tokenizer-0.0.1.tar.gz 1.4 kB Without provenance Details

Built distribution (wheel)

Table of built distributions (wheels) for llm-tokenizer 0.0.1
File Interpreter ABI Platform
llm_tokenizer-0.0.1-py3-none-any.whl Python 3 none any Details

Total release size: 3.1 kB

Release files / llm_tokenizer-0.0.1.tar.gz

Download URL llm_tokenizer-0.0.1.tar.gz
Size 1.4 kB
Tags Source
SHA-256 checksum
How to use checksums
e9160dfa49b87064e4128ed84cfd265a6aed808876ec0f7d19b08402eb6e3cec
BLAKE2b-256 checksum
How to use checksums
659c45d80c3da261b0940588ca963342a2eb19943f55dc65495f82726ac8a15a
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/6.2.0 CPython/3.9.6

Release files / llm_tokenizer-0.0.1-py3-none-any.whl

Download URL llm_tokenizer-0.0.1-py3-none-any.whl
Size 1.7 kB
Tags Python 3
SHA-256 checksum
How to use checksums
245075c82f718e1d5651fb6ebf397859540441c0fcc78bbab9546cb5a93fc8ef
BLAKE2b-256 checksum
How to use checksums
8f868bb38c4d0f5d508ea4b0b5ce1ea84420f8709e2c7b35dcab11152b16eb1b
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/6.2.0 CPython/3.9.6

Security for llm-tokenizer 0.0.1

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

Release history Release notifications | RSS feed

This release

0.0.1 This release

2 release files No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page