rtok
A Python tokenizer for LLMs using GitHub's linear BPE implementation.
[rtok]
ebook (90k) 0.12907366300350986
a*100 7.582298712804914e-05
a*1000 0.00013318200944922864
a*10000 0.0013674889924004674
a*100000 0.01401260400598403
[tiktoken]
ebook (90k) 0.23613008800020907
a*100 0.00018489900685381144
a*1000 0.003490732007776387
a*10000 0.3407805879978696
a*100000 33.41563105300884
API
rtok.openai.get_o200k_base() -> Encoder
rtok.openai.get_cl100k_base() -> Encoder
Encoder.count(str)
Encoder.count_till_limit(str, limit: int) -> Optional[int]
Encoder.encode(str) -> [int]
Encoder.decode([int]) -> str
Encoder.encode and Encoder.decode are compatible with tiktoken. See test.py.
Encoder.count_till_limit() returns None if the count exceeds the limit.
1 maintainer
Metadata
1 maintainer
Release files for rtok 0.1.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | |
|---|---|---|---|---|
| rtok-0.1.0.tar.gz | 92.5 kB | Without provenance | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| rtok-0.1.0-cp312-cp312-manylinux_2_34_x86_64.whl | CPython 3.12 | CPython 3.12 | Linux glibc 2.34+ x86-64 | Details |
Total release size: 25.9 MB
Release files / rtok-0.1.0.tar.gz
| Download URL | rtok-0.1.0.tar.gz |
|---|---|
| Size | 92.5 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
e35174a23039851206611d463f1a959a7fc8482a6cea0501576a24331fc078a9
|
|
BLAKE2b-256 checksum How to use checksums |
2f83be406291a8bd02e95d206d01aa76b56d1f07094fc10ee6344e241fc321b4
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
maturin/1.8.2
|
Release files / rtok-0.1.0-cp312-cp312-manylinux_2_34_x86_64.whl
| Download URL | rtok-0.1.0-cp312-cp312-manylinux_2_34_x86_64.whl |
|---|---|
| Size | 25.8 MB |
| Tags | CPython 3.12 Linux glibc 2.34+ x86-64 |
|
SHA-256 checksum How to use checksums |
6eb442fed093a0751db32b8557ea6e3868df66f6b0e484ef48a6b000ba6b6c82
|
|
BLAKE2b-256 checksum How to use checksums |
6a35c51dbe1274a264fbe88f1ad4cc3df7a46cd4badd8d2c653299963a88d206
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
maturin/1.8.2
|
Security for rtok 0.1.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
