This release is a pre-release and may not be stable for production use.
tokenizers Python bindings
This module is still experimental, expect the API to change or break. If used in a production setting, please pin the version number.
Install
pip install --pre tokenizers
Usage
from tokenizers import Padding, Tokenizer
# A `Tokenizer` represents a tokenizer's spec (json) and can be edited.
# You can load it from the JSON file
tokenizer = Tokenizer.from_file("tokenizer.json")
# Or load the `tokenizer.json` of a model repo on the Hub.
# Downloads go through the `huggingface_hub` package
tokenizer = Tokenizer.from_pretrained("openai-community/gpt2", revision="main")
# `compile()` compiles the specification to an optimized tokenizer in rust.
# The resulting `CompiledTokenizer` object can encode text to token ids and decode ids back to text.
# It's immutable and can be shared among threads.
compiled = tokenizer.compile()
encoding = compiled.encode("Hello there, how are you?")
encoding.ids # [101, 7592, 2045, 1010, 2129, 2024, 2017, 1029, 102]
encoding.type_ids # [0, 0, 0, 0, 0, 0, 0, 0, 0]
encoding.attention_mask # [1, 1, 1, 1, 1, 1, 1, 1, 1]
# The same fields for array libraries, through DLPack. Each read is an `int64` copy.
torch.tensor(encoding.ids_dlpack) # tensor([101, 7592, ...]), torch 2.6 or later
np.from_dlpack(encoding.attention_mask_dlpack) # array([1, 1, ...])
compiled.encode("Hello", add_special_tokens=False).ids # no [CLS]/[SEP]
compiled.decode(encoding.ids) # "hello there, how are you?"
compiled.decode(encoding.ids, skip_special_tokens=False)
# Pad one call. `length=None` pads each batch to its longest item.
short, long = compiled.encode_batch(
["Hello", "Hello there, how are you?"],
padding=Padding(direction="left", pad_id=0, pad_token="[PAD]"),
)
len(short) == len(long) # True
short.attention_mask # [0, 0, 0, 0, 0, 0, 1, 1, 1]
compiled.encode("Hello", padding=Padding(length=16)) # exactly 16 ids
# Padding and truncation can also be set on the spec.
# `padding=None` switches the default off for one call.
tokenizer.padding = Padding(pad_to_multiple_of=8)
padded = tokenizer.compile()
padded.encode("Hello", padding=None)
examples/ holds a runnable script for each of these; make examples runs them all.
multiprocessing and Worker processes
Every class here can be pickled, so a tokenizer can be sent to a worker process and its encodings
sent back: multiprocessing, a PyTorch DataLoader, datasets.map(num_proc=...).
A pickled CompiledTokenizer carries its whole tokenizer.json, and unpickling compiles it again.
With the "spawn" start method, we recommend compiling the tokenizer once per worker to avoid unpickle compilation overhead.
def compile_tokenizer(tokenizer):
global compiled
compiled = tokenizer.compile()
def encode(text):
return compiled.encode(text)
with multiprocessing.get_context("spawn").Pool(4, initializer=compile_tokenizer, initargs=(tokenizer,)) as pool:
encoded = list(pool.imap(encode, texts, chunksize=32))
With the "fork" start method, compiling before starting the pool is recommended: the workers inherit the compiled tokenizer
and nothing is pickled.
Developers
make developregenerates the.pyistub, rebuilds the extension (debug) and reinstalls it in editable mode.make testdoes the above, then runstests/with pytest.make examplesdoes the above, then runs every script inexamples/.make stubsregeneratespython/tokenizers/tokenizers.pyi, including the memberspython/tokenizers/__init__.pyattaches to the Rust classes.make style/make check-styleregenerate the stub, then format/lint the Rust and Python sides and type-check withty. CI runsmake check-styleand fails if the regenerated stub differs from the committed one.
Metadata
Release files for tokenizers 1.0.0rc5
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| tokenizers-1.0.0rc5.tar.gz | 473.4 kB | With provenance | Details for tokenizers-1.0.0rc5.tar.gz |
Built distributions (wheels)
Total release size: 45.8 MB
Release files / tokenizers-1.0.0rc5.tar.gz
| Download URL | tokenizers-1.0.0rc5.tar.gz |
|---|---|
| Size | 473.4 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
fb6e30d77b8b2a8ed9ce02c000e81e5841f2aa9d402c3b6019442ec70951e4cc
|
|
BLAKE2b-256 checksum How to use checksums |
5693ac5b5c177eaeda69d2d31e7acf8006d4022ad7da4729958ca1851120c3da
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 20, 2026.
Transparency logRelease files / tokenizers-1.0.0rc5-cp314-cp314t-win_amd64.whl
| Download URL | tokenizers-1.0.0rc5-cp314-cp314t-win_amd64.whl |
|---|---|
| Size | 1.8 MB |
| Tags | CPython 3.14 CPython 3.14 free-threading Windows x86-64 |
|
SHA-256 checksum How to use checksums |
60da685a57087da42b8975d8fa06c3cece017d592b33cb741d5f3ebbdd295965
|
|
BLAKE2b-256 checksum How to use checksums |
92ca9bd2cdb913337f8481c52f67199d19f63ded0c3a20c1b8b62c2c01b7ce5e
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 20, 2026.
Transparency logRelease files / tokenizers-1.0.0rc5-cp314-cp314t-musllinux_1_2_x86_64.whl
| Download URL | tokenizers-1.0.0rc5-cp314-cp314t-musllinux_1_2_x86_64.whl |
|---|---|
| Size | 2.2 MB |
| Tags | CPython 3.14 CPython 3.14 free-threading Linux musl 1.2+ x86-64 |
|
SHA-256 checksum How to use checksums |
da0c339102d24bdc0f2a1b9d138b658d2e1c1068683ba5163bfe1aee643b51b9
|
|
BLAKE2b-256 checksum How to use checksums |
832a5f50989f150764c85152d6926e442ef49860627cb9df6bd7bd5bcfb1ec0c
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 20, 2026.
Transparency logRelease files / tokenizers-1.0.0rc5-cp314-cp314t-musllinux_1_2_aarch64.whl
| Download URL | tokenizers-1.0.0rc5-cp314-cp314t-musllinux_1_2_aarch64.whl |
|---|---|
| Size | 2.1 MB |
| Tags | CPython 3.14 CPython 3.14 free-threading Linux musl 1.2+ ARM64 |
|
SHA-256 checksum How to use checksums |
05051df7a2a312d652f3a9a40a705c946b78df29b9130b1a71a31df956f088f7
|
|
BLAKE2b-256 checksum How to use checksums |
dc5849905cbd062b0eb455172a8a837ad0c7dc0ea1ac21a318971ac37da28d4a
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 20, 2026.
Transparency logRelease files / tokenizers-1.0.0rc5-cp314-cp314t-manylinux2014_x86_64.manylinux_2_17_x86_64.whl
| Download URL | tokenizers-1.0.0rc5-cp314-cp314t-manylinux2014_x86_64.manylinux_2_17_x86_64.whl |
|---|---|
| Size | 2.0 MB |
| Tags | CPython 3.14 CPython 3.14 free-threading Linux glibc 2.17+ x86-64 |
|
SHA-256 checksum How to use checksums |
aeb234fdca831930fce5955feac8135a38bde59c75f0bb5704c4598ec0cf5daf
|
|
BLAKE2b-256 checksum How to use checksums |
a18ca354bc18ddea381747733b1485dd71a49a2da51539de1c97f6cd6746584c
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 20, 2026.
Transparency logRelease files / tokenizers-1.0.0rc5-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.whl
| Download URL | tokenizers-1.0.0rc5-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.whl |
|---|---|
| Size | 1.9 MB |
| Tags | CPython 3.14 CPython 3.14 free-threading Linux glibc 2.17+ ARM64 |
|
SHA-256 checksum How to use checksums |
660b11f0715f46b3d5c58f0655f3b3db2231b3ffc559853b8d48892889ab92fd
|
|
BLAKE2b-256 checksum How to use checksums |
43d4bb77b0113ac21b0329ecdd7a55c851e3d87b1e19961bb79f10183352e56e
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 20, 2026.
Transparency logRelease files / tokenizers-1.0.0rc5-cp314-cp314t-macosx_11_0_arm64.whl
| Download URL | tokenizers-1.0.0rc5-cp314-cp314t-macosx_11_0_arm64.whl |
|---|---|
| Size | 1.8 MB |
| Tags | CPython 3.14 CPython 3.14 free-threading macOS 11.0+ ARM64 |
|
SHA-256 checksum How to use checksums |
b889bf50d1f9cd893a103977da000c8f93a0f092218ee29b31708fef39943c61
|
|
BLAKE2b-256 checksum How to use checksums |
e64f8b8159f49d84a2894225a9a8dee0818cf873560fe5313f48b616285da47f
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 20, 2026.
Transparency logRelease files / tokenizers-1.0.0rc5-cp314-cp314t-macosx_10_12_x86_64.whl
| Download URL | tokenizers-1.0.0rc5-cp314-cp314t-macosx_10_12_x86_64.whl |
|---|---|
| Size | 1.9 MB |
| Tags | CPython 3.14 CPython 3.14 free-threading macOS 10.12+ x86-64 |
|
SHA-256 checksum How to use checksums |
d2e0fa78af07c67dd59b44b302ffa3768197d7556a7f6ba6a82e28c559a25a8c
|
|
BLAKE2b-256 checksum How to use checksums |
c77d05c6eab87e6e1b53d2f9b5c2ce9ffc019ba2b7f1b0aad96bc8d04be09d14
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 20, 2026.
Transparency logRelease files / tokenizers-1.0.0rc5-cp311-abi3-win_arm64.whl
| Download URL | tokenizers-1.0.0rc5-cp311-abi3-win_arm64.whl |
|---|---|
| Size | 1.7 MB |
| Tags | CPython 3.11 Windows ARM64 abi3 |
|
SHA-256 checksum How to use checksums |
2e57a1f2040e0ee87372f51187a3fc1878b48bd5f8f694c9e8419d144681c379
|
|
BLAKE2b-256 checksum How to use checksums |
b4537ed7b4b23a7c4b1c68af054be3ac980827c51a146ac426395c603ea7ad3f
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 20, 2026.
Transparency logRelease files / tokenizers-1.0.0rc5-cp311-abi3-win_amd64.whl
| Download URL | tokenizers-1.0.0rc5-cp311-abi3-win_amd64.whl |
|---|---|
| Size | 1.8 MB |
| Tags | CPython 3.11 Windows x86-64 abi3 |
|
SHA-256 checksum How to use checksums |
b953761b0bb18062a7074c765ff63f578b9c9ce6ec2d86023efac1b8f68ccc22
|
|
BLAKE2b-256 checksum How to use checksums |
4ee3f4f92de5799efdac5a8354d4baf4fbbe9d050063a640bfc093de2102d4ee
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 20, 2026.
Transparency logRelease files / tokenizers-1.0.0rc5-cp311-abi3-win32.whl
| Download URL | tokenizers-1.0.0rc5-cp311-abi3-win32.whl |
|---|---|
| Size | 1.7 MB |
| Tags | CPython 3.11 Windows x86-32 abi3 |
|
SHA-256 checksum How to use checksums |
adf6ce5659f26c1f283d9d0c68952e7b4130c75296aedc35edc230d54b549765
|
|
BLAKE2b-256 checksum How to use checksums |
2d90b525df396e97dc5291bfe949770a9c32d976fbb88bf66c69ef0ed28142bb
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 20, 2026.
Transparency logRelease files / tokenizers-1.0.0rc5-cp311-abi3-musllinux_1_2_x86_64.whl
| Download URL | tokenizers-1.0.0rc5-cp311-abi3-musllinux_1_2_x86_64.whl |
|---|---|
| Size | 2.2 MB |
| Tags | CPython 3.11 Linux musl 1.2+ x86-64 abi3 |
|
SHA-256 checksum How to use checksums |
a20086309c2004b3230409ca20894e3082b3dcb606bec88a9d9bb0b6ce7ef44e
|
|
BLAKE2b-256 checksum How to use checksums |
724400526e446888b28eef55049133ac52cace30ddbfddd360f7f3a36e745598
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 20, 2026.
Transparency logRelease files / tokenizers-1.0.0rc5-cp311-abi3-musllinux_1_2_i686.whl
| Download URL | tokenizers-1.0.0rc5-cp311-abi3-musllinux_1_2_i686.whl |
|---|---|
| Size | 2.2 MB |
| Tags | CPython 3.11 Linux musl 1.2+ x86-32 abi3 |
|
SHA-256 checksum How to use checksums |
747261784c0e91df82f5301c08947558ff779e898d19de2086a05c93d7876130
|
|
BLAKE2b-256 checksum How to use checksums |
c61a735b999f461b2ea9a69bca05063caf6e9515ce2b9c47db0165215a8d06f6
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 20, 2026.
Transparency logRelease files / tokenizers-1.0.0rc5-cp311-abi3-musllinux_1_2_armv7l.whl
| Download URL | tokenizers-1.0.0rc5-cp311-abi3-musllinux_1_2_armv7l.whl |
|---|---|
| Size | 2.2 MB |
| Tags | CPython 3.11 Linux musl 1.2+ ARMv7l abi3 |
|
SHA-256 checksum How to use checksums |
2333d9801f178ab8fdbc6c0967182bbd49024566dfd77903b9c2caa2487d7f46
|
|
BLAKE2b-256 checksum How to use checksums |
2d1978d01190da34f431ea0bea426f3c1a2f6cdf64a0804d99ba124cbd629ecc
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 20, 2026.
Transparency logRelease files / tokenizers-1.0.0rc5-cp311-abi3-musllinux_1_2_aarch64.whl
| Download URL | tokenizers-1.0.0rc5-cp311-abi3-musllinux_1_2_aarch64.whl |
|---|---|
| Size | 2.1 MB |
| Tags | CPython 3.11 Linux musl 1.2+ ARM64 abi3 |
|
SHA-256 checksum How to use checksums |
bdfe33571db4c20b11c56b2c619873b6da3cd30ee80ecd438a7261b105749c71
|
|
BLAKE2b-256 checksum How to use checksums |
f20a5492baf9d29214f65d5fe5c72f201f6203dcb063d43009cdf44f4f655e83
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 20, 2026.
Transparency logRelease files / tokenizers-1.0.0rc5-cp311-abi3-manylinux_2_31_riscv64.whl
| Download URL | tokenizers-1.0.0rc5-cp311-abi3-manylinux_2_31_riscv64.whl |
|---|---|
| Size | 2.0 MB |
| Tags | CPython 3.11 Linux glibc 2.31+ RISC-V 64 abi3 |
|
SHA-256 checksum How to use checksums |
f5f6308b4f49fc1751d5d2b58fbe3f951b44309b55242ddf2fa1c5bc0e0ae35d
|
|
BLAKE2b-256 checksum How to use checksums |
77478ebfdcd9dea206b820716a590138179deaa47ad61fc3731221a9f7c16539
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 20, 2026.
Transparency logRelease files / tokenizers-1.0.0rc5-cp311-abi3-manylinux2014_x86_64.manylinux_2_17_x86_64.whl
| Download URL | tokenizers-1.0.0rc5-cp311-abi3-manylinux2014_x86_64.manylinux_2_17_x86_64.whl |
|---|---|
| Size | 2.0 MB |
| Tags | CPython 3.11 Linux glibc 2.17+ x86-64 abi3 |
|
SHA-256 checksum How to use checksums |
2c2ff15cbca2eef9219ce215aaa51f6969b4a68b267030a0d0195af391e31c7a
|
|
BLAKE2b-256 checksum How to use checksums |
465c9a994e4d6a42da0f52fd02a257f128414acb2276e32a5ff48c14199ae823
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 20, 2026.
Transparency logRelease files / tokenizers-1.0.0rc5-cp311-abi3-manylinux2014_s390x.manylinux_2_17_s390x.whl
| Download URL | tokenizers-1.0.0rc5-cp311-abi3-manylinux2014_s390x.manylinux_2_17_s390x.whl |
|---|---|
| Size | 2.1 MB |
| Tags | CPython 3.11 Linux glibc 2.17+ IBM System/390x abi3 |
|
SHA-256 checksum How to use checksums |
7bd25daef7384ccd4dc0bd7daebdf85f97e52415b1a44daaef3e645efed62aa8
|
|
BLAKE2b-256 checksum How to use checksums |
87bc9a49229b9e428fa5c9b30ecbdc6342510f99ffefc8f771e5a90b429f3535
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 20, 2026.
Transparency logRelease files / tokenizers-1.0.0rc5-cp311-abi3-manylinux2014_ppc64le.manylinux_2_17_ppc64le.whl
| Download URL | tokenizers-1.0.0rc5-cp311-abi3-manylinux2014_ppc64le.manylinux_2_17_ppc64le.whl |
|---|---|
| Size | 2.1 MB |
| Tags | CPython 3.11 Linux glibc 2.17+ PowerPC 64-le abi3 |
|
SHA-256 checksum How to use checksums |
af37930aeec644b140c2fd03db79505b41a5969ca359aec45b04b88fd6343dd0
|
|
BLAKE2b-256 checksum How to use checksums |
751c0815c31a738e6686dd1ad02f686abcecc1bc429e6243cd65217c198cb0f6
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 20, 2026.
Transparency logRelease files / tokenizers-1.0.0rc5-cp311-abi3-manylinux2014_armv7l.manylinux_2_17_armv7l.whl
| Download URL | tokenizers-1.0.0rc5-cp311-abi3-manylinux2014_armv7l.manylinux_2_17_armv7l.whl |
|---|---|
| Size | 1.9 MB |
| Tags | CPython 3.11 Linux glibc 2.17+ ARMv7l abi3 |
|
SHA-256 checksum How to use checksums |
9666181150851269a63349a6dffa5690ff828a4f9c4208d3a620feee96bb029c
|
|
BLAKE2b-256 checksum How to use checksums |
6637d49f656c7d0d7fc006d5a7bf65d5734ba1533ae3c96e727327a0a8d63aed
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 20, 2026.
Transparency logRelease files / tokenizers-1.0.0rc5-cp311-abi3-manylinux2014_aarch64.manylinux_2_17_aarch64.whl
| Download URL | tokenizers-1.0.0rc5-cp311-abi3-manylinux2014_aarch64.manylinux_2_17_aarch64.whl |
|---|---|
| Size | 1.9 MB |
| Tags | CPython 3.11 Linux glibc 2.17+ ARM64 abi3 |
|
SHA-256 checksum How to use checksums |
c86a2153ff600fc5ef37e4c42763fea83474b89969adaba676e94e4c6457ef1d
|
|
BLAKE2b-256 checksum How to use checksums |
7e2e69e97da75141d2adce910a3e4f306141febe88ef6238da762588685744f5
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 20, 2026.
Transparency logRelease files / tokenizers-1.0.0rc5-cp311-abi3-manylinux2010_i686.manylinux_2_12_i686.whl
| Download URL | tokenizers-1.0.0rc5-cp311-abi3-manylinux2010_i686.manylinux_2_12_i686.whl |
|---|---|
| Size | 2.0 MB |
| Tags | CPython 3.11 Linux glibc 2.12+ x86-32 abi3 |
|
SHA-256 checksum How to use checksums |
b2a8437d3b7edb7a2d28c9d6b348141630b99d8fae373a6c6a56a66c659695aa
|
|
BLAKE2b-256 checksum How to use checksums |
51d56d5a0c906d7cceff5c8e22007942f754de73717fcaade47d638f3742f9fe
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 20, 2026.
Transparency logRelease files / tokenizers-1.0.0rc5-cp311-abi3-macosx_11_0_arm64.whl
| Download URL | tokenizers-1.0.0rc5-cp311-abi3-macosx_11_0_arm64.whl |
|---|---|
| Size | 1.8 MB |
| Tags | CPython 3.11 abi3 macOS 11.0+ ARM64 |
|
SHA-256 checksum How to use checksums |
c6d861ee0d7d9605fb1fbb76ddd915b4a790b991d011943cd9ca4fba9f34a1a5
|
|
BLAKE2b-256 checksum How to use checksums |
82c16a29e93379eb15aea7ad2b6a111938dce524bb78fcb086e1d4989ece61b2
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 20, 2026.
Transparency logRelease files / tokenizers-1.0.0rc5-cp311-abi3-macosx_10_12_x86_64.whl
| Download URL | tokenizers-1.0.0rc5-cp311-abi3-macosx_10_12_x86_64.whl |
|---|---|
| Size | 1.9 MB |
| Tags | CPython 3.11 abi3 macOS 10.12+ x86-64 |
|
SHA-256 checksum How to use checksums |
1ef7781876ac7b100c158c88dc7d288b04c988e0d75529b34a0cd588f00d0184
|
|
BLAKE2b-256 checksum How to use checksums |
b1006ac4b44896277b2f37393c67483083a4b024773e9bf6c83e41d686cd1943
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 20, 2026.
Transparency logSecurity for tokenizers 1.0.0rc5
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
Full provenance
All 24 files in this release have been uploaded with provenance metadata.
Attestation for tokenizers-1.0.0rc5.tar.gz
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 20, 2026.
Transparency logA note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.