This release is a pre-release and may not be stable for production use.
tokenizers Python bindings
This module is still experimental, expect the API to change or break. If used in a production setting, please pin the version number.
Install
uv sync
Usage
from tokenizers import Padding, Tokenizer
tokenizer = Tokenizer.from_file("tokenizer.json")
# Or the `tokenizer.json` of a model repo on the Hub. `huggingface_hub` downloads and caches it,
# with the same cache, login token and offline switch as transformers.
tokenizer = Tokenizer.from_pretrained("openai-community/gpt2", revision="main")
encoding = tokenizer.encode("Hello there, how are you?")
encoding.ids # [101, 7592, 2045, 1010, 2129, 2024, 2017, 1029, 102]
encoding.type_ids # [0, 0, 0, 0, 0, 0, 0, 0, 0]
encoding.attention_mask # [1, 1, 1, 1, 1, 1, 1, 1, 1]
# The same fields as numpy arrays. Each read returns a new copy that the caller owns.
encoding.ids_array # array([101, 7592, 2045, 1010, 2129, 2024, 2017, 1029, 102], dtype=uint32)
encoding.type_ids_array # array([0, 0, 0, 0, 0, 0, 0, 0, 0], dtype=uint32)
encoding.attention_mask_array # array([1, 1, 1, 1, 1, 1, 1, 1, 1], dtype=uint32)
torch.from_numpy(encoding.ids_array) # no-copy translation
tokenizer.encode("Hello", add_special_tokens=False).ids # no [CLS]/[SEP]
tokenizer.decode(encoding.ids) # "hello there, how are you?"
tokenizer.decode(encoding.ids_array) # a numpy array works too
tokenizer.decode(encoding.ids, skip_special_tokens=False)
# Override the `Padding` settings. `length=None` pads each batch to its longest
# item; `None` switches padding off.
tokenizer.padding = Padding(direction="left", pad_id=0, pad_token="[PAD]")
short, long = tokenizer.encode_batch(["Hello", "Hello there, how are you?"])
len(short) == len(long) # True
short.attention_mask # [0, 0, 0, 0, 0, 0, 1, 1, 1]
tokenizer.padding = Padding(length=16) # every encoding is exactly 16 ids
tokenizer.padding = Padding(pad_to_multiple_of=8)
tokenizer.padding = None
# Or for one call only, leaving the attribute as it is. `None` switches it off for that call.
tokenizer.encode_batch(["Hello", "Hello there"], padding=Padding(length=16))
tokenizer.encode("Hello", padding=None)
examples/ holds a runnable script for each of these; make examples runs them all.
Worker processes
Every class here can be pickled, so a tokenizer can be sent to a worker process and its encodings
sent back: multiprocessing, a PyTorch DataLoader, datasets.map(num_proc=...).
def encode(tokenizer, texts):
return tokenizer.encode_batch(texts)
with multiprocessing.get_context("spawn").Pool(4) as pool:
encoded = pool.starmap(encode, [(tokenizer, chunk) for chunk in chunks])
The tokenizer is pickled again with every task it is sent with, so give each worker one big chunk
of texts rather than many small tasks. examples/multiprocessing_workers.py is the whole script.
Day to day
make developregenerates the.pyistub, rebuilds the extension (debug) and reinstalls it in editable mode.make testdoes the above, then runstests/with pytest.make examplesdoes the above, then runs every script inexamples/.make stubsregeneratespython/tokenizers/tokenizers.pyi, including the memberspython/tokenizers/__init__.pyattaches to the Rust classes.make style/make check-styleregenerate the stub, then format/lint the Rust and Python sides and type-check withty. CI runsmake check-styleand fails if the regenerated stub differs from the committed one.
4 maintainers
Metadata
4 maintainers
Release files for tokenizers 1.0.0rc3
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | |
|---|---|---|---|---|
| tokenizers-1.0.0rc3.tar.gz | 467.9 kB | With provenance | Details |
Built distributions (wheels)
Total release size: 42.6 MB
Release files / tokenizers-1.0.0rc3.tar.gz
| Download URL | tokenizers-1.0.0rc3.tar.gz |
|---|---|
| Size | 467.9 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
a0bb19c9b219f59e39933c3b7c5621187174f9ff000cc5bd851d2e0c86e7f4b2
|
|
BLAKE2b-256 checksum How to use checksums |
995a42947f99359de8ac229f97f8e43c738860d90f522868446ab4bc5cca4adf
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 19, 2026.
Transparency logRelease files / tokenizers-1.0.0rc3-cp314-cp314t-win_amd64.whl
| Download URL | tokenizers-1.0.0rc3-cp314-cp314t-win_amd64.whl |
|---|---|
| Size | 1.7 MB |
| Tags | CPython 3.14 CPython 3.14 free-threading Windows x86-64 |
|
SHA-256 checksum How to use checksums |
22deed3acf83ca477478646afc573747ac4f203891d8cf27886bbc4a0b325162
|
|
BLAKE2b-256 checksum How to use checksums |
ddbb72cbd20cfc9e61865e6af18fb2109bff22a6eab6cdc9e40947a363ce06b1
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 19, 2026.
Transparency logRelease files / tokenizers-1.0.0rc3-cp314-cp314t-musllinux_1_2_x86_64.whl
| Download URL | tokenizers-1.0.0rc3-cp314-cp314t-musllinux_1_2_x86_64.whl |
|---|---|
| Size | 2.1 MB |
| Tags | CPython 3.14 CPython 3.14 free-threading Linux musl 1.2+ x86-64 |
|
SHA-256 checksum How to use checksums |
f9469a6ba54b83efe1aba10a7c64230ad98fea3b192528cb989cb1e94055de94
|
|
BLAKE2b-256 checksum How to use checksums |
67f440e4f5a05a4a678668f9b0ac8d06b73cd328f691efa048063d6acd7fc616
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 19, 2026.
Transparency logRelease files / tokenizers-1.0.0rc3-cp314-cp314t-musllinux_1_2_aarch64.whl
| Download URL | tokenizers-1.0.0rc3-cp314-cp314t-musllinux_1_2_aarch64.whl |
|---|---|
| Size | 1.9 MB |
| Tags | CPython 3.14 CPython 3.14 free-threading Linux musl 1.2+ ARM64 |
|
SHA-256 checksum How to use checksums |
0dfebe55a928b3fa2ed1bfc83ca35c054ee2f9451da5bd1feff47e5efcd09eda
|
|
BLAKE2b-256 checksum How to use checksums |
31e1a919dd4d440496cbdb721b185059612ad4f8001f1c7f1da006105db159e1
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 19, 2026.
Transparency logRelease files / tokenizers-1.0.0rc3-cp314-cp314t-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
| Download URL | tokenizers-1.0.0rc3-cp314-cp314t-manylinux_2_17_x86_64.manylinux2014_x86_64.whl |
|---|---|
| Size | 1.8 MB |
| Tags | CPython 3.14 CPython 3.14 free-threading Linux glibc 2.17+ x86-64 |
|
SHA-256 checksum How to use checksums |
7d7d1ba12cc435d8b6ccd4b8000452248bb41edf674380f987e9f6280b6441c0
|
|
BLAKE2b-256 checksum How to use checksums |
07a47b9fa1485921919b265fb72c22a90ec2a93e93f52e0c9a61d22f39895bd6
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 19, 2026.
Transparency logRelease files / tokenizers-1.0.0rc3-cp314-cp314t-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
| Download URL | tokenizers-1.0.0rc3-cp314-cp314t-manylinux_2_17_aarch64.manylinux2014_aarch64.whl |
|---|---|
| Size | 1.8 MB |
| Tags | CPython 3.14 CPython 3.14 free-threading Linux glibc 2.17+ ARM64 |
|
SHA-256 checksum How to use checksums |
5b6ae2527f3a84cb07d74f64917fd73ef1a2f61c8302a2aa6b08d8f4d3c917dc
|
|
BLAKE2b-256 checksum How to use checksums |
56051cee5521d8a2ddbc6eafafc2756d5ae13c94c5fd327e8adc3aeab9f8e2f0
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 19, 2026.
Transparency logRelease files / tokenizers-1.0.0rc3-cp314-cp314t-macosx_11_0_arm64.whl
| Download URL | tokenizers-1.0.0rc3-cp314-cp314t-macosx_11_0_arm64.whl |
|---|---|
| Size | 1.7 MB |
| Tags | CPython 3.14 CPython 3.14 free-threading macOS 11.0+ ARM64 |
|
SHA-256 checksum How to use checksums |
35305ef35542fc3a8f198b5ddf56a2dc14365224fe8300b6c6c73407e8685778
|
|
BLAKE2b-256 checksum How to use checksums |
c79af83db3fd8e3cdd53ad447e7fc4602a328d0e3c59cf9f4e57219a6e946897
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 19, 2026.
Transparency logRelease files / tokenizers-1.0.0rc3-cp314-cp314t-macosx_10_12_x86_64.whl
| Download URL | tokenizers-1.0.0rc3-cp314-cp314t-macosx_10_12_x86_64.whl |
|---|---|
| Size | 1.8 MB |
| Tags | CPython 3.14 CPython 3.14 free-threading macOS 10.12+ x86-64 |
|
SHA-256 checksum How to use checksums |
98464abb0168203b14cbf2e8c7bddd1800ef9d0e3c1e163c87ff38658ac03f02
|
|
BLAKE2b-256 checksum How to use checksums |
98680e930055bd8764995c479a7ad50ed622b5c82c9e2c2adc6ed63312473c01
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 19, 2026.
Transparency logRelease files / tokenizers-1.0.0rc3-cp310-abi3-win_arm64.whl
| Download URL | tokenizers-1.0.0rc3-cp310-abi3-win_arm64.whl |
|---|---|
| Size | 1.6 MB |
| Tags | CPython 3.10 Windows ARM64 abi3 |
|
SHA-256 checksum How to use checksums |
1fca88f9713747cdf192496a676acc4410a312aaadc8e0357c9043a44a22ee60
|
|
BLAKE2b-256 checksum How to use checksums |
d21a0e378ad8f01c7246228c25daa9824f5fd06fc7858345449062805ef2cfb6
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 19, 2026.
Transparency logRelease files / tokenizers-1.0.0rc3-cp310-abi3-win_amd64.whl
| Download URL | tokenizers-1.0.0rc3-cp310-abi3-win_amd64.whl |
|---|---|
| Size | 1.7 MB |
| Tags | CPython 3.10 Windows x86-64 abi3 |
|
SHA-256 checksum How to use checksums |
496d27d95c65c515c577b7e27e75119138b4e0ea6b6d4479f06a3715ad8185a2
|
|
BLAKE2b-256 checksum How to use checksums |
1eb787ba644e121ee57a5cbda7901d1bfe3e1fc697556f0399db6bfe50c90389
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 19, 2026.
Transparency logRelease files / tokenizers-1.0.0rc3-cp310-abi3-win32.whl
| Download URL | tokenizers-1.0.0rc3-cp310-abi3-win32.whl |
|---|---|
| Size | 1.5 MB |
| Tags | CPython 3.10 Windows x86-32 abi3 |
|
SHA-256 checksum How to use checksums |
bc8d40079c804f40a47fb6a63dc00dd0230ded00dd82e64cc346aa9ce5ed46ec
|
|
BLAKE2b-256 checksum How to use checksums |
03dc69704acd3028f4eee4908d5709fd864e8a3c533153b680d50bbfc261f313
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 19, 2026.
Transparency logRelease files / tokenizers-1.0.0rc3-cp310-abi3-musllinux_1_2_x86_64.whl
| Download URL | tokenizers-1.0.0rc3-cp310-abi3-musllinux_1_2_x86_64.whl |
|---|---|
| Size | 2.1 MB |
| Tags | CPython 3.10 Linux musl 1.2+ x86-64 abi3 |
|
SHA-256 checksum How to use checksums |
4bda3135d47058f23a383960d14aecf02d28fa6ceaab69c2760c50d163642d7f
|
|
BLAKE2b-256 checksum How to use checksums |
b3c1da933b09a497e398008be47e2f989af38abe42b69681eb73b2eadaea54ef
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 19, 2026.
Transparency logRelease files / tokenizers-1.0.0rc3-cp310-abi3-musllinux_1_2_i686.whl
| Download URL | tokenizers-1.0.0rc3-cp310-abi3-musllinux_1_2_i686.whl |
|---|---|
| Size | 2.0 MB |
| Tags | CPython 3.10 Linux musl 1.2+ x86-32 abi3 |
|
SHA-256 checksum How to use checksums |
075054ff8093a48f8b41c41db823b094b6c2700ca41d58e0050c95e063dee936
|
|
BLAKE2b-256 checksum How to use checksums |
033e1589864f764309f981b5020622a2bd1be1b361896eab391e563299576b1b
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 19, 2026.
Transparency logRelease files / tokenizers-1.0.0rc3-cp310-abi3-musllinux_1_2_armv7l.whl
| Download URL | tokenizers-1.0.0rc3-cp310-abi3-musllinux_1_2_armv7l.whl |
|---|---|
| Size | 2.1 MB |
| Tags | CPython 3.10 Linux musl 1.2+ ARMv7l abi3 |
|
SHA-256 checksum How to use checksums |
f537d4636cb8d9224cf96a0e7643c72d713d22a33bef5084e94f9af31389a4bd
|
|
BLAKE2b-256 checksum How to use checksums |
6b6bfa34bcd5f079b746b5d64df7b1db0c78f2be28b46fd8a2f0fa2066047aad
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 19, 2026.
Transparency logRelease files / tokenizers-1.0.0rc3-cp310-abi3-musllinux_1_2_aarch64.whl
| Download URL | tokenizers-1.0.0rc3-cp310-abi3-musllinux_1_2_aarch64.whl |
|---|---|
| Size | 2.0 MB |
| Tags | CPython 3.10 Linux musl 1.2+ ARM64 abi3 |
|
SHA-256 checksum How to use checksums |
05d195d9a54fa9b3b59908aab6a12e16c32bce64fd652fd6d66f89572b37b329
|
|
BLAKE2b-256 checksum How to use checksums |
79738c8304c4339b78038ac599a7d86ffa945387c48c2b6a0cad12f2ea13da20
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 19, 2026.
Transparency logRelease files / tokenizers-1.0.0rc3-cp310-abi3-manylinux_2_31_riscv64.whl
| Download URL | tokenizers-1.0.0rc3-cp310-abi3-manylinux_2_31_riscv64.whl |
|---|---|
| Size | 1.8 MB |
| Tags | CPython 3.10 Linux glibc 2.31+ RISC-V 64 abi3 |
|
SHA-256 checksum How to use checksums |
a8f2a26bc0ae07347dcbec716c10aeb43dc21f237736f020ff00ae759c57d0a4
|
|
BLAKE2b-256 checksum How to use checksums |
c2b777b62f339f53900206006dfc2dc7f29d59486c2300e11aa44b8a3ab6994b
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 19, 2026.
Transparency logRelease files / tokenizers-1.0.0rc3-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
| Download URL | tokenizers-1.0.0rc3-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl |
|---|---|
| Size | 1.8 MB |
| Tags | CPython 3.10 Linux glibc 2.17+ x86-64 abi3 |
|
SHA-256 checksum How to use checksums |
8b7aa195d351b39bf640626062aa807ca34cca647e3f433596788077c0736699
|
|
BLAKE2b-256 checksum How to use checksums |
0c5859280be5b2a2f1d5641561586afa0a56e520e6195882d4b90fbad27cfa88
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 19, 2026.
Transparency logRelease files / tokenizers-1.0.0rc3-cp310-abi3-manylinux_2_17_s390x.manylinux2014_s390x.whl
| Download URL | tokenizers-1.0.0rc3-cp310-abi3-manylinux_2_17_s390x.manylinux2014_s390x.whl |
|---|---|
| Size | 1.9 MB |
| Tags | CPython 3.10 Linux glibc 2.17+ IBM System/390x abi3 |
|
SHA-256 checksum How to use checksums |
2071fc7dad1aca82dc6ba0f310ff02506b04a6f37e558b1735b9bdadac53237a
|
|
BLAKE2b-256 checksum How to use checksums |
a116938f309afbe9079d81fef252345d0a80efc9a3e618e5eb52675a29158306
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 19, 2026.
Transparency logRelease files / tokenizers-1.0.0rc3-cp310-abi3-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl
| Download URL | tokenizers-1.0.0rc3-cp310-abi3-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl |
|---|---|
| Size | 1.9 MB |
| Tags | CPython 3.10 Linux glibc 2.17+ PowerPC 64-le abi3 |
|
SHA-256 checksum How to use checksums |
bed8446a3a465f49c35195aa213aba1306362a3aae8e21439be46ce30cbcab0d
|
|
BLAKE2b-256 checksum How to use checksums |
10eb9cb966278e3de9ff595ed4893f94250f3b203b5c2bf9c89e219eec3bfb51
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 19, 2026.
Transparency logRelease files / tokenizers-1.0.0rc3-cp310-abi3-manylinux_2_17_armv7l.manylinux2014_armv7l.whl
| Download URL | tokenizers-1.0.0rc3-cp310-abi3-manylinux_2_17_armv7l.manylinux2014_armv7l.whl |
|---|---|
| Size | 1.8 MB |
| Tags | CPython 3.10 Linux glibc 2.17+ ARMv7l abi3 |
|
SHA-256 checksum How to use checksums |
bdb49e437026ebf1f096afb64e7ef0c303905ecf9cd4cf748b147f511567b2a7
|
|
BLAKE2b-256 checksum How to use checksums |
98057d81b74d58c8d1809faa619ee94267cbdfdd80b509cfd52b0b139b45e489
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 19, 2026.
Transparency logRelease files / tokenizers-1.0.0rc3-cp310-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
| Download URL | tokenizers-1.0.0rc3-cp310-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl |
|---|---|
| Size | 1.8 MB |
| Tags | CPython 3.10 Linux glibc 2.17+ ARM64 abi3 |
|
SHA-256 checksum How to use checksums |
16835d0b8fadda45f82debb3b2c33896dd94bbf50c4b0854e1ff29780b8d3af3
|
|
BLAKE2b-256 checksum How to use checksums |
0af6e4168f4b169cd10b7e9e8026098a2b5bbf5676bda7e3e0d2e1b6a810e51f
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 19, 2026.
Transparency logRelease files / tokenizers-1.0.0rc3-cp310-abi3-manylinux_2_12_i686.manylinux2010_i686.whl
| Download URL | tokenizers-1.0.0rc3-cp310-abi3-manylinux_2_12_i686.manylinux2010_i686.whl |
|---|---|
| Size | 1.9 MB |
| Tags | CPython 3.10 Linux glibc 2.12+ x86-32 abi3 |
|
SHA-256 checksum How to use checksums |
9d531c26ffeace3f4d0c6d541bf2080f6d5030b6e0ff90bbb64badffe0fc078f
|
|
BLAKE2b-256 checksum How to use checksums |
4adb397cdb4bd5f6aa32b9e27d2df73e2f8b0699b8b77df2205399a86a6ea339
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 19, 2026.
Transparency logRelease files / tokenizers-1.0.0rc3-cp310-abi3-macosx_11_0_arm64.whl
| Download URL | tokenizers-1.0.0rc3-cp310-abi3-macosx_11_0_arm64.whl |
|---|---|
| Size | 1.7 MB |
| Tags | CPython 3.10 abi3 macOS 11.0+ ARM64 |
|
SHA-256 checksum How to use checksums |
71f3da74b33d48da337dc2d86b994d884d9ef8b480cebb85e7a451f49d388a41
|
|
BLAKE2b-256 checksum How to use checksums |
72c035529945dba260898f19b96ae4f314e0502ee954c04225b640fcb700346e
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 19, 2026.
Transparency logRelease files / tokenizers-1.0.0rc3-cp310-abi3-macosx_10_12_x86_64.whl
| Download URL | tokenizers-1.0.0rc3-cp310-abi3-macosx_10_12_x86_64.whl |
|---|---|
| Size | 1.8 MB |
| Tags | CPython 3.10 abi3 macOS 10.12+ x86-64 |
|
SHA-256 checksum How to use checksums |
5668cca210f58ae2fd07d51c48ec30f965c26e2d1c8d7f865acbcca0b18521a4
|
|
BLAKE2b-256 checksum How to use checksums |
112105d6bd86d3ae96aa869374a8585040d1b9c352fce34133e971287790655b
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 19, 2026.
Transparency logSecurity for tokenizers 1.0.0rc3
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
Full provenance
All 24 files in this release have been uploaded with provenance metadata.
Attestation for tokenizers-1.0.0rc3.tar.gz
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 19, 2026.
Transparency logA note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.
