Skip to main content
Pre-release

This release is a pre-release and may not be stable for production use.

tokenizers Python bindings

This module is still experimental, expect the API to change or break. If used in a production setting, please pin the version number.

Install

pip install --pre tokenizers

Usage

from tokenizers import Padding, Tokenizer

# A `Tokenizer` represents a tokenizer's spec (json) and can be edited.
# You can load it from the JSON file
tokenizer = Tokenizer.from_file("tokenizer.json")
# Or load the `tokenizer.json` of a model repo on the Hub.
# Downloads go through the `huggingface_hub` package
tokenizer = Tokenizer.from_pretrained("openai-community/gpt2", revision="main")

# `compile()` compiles the specification to an optimized tokenizer in rust.
# The resulting `CompiledTokenizer` object can encode text to token ids and decode ids back to text.
# It's immutable and can be shared among threads.
compiled = tokenizer.compile()

encoding = compiled.encode("Hello there, how are you?")
encoding.ids             # [101, 7592, 2045, 1010, 2129, 2024, 2017, 1029, 102]
encoding.type_ids        # [0, 0, 0, 0, 0, 0, 0, 0, 0]
encoding.attention_mask  # [1, 1, 1, 1, 1, 1, 1, 1, 1]

# The same fields for array libraries, through DLPack. Each read is an `int64` copy.
torch.tensor(encoding.ids_dlpack)   # tensor([101, 7592, ...]), torch 2.6 or later
np.from_dlpack(encoding.attention_mask_dlpack)  # array([1, 1, ...])

compiled.encode("Hello", add_special_tokens=False).ids  # no [CLS]/[SEP]
compiled.decode(encoding.ids)                           # "hello there, how are you?"
compiled.decode(encoding.ids, skip_special_tokens=False)

# Pad one call. `length=None` pads each batch to its longest item.
short, long = compiled.encode_batch(
    ["Hello", "Hello there, how are you?"],
    padding=Padding(direction="left", pad_id=0, pad_token="[PAD]"),
)
len(short) == len(long)  # True
short.attention_mask     # [0, 0, 0, 0, 0, 0, 1, 1, 1]
compiled.encode("Hello", padding=Padding(length=16))  # exactly 16 ids

# Padding and truncation can also be set on the spec.
# `padding=None` switches the default off for one call.
tokenizer.padding = Padding(pad_to_multiple_of=8)
padded = tokenizer.compile()
padded.encode("Hello", padding=None)

examples/ holds a runnable script for each of these; make examples runs them all.

multiprocessing and Worker processes

Every class here can be pickled, so a tokenizer can be sent to a worker process and its encodings sent back: multiprocessing, a PyTorch DataLoader, datasets.map(num_proc=...).

A pickled CompiledTokenizer carries its whole tokenizer.json, and unpickling compiles it again. With the "spawn" start method, we recommend compiling the tokenizer once per worker to avoid unpickle compilation overhead.

def compile_tokenizer(tokenizer):
    global compiled
    compiled = tokenizer.compile()


def encode(text):
    return compiled.encode(text)


with multiprocessing.get_context("spawn").Pool(4, initializer=compile_tokenizer, initargs=(tokenizer,)) as pool:
    encoded = list(pool.imap(encode, texts, chunksize=32))

With the "fork" start method, compiling before starting the pool is recommended: the workers inherit the compiled tokenizer and nothing is pickled.

Developers

  • make develop regenerates the .pyi stub, rebuilds the extension (debug) and reinstalls it in editable mode.
  • make test does the above, then runs tests/ with pytest.
  • make examples does the above, then runs every script in examples/.
  • make stubs regenerates python/tokenizers/tokenizers.pyi, including the members python/tokenizers/__init__.py attaches to the Rust classes.
  • make style / make check-style regenerate the stub, then format/lint the Rust and Python sides and type-check with ty. CI runs make check-style and fails if the regenerated stub differs from the committed one.

Metadata

Release files for tokenizers 1.0.0rc5

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for tokenizers 1.0.0rc5
File Size Uploaded Provenance Actions
tokenizers-1.0.0rc5.tar.gz 473.4 kB With provenance Details for tokenizers-1.0.0rc5.tar.gz

Built distributions (wheels)

Table of built distributions (wheels) for tokenizers 1.0.0rc5
File Actions
tokenizers-1.0.0rc5-cp314-cp314t-win_amd64.whl CPython 3.14 CPython 3.14 free-threading Windows x86-64 Details for tokenizers-1.0.0rc5-cp314-cp314t-win_amd64.whl
tokenizers-1.0.0rc5-cp314-cp314t-musllinux_1_2_x86_64.whl CPython 3.14 CPython 3.14 free-threading Linux musl 1.2+ x86-64 Details for tokenizers-1.0.0rc5-cp314-cp314t-musllinux_1_2_x86_64.whl
tokenizers-1.0.0rc5-cp314-cp314t-musllinux_1_2_aarch64.whl CPython 3.14 CPython 3.14 free-threading Linux musl 1.2+ ARM64 Details for tokenizers-1.0.0rc5-cp314-cp314t-musllinux_1_2_aarch64.whl
tokenizers-1.0.0rc5-cp314-cp314t-manylinux2014_x86_64.manylinux_2_17_x86_64.whl CPython 3.14 CPython 3.14 free-threading Linux glibc 2.17+ x86-64 Details for tokenizers-1.0.0rc5-cp314-cp314t-manylinux2014_x86_64.manylinux_2_17_x86_64.whl
tokenizers-1.0.0rc5-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.whl CPython 3.14 CPython 3.14 free-threading Linux glibc 2.17+ ARM64 Details for tokenizers-1.0.0rc5-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.whl
tokenizers-1.0.0rc5-cp314-cp314t-macosx_11_0_arm64.whl CPython 3.14 CPython 3.14 free-threading macOS 11.0+ ARM64 Details for tokenizers-1.0.0rc5-cp314-cp314t-macosx_11_0_arm64.whl
tokenizers-1.0.0rc5-cp314-cp314t-macosx_10_12_x86_64.whl CPython 3.14 CPython 3.14 free-threading macOS 10.12+ x86-64 Details for tokenizers-1.0.0rc5-cp314-cp314t-macosx_10_12_x86_64.whl
tokenizers-1.0.0rc5-cp311-abi3-win_arm64.whl CPython 3.11 abi3 Windows ARM64 Details for tokenizers-1.0.0rc5-cp311-abi3-win_arm64.whl
tokenizers-1.0.0rc5-cp311-abi3-win_amd64.whl CPython 3.11 abi3 Windows x86-64 Details for tokenizers-1.0.0rc5-cp311-abi3-win_amd64.whl
tokenizers-1.0.0rc5-cp311-abi3-win32.whl CPython 3.11 abi3 Windows x86-32 Details for tokenizers-1.0.0rc5-cp311-abi3-win32.whl
tokenizers-1.0.0rc5-cp311-abi3-musllinux_1_2_x86_64.whl CPython 3.11 abi3 Linux musl 1.2+ x86-64 Details for tokenizers-1.0.0rc5-cp311-abi3-musllinux_1_2_x86_64.whl
tokenizers-1.0.0rc5-cp311-abi3-musllinux_1_2_i686.whl CPython 3.11 abi3 Linux musl 1.2+ x86-32 Details for tokenizers-1.0.0rc5-cp311-abi3-musllinux_1_2_i686.whl
tokenizers-1.0.0rc5-cp311-abi3-musllinux_1_2_armv7l.whl CPython 3.11 abi3 Linux musl 1.2+ ARMv7l Details for tokenizers-1.0.0rc5-cp311-abi3-musllinux_1_2_armv7l.whl
tokenizers-1.0.0rc5-cp311-abi3-musllinux_1_2_aarch64.whl CPython 3.11 abi3 Linux musl 1.2+ ARM64 Details for tokenizers-1.0.0rc5-cp311-abi3-musllinux_1_2_aarch64.whl
tokenizers-1.0.0rc5-cp311-abi3-manylinux_2_31_riscv64.whl CPython 3.11 abi3 Linux glibc 2.31+ RISC-V 64 Details for tokenizers-1.0.0rc5-cp311-abi3-manylinux_2_31_riscv64.whl
tokenizers-1.0.0rc5-cp311-abi3-manylinux2014_x86_64.manylinux_2_17_x86_64.whl CPython 3.11 abi3 Linux glibc 2.17+ x86-64 Details for tokenizers-1.0.0rc5-cp311-abi3-manylinux2014_x86_64.manylinux_2_17_x86_64.whl
tokenizers-1.0.0rc5-cp311-abi3-manylinux2014_s390x.manylinux_2_17_s390x.whl CPython 3.11 abi3 Linux glibc 2.17+ IBM System/390x Details for tokenizers-1.0.0rc5-cp311-abi3-manylinux2014_s390x.manylinux_2_17_s390x.whl
tokenizers-1.0.0rc5-cp311-abi3-manylinux2014_ppc64le.manylinux_2_17_ppc64le.whl CPython 3.11 abi3 Linux glibc 2.17+ PowerPC 64-le Details for tokenizers-1.0.0rc5-cp311-abi3-manylinux2014_ppc64le.manylinux_2_17_ppc64le.whl
tokenizers-1.0.0rc5-cp311-abi3-manylinux2014_armv7l.manylinux_2_17_armv7l.whl CPython 3.11 abi3 Linux glibc 2.17+ ARMv7l Details for tokenizers-1.0.0rc5-cp311-abi3-manylinux2014_armv7l.manylinux_2_17_armv7l.whl
tokenizers-1.0.0rc5-cp311-abi3-manylinux2014_aarch64.manylinux_2_17_aarch64.whl CPython 3.11 abi3 Linux glibc 2.17+ ARM64 Details for tokenizers-1.0.0rc5-cp311-abi3-manylinux2014_aarch64.manylinux_2_17_aarch64.whl
tokenizers-1.0.0rc5-cp311-abi3-manylinux2010_i686.manylinux_2_12_i686.whl CPython 3.11 abi3 Linux glibc 2.12+ x86-32 Details for tokenizers-1.0.0rc5-cp311-abi3-manylinux2010_i686.manylinux_2_12_i686.whl
tokenizers-1.0.0rc5-cp311-abi3-macosx_11_0_arm64.whl CPython 3.11 abi3 macOS 11.0+ ARM64 Details for tokenizers-1.0.0rc5-cp311-abi3-macosx_11_0_arm64.whl
tokenizers-1.0.0rc5-cp311-abi3-macosx_10_12_x86_64.whl CPython 3.11 abi3 macOS 10.12+ x86-64 Details for tokenizers-1.0.0rc5-cp311-abi3-macosx_10_12_x86_64.whl

Total release size: 45.8 MB

Release files / tokenizers-1.0.0rc5.tar.gz

Download URL tokenizers-1.0.0rc5.tar.gz
Size 473.4 kB
Tags Source
SHA-256 checksum
How to use checksums
fb6e30d77b8b2a8ed9ce02c000e81e5841f2aa9d402c3b6019442ec70951e4cc
BLAKE2b-256 checksum
How to use checksums
5693ac5b5c177eaeda69d2d31e7acf8006d4022ad7da4729958ca1851120c3da
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 20, 2026.

Transparency log

Release files / tokenizers-1.0.0rc5-cp314-cp314t-win_amd64.whl

Download URL tokenizers-1.0.0rc5-cp314-cp314t-win_amd64.whl
Size 1.8 MB
Tags CPython 3.14 CPython 3.14 free-threading Windows x86-64
SHA-256 checksum
How to use checksums
60da685a57087da42b8975d8fa06c3cece017d592b33cb741d5f3ebbdd295965
BLAKE2b-256 checksum
How to use checksums
92ca9bd2cdb913337f8481c52f67199d19f63ded0c3a20c1b8b62c2c01b7ce5e
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 20, 2026.

Transparency log

Release files / tokenizers-1.0.0rc5-cp314-cp314t-musllinux_1_2_x86_64.whl

Download URL tokenizers-1.0.0rc5-cp314-cp314t-musllinux_1_2_x86_64.whl
Size 2.2 MB
Tags CPython 3.14 CPython 3.14 free-threading Linux musl 1.2+ x86-64
SHA-256 checksum
How to use checksums
da0c339102d24bdc0f2a1b9d138b658d2e1c1068683ba5163bfe1aee643b51b9
BLAKE2b-256 checksum
How to use checksums
832a5f50989f150764c85152d6926e442ef49860627cb9df6bd7bd5bcfb1ec0c
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 20, 2026.

Transparency log

Release files / tokenizers-1.0.0rc5-cp314-cp314t-musllinux_1_2_aarch64.whl

Download URL tokenizers-1.0.0rc5-cp314-cp314t-musllinux_1_2_aarch64.whl
Size 2.1 MB
Tags CPython 3.14 CPython 3.14 free-threading Linux musl 1.2+ ARM64
SHA-256 checksum
How to use checksums
05051df7a2a312d652f3a9a40a705c946b78df29b9130b1a71a31df956f088f7
BLAKE2b-256 checksum
How to use checksums
dc5849905cbd062b0eb455172a8a837ad0c7dc0ea1ac21a318971ac37da28d4a
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 20, 2026.

Transparency log

Release files / tokenizers-1.0.0rc5-cp314-cp314t-manylinux2014_x86_64.manylinux_2_17_x86_64.whl

Download URL tokenizers-1.0.0rc5-cp314-cp314t-manylinux2014_x86_64.manylinux_2_17_x86_64.whl
Size 2.0 MB
Tags CPython 3.14 CPython 3.14 free-threading Linux glibc 2.17+ x86-64
SHA-256 checksum
How to use checksums
aeb234fdca831930fce5955feac8135a38bde59c75f0bb5704c4598ec0cf5daf
BLAKE2b-256 checksum
How to use checksums
a18ca354bc18ddea381747733b1485dd71a49a2da51539de1c97f6cd6746584c
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 20, 2026.

Transparency log

Release files / tokenizers-1.0.0rc5-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.whl

Download URL tokenizers-1.0.0rc5-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.whl
Size 1.9 MB
Tags CPython 3.14 CPython 3.14 free-threading Linux glibc 2.17+ ARM64
SHA-256 checksum
How to use checksums
660b11f0715f46b3d5c58f0655f3b3db2231b3ffc559853b8d48892889ab92fd
BLAKE2b-256 checksum
How to use checksums
43d4bb77b0113ac21b0329ecdd7a55c851e3d87b1e19961bb79f10183352e56e
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 20, 2026.

Transparency log

Release files / tokenizers-1.0.0rc5-cp314-cp314t-macosx_11_0_arm64.whl

Download URL tokenizers-1.0.0rc5-cp314-cp314t-macosx_11_0_arm64.whl
Size 1.8 MB
Tags CPython 3.14 CPython 3.14 free-threading macOS 11.0+ ARM64
SHA-256 checksum
How to use checksums
b889bf50d1f9cd893a103977da000c8f93a0f092218ee29b31708fef39943c61
BLAKE2b-256 checksum
How to use checksums
e64f8b8159f49d84a2894225a9a8dee0818cf873560fe5313f48b616285da47f
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 20, 2026.

Transparency log

Release files / tokenizers-1.0.0rc5-cp314-cp314t-macosx_10_12_x86_64.whl

Download URL tokenizers-1.0.0rc5-cp314-cp314t-macosx_10_12_x86_64.whl
Size 1.9 MB
Tags CPython 3.14 CPython 3.14 free-threading macOS 10.12+ x86-64
SHA-256 checksum
How to use checksums
d2e0fa78af07c67dd59b44b302ffa3768197d7556a7f6ba6a82e28c559a25a8c
BLAKE2b-256 checksum
How to use checksums
c77d05c6eab87e6e1b53d2f9b5c2ce9ffc019ba2b7f1b0aad96bc8d04be09d14
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 20, 2026.

Transparency log

Release files / tokenizers-1.0.0rc5-cp311-abi3-win_arm64.whl

Download URL tokenizers-1.0.0rc5-cp311-abi3-win_arm64.whl
Size 1.7 MB
Tags CPython 3.11 Windows ARM64 abi3
SHA-256 checksum
How to use checksums
2e57a1f2040e0ee87372f51187a3fc1878b48bd5f8f694c9e8419d144681c379
BLAKE2b-256 checksum
How to use checksums
b4537ed7b4b23a7c4b1c68af054be3ac980827c51a146ac426395c603ea7ad3f
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 20, 2026.

Transparency log

Release files / tokenizers-1.0.0rc5-cp311-abi3-win_amd64.whl

Download URL tokenizers-1.0.0rc5-cp311-abi3-win_amd64.whl
Size 1.8 MB
Tags CPython 3.11 Windows x86-64 abi3
SHA-256 checksum
How to use checksums
b953761b0bb18062a7074c765ff63f578b9c9ce6ec2d86023efac1b8f68ccc22
BLAKE2b-256 checksum
How to use checksums
4ee3f4f92de5799efdac5a8354d4baf4fbbe9d050063a640bfc093de2102d4ee
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 20, 2026.

Transparency log

Release files / tokenizers-1.0.0rc5-cp311-abi3-win32.whl

Download URL tokenizers-1.0.0rc5-cp311-abi3-win32.whl
Size 1.7 MB
Tags CPython 3.11 Windows x86-32 abi3
SHA-256 checksum
How to use checksums
adf6ce5659f26c1f283d9d0c68952e7b4130c75296aedc35edc230d54b549765
BLAKE2b-256 checksum
How to use checksums
2d90b525df396e97dc5291bfe949770a9c32d976fbb88bf66c69ef0ed28142bb
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 20, 2026.

Transparency log

Release files / tokenizers-1.0.0rc5-cp311-abi3-musllinux_1_2_x86_64.whl

Download URL tokenizers-1.0.0rc5-cp311-abi3-musllinux_1_2_x86_64.whl
Size 2.2 MB
Tags CPython 3.11 Linux musl 1.2+ x86-64 abi3
SHA-256 checksum
How to use checksums
a20086309c2004b3230409ca20894e3082b3dcb606bec88a9d9bb0b6ce7ef44e
BLAKE2b-256 checksum
How to use checksums
724400526e446888b28eef55049133ac52cace30ddbfddd360f7f3a36e745598
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 20, 2026.

Transparency log

Release files / tokenizers-1.0.0rc5-cp311-abi3-musllinux_1_2_i686.whl

Download URL tokenizers-1.0.0rc5-cp311-abi3-musllinux_1_2_i686.whl
Size 2.2 MB
Tags CPython 3.11 Linux musl 1.2+ x86-32 abi3
SHA-256 checksum
How to use checksums
747261784c0e91df82f5301c08947558ff779e898d19de2086a05c93d7876130
BLAKE2b-256 checksum
How to use checksums
c61a735b999f461b2ea9a69bca05063caf6e9515ce2b9c47db0165215a8d06f6
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 20, 2026.

Transparency log

Release files / tokenizers-1.0.0rc5-cp311-abi3-musllinux_1_2_armv7l.whl

Download URL tokenizers-1.0.0rc5-cp311-abi3-musllinux_1_2_armv7l.whl
Size 2.2 MB
Tags CPython 3.11 Linux musl 1.2+ ARMv7l abi3
SHA-256 checksum
How to use checksums
2333d9801f178ab8fdbc6c0967182bbd49024566dfd77903b9c2caa2487d7f46
BLAKE2b-256 checksum
How to use checksums
2d1978d01190da34f431ea0bea426f3c1a2f6cdf64a0804d99ba124cbd629ecc
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 20, 2026.

Transparency log

Release files / tokenizers-1.0.0rc5-cp311-abi3-musllinux_1_2_aarch64.whl

Download URL tokenizers-1.0.0rc5-cp311-abi3-musllinux_1_2_aarch64.whl
Size 2.1 MB
Tags CPython 3.11 Linux musl 1.2+ ARM64 abi3
SHA-256 checksum
How to use checksums
bdfe33571db4c20b11c56b2c619873b6da3cd30ee80ecd438a7261b105749c71
BLAKE2b-256 checksum
How to use checksums
f20a5492baf9d29214f65d5fe5c72f201f6203dcb063d43009cdf44f4f655e83
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 20, 2026.

Transparency log

Release files / tokenizers-1.0.0rc5-cp311-abi3-manylinux_2_31_riscv64.whl

Download URL tokenizers-1.0.0rc5-cp311-abi3-manylinux_2_31_riscv64.whl
Size 2.0 MB
Tags CPython 3.11 Linux glibc 2.31+ RISC-V 64 abi3
SHA-256 checksum
How to use checksums
f5f6308b4f49fc1751d5d2b58fbe3f951b44309b55242ddf2fa1c5bc0e0ae35d
BLAKE2b-256 checksum
How to use checksums
77478ebfdcd9dea206b820716a590138179deaa47ad61fc3731221a9f7c16539
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 20, 2026.

Transparency log

Release files / tokenizers-1.0.0rc5-cp311-abi3-manylinux2014_x86_64.manylinux_2_17_x86_64.whl

Download URL tokenizers-1.0.0rc5-cp311-abi3-manylinux2014_x86_64.manylinux_2_17_x86_64.whl
Size 2.0 MB
Tags CPython 3.11 Linux glibc 2.17+ x86-64 abi3
SHA-256 checksum
How to use checksums
2c2ff15cbca2eef9219ce215aaa51f6969b4a68b267030a0d0195af391e31c7a
BLAKE2b-256 checksum
How to use checksums
465c9a994e4d6a42da0f52fd02a257f128414acb2276e32a5ff48c14199ae823
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 20, 2026.

Transparency log

Release files / tokenizers-1.0.0rc5-cp311-abi3-manylinux2014_s390x.manylinux_2_17_s390x.whl

Download URL tokenizers-1.0.0rc5-cp311-abi3-manylinux2014_s390x.manylinux_2_17_s390x.whl
Size 2.1 MB
Tags CPython 3.11 Linux glibc 2.17+ IBM System/390x abi3
SHA-256 checksum
How to use checksums
7bd25daef7384ccd4dc0bd7daebdf85f97e52415b1a44daaef3e645efed62aa8
BLAKE2b-256 checksum
How to use checksums
87bc9a49229b9e428fa5c9b30ecbdc6342510f99ffefc8f771e5a90b429f3535
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 20, 2026.

Transparency log

Release files / tokenizers-1.0.0rc5-cp311-abi3-manylinux2014_ppc64le.manylinux_2_17_ppc64le.whl

Download URL tokenizers-1.0.0rc5-cp311-abi3-manylinux2014_ppc64le.manylinux_2_17_ppc64le.whl
Size 2.1 MB
Tags CPython 3.11 Linux glibc 2.17+ PowerPC 64-le abi3
SHA-256 checksum
How to use checksums
af37930aeec644b140c2fd03db79505b41a5969ca359aec45b04b88fd6343dd0
BLAKE2b-256 checksum
How to use checksums
751c0815c31a738e6686dd1ad02f686abcecc1bc429e6243cd65217c198cb0f6
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 20, 2026.

Transparency log

Release files / tokenizers-1.0.0rc5-cp311-abi3-manylinux2014_armv7l.manylinux_2_17_armv7l.whl

Download URL tokenizers-1.0.0rc5-cp311-abi3-manylinux2014_armv7l.manylinux_2_17_armv7l.whl
Size 1.9 MB
Tags CPython 3.11 Linux glibc 2.17+ ARMv7l abi3
SHA-256 checksum
How to use checksums
9666181150851269a63349a6dffa5690ff828a4f9c4208d3a620feee96bb029c
BLAKE2b-256 checksum
How to use checksums
6637d49f656c7d0d7fc006d5a7bf65d5734ba1533ae3c96e727327a0a8d63aed
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 20, 2026.

Transparency log

Release files / tokenizers-1.0.0rc5-cp311-abi3-manylinux2014_aarch64.manylinux_2_17_aarch64.whl

Download URL tokenizers-1.0.0rc5-cp311-abi3-manylinux2014_aarch64.manylinux_2_17_aarch64.whl
Size 1.9 MB
Tags CPython 3.11 Linux glibc 2.17+ ARM64 abi3
SHA-256 checksum
How to use checksums
c86a2153ff600fc5ef37e4c42763fea83474b89969adaba676e94e4c6457ef1d
BLAKE2b-256 checksum
How to use checksums
7e2e69e97da75141d2adce910a3e4f306141febe88ef6238da762588685744f5
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 20, 2026.

Transparency log

Release files / tokenizers-1.0.0rc5-cp311-abi3-manylinux2010_i686.manylinux_2_12_i686.whl

Download URL tokenizers-1.0.0rc5-cp311-abi3-manylinux2010_i686.manylinux_2_12_i686.whl
Size 2.0 MB
Tags CPython 3.11 Linux glibc 2.12+ x86-32 abi3
SHA-256 checksum
How to use checksums
b2a8437d3b7edb7a2d28c9d6b348141630b99d8fae373a6c6a56a66c659695aa
BLAKE2b-256 checksum
How to use checksums
51d56d5a0c906d7cceff5c8e22007942f754de73717fcaade47d638f3742f9fe
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 20, 2026.

Transparency log

Release files / tokenizers-1.0.0rc5-cp311-abi3-macosx_11_0_arm64.whl

Download URL tokenizers-1.0.0rc5-cp311-abi3-macosx_11_0_arm64.whl
Size 1.8 MB
Tags CPython 3.11 abi3 macOS 11.0+ ARM64
SHA-256 checksum
How to use checksums
c6d861ee0d7d9605fb1fbb76ddd915b4a790b991d011943cd9ca4fba9f34a1a5
BLAKE2b-256 checksum
How to use checksums
82c16a29e93379eb15aea7ad2b6a111938dce524bb78fcb086e1d4989ece61b2
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 20, 2026.

Transparency log

Release files / tokenizers-1.0.0rc5-cp311-abi3-macosx_10_12_x86_64.whl

Download URL tokenizers-1.0.0rc5-cp311-abi3-macosx_10_12_x86_64.whl
Size 1.9 MB
Tags CPython 3.11 abi3 macOS 10.12+ x86-64
SHA-256 checksum
How to use checksums
1ef7781876ac7b100c158c88dc7d288b04c988e0d75529b34a0cd588f00d0184
BLAKE2b-256 checksum
How to use checksums
b1006ac4b44896277b2f37393c67483083a4b024773e9bf6c83e41d686cd1943
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 20, 2026.

Transparency log

Security for tokenizers 1.0.0rc5

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

All 24 files in this release have been uploaded with provenance metadata.

Select from 24 files

Attestation for tokenizers-1.0.0rc5.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 20, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release
Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page