This release is a pre-release and may not be stable for production use.
tokenizers Python bindings
This module is still experimental, expect the API to change or break. If used in a production setting, please pin the version number.
Install
pip install --pre tokenizers
Usage
from tokenizers import Padding, Tokenizer
# A `Tokenizer` represents a tokenizer's spec (json) and can be edited.
# You can load it from the JSON file
tokenizer = Tokenizer.from_file("tokenizer.json")
# Or load the `tokenizer.json` of a model repo on the Hub.
# Downloads go through the `huggingface_hub` package
tokenizer = Tokenizer.from_pretrained("openai-community/gpt2", revision="main")
# `compile()` compiles the specification to an optimized tokenizer in rust.
# The resulting `CompiledTokenizer` object can encode text to token ids and decode ids back to text.
# It's immutable and can be shared among threads.
compiled = tokenizer.compile()
encoding = compiled.encode("Hello there, how are you?")
encoding.ids # [101, 7592, 2045, 1010, 2129, 2024, 2017, 1029, 102]
encoding.type_ids # [0, 0, 0, 0, 0, 0, 0, 0, 0]
encoding.attention_mask # [1, 1, 1, 1, 1, 1, 1, 1, 1]
# The same fields as numpy arrays. Each read returns a *new copy* that the caller owns.
encoding.ids_array # array([101, 7592, 2045, 1010, 2129, 2024, 2017, 1029, 102], dtype=uint32)
encoding.type_ids_array # array([0, 0, 0, 0, 0, 0, 0, 0, 0], dtype=uint32)
encoding.attention_mask_array # array([1, 1, 1, 1, 1, 1, 1, 1, 1], dtype=uint32)
torch.from_numpy(encoding.ids_array) # no-copy translation
compiled.encode("Hello", add_special_tokens=False).ids # no [CLS]/[SEP]
compiled.decode(encoding.ids) # "hello there, how are you?"
compiled.decode(encoding.ids_array) # a numpy array works too
compiled.decode(encoding.ids, skip_special_tokens=False)
# Pad one call. `length=None` pads each batch to its longest item.
short, long = compiled.encode_batch(
["Hello", "Hello there, how are you?"],
padding=Padding(direction="left", pad_id=0, pad_token="[PAD]"),
)
len(short) == len(long) # True
short.attention_mask # [0, 0, 0, 0, 0, 0, 1, 1, 1]
compiled.encode("Hello", padding=Padding(length=16)) # exactly 16 ids
# Padding and truncation can also be set on the spec.
# `padding=None` switches the default off for one call.
tokenizer.padding = Padding(pad_to_multiple_of=8)
padded = tokenizer.compile()
padded.encode("Hello", padding=None)
Edit and save a tokenizer
from tokenizers import AddedToken, normalizers
# Set or replace a component. Reading `tokenizer.normalizer` returns a live view, not a copy.
tokenizer.normalizer = normalizers.Sequence([normalizers.NFC(), normalizers.Lowercase()])
# Added tokens are matched in the text before the model runs.
tokenizer.add_special_tokens(["<|im_end|>"])
tokenizer.add_tokens([AddedToken("<think>", single_word=True)])
tokenizer.save("tokenizer.json")
compiled = tokenizer.compile() # compile again to encode with the edits
compiled.token_to_id("<think>") # a new token gets the next free id
compiled.get_vocab_size() # counts the added tokens too
examples/composition.py builds a tokenizer from a model, then edits the model after the tokenizer holds it.
Decode a stream of ids
DecodeStream turns ids into text as they arrive, such as a model's generated tokens.
step returns None while the text ends with an incomplete character.
from tokenizers.decoders import DecodeStream
stream = DecodeStream(skip_special_tokens=True)
for id in generated_ids:
print(stream.step(compiled, id) or "", end="")
examples/ holds a runnable script for each of these; make examples runs them all.
multiprocessing and Worker processes
Every class here can be pickled, so a tokenizer can be sent to a worker process and its encodings
sent back: multiprocessing, a PyTorch DataLoader, datasets.map(num_proc=...).
A pickled CompiledTokenizer carries its whole tokenizer.json, and unpickling compiles it again.
With the "spawn" start method, we recommend compiling the tokenizer once per worker to avoid unpickle compilation overhead.
def compile_tokenizer(tokenizer):
global compiled
compiled = tokenizer.compile()
def encode(text):
return compiled.encode(text)
with multiprocessing.get_context("spawn").Pool(4, initializer=compile_tokenizer, initargs=(tokenizer,)) as pool:
encoded = list(pool.imap(encode, texts, chunksize=32))
With the "fork" start method, compiling before starting the pool is recommended: the workers inherit the compiled tokenizer
and nothing is pickled.
Developers
make developregenerates the.pyistub, rebuilds the extension (debug) and reinstalls it in editable mode.make testdoes the above, then runstests/with pytest.make examplesdoes the above, then runs every script inexamples/.make stubsregenerates the.pyistubs inpython/tokenizers/(tokenizers.pyi,models.pyi,normalizers.pyi, ...), including the memberspython/tokenizers/__init__.pyattaches to the Rust classes.make style/make check-styleregenerate the stub, then format/lint the Rust and Python sides and type-check withty. CI runsmake check-styleand fails if the regenerated stub differs from the committed one.
4 maintainers
Metadata
4 maintainers
Release files for tokenizers 1.0.0rc4
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| tokenizers-1.0.0rc4.tar.gz | 521.1 kB | With provenance | Details for tokenizers-1.0.0rc4.tar.gz |
Built distributions (wheels)
Total release size: 45.9 MB
Release files / tokenizers-1.0.0rc4.tar.gz
| Download URL | tokenizers-1.0.0rc4.tar.gz |
|---|---|
| Size | 521.1 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
7a21543b388e35b5ecd395a1177ed5c550cb5820717e0966de9542bfe6ca1fd4
|
|
BLAKE2b-256 checksum How to use checksums |
c536da1061089b1f3501ba3b9baec4afc0394752c00f17c641bcdd2429b1edcd
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 9, 2026.
Transparency logRelease files / tokenizers-1.0.0rc4-cp314-cp314t-win_amd64.whl
| Download URL | tokenizers-1.0.0rc4-cp314-cp314t-win_amd64.whl |
|---|---|
| Size | 1.8 MB |
| Tags | CPython 3.14 CPython 3.14 free-threading Windows x86-64 |
|
SHA-256 checksum How to use checksums |
21af7e39baf55720bfbabbad66e6f3708eb5fb80ab529ba531d38fad03cd7e5e
|
|
BLAKE2b-256 checksum How to use checksums |
ae10dc3389119acc41d6c906bb1c40d6292065050c9bbb3c5565b4d4cdea56ff
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 9, 2026.
Transparency logRelease files / tokenizers-1.0.0rc4-cp314-cp314t-musllinux_1_2_x86_64.whl
| Download URL | tokenizers-1.0.0rc4-cp314-cp314t-musllinux_1_2_x86_64.whl |
|---|---|
| Size | 2.2 MB |
| Tags | CPython 3.14 CPython 3.14 free-threading Linux musl 1.2+ x86-64 |
|
SHA-256 checksum How to use checksums |
509c9c84a88f3acd16c3dfe04781865866c4ef6e98b400daba2a6df290277a36
|
|
BLAKE2b-256 checksum How to use checksums |
4dfb93a3508d8fc8a4902a04c9f89db68c392d2920968ea3f61175cc8df8e84e
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 9, 2026.
Transparency logRelease files / tokenizers-1.0.0rc4-cp314-cp314t-musllinux_1_2_aarch64.whl
| Download URL | tokenizers-1.0.0rc4-cp314-cp314t-musllinux_1_2_aarch64.whl |
|---|---|
| Size | 2.1 MB |
| Tags | CPython 3.14 CPython 3.14 free-threading Linux musl 1.2+ ARM64 |
|
SHA-256 checksum How to use checksums |
fa28d63391dd8bb6c0449d928ac80b2e97089ec19d002415d479200e791d6fc7
|
|
BLAKE2b-256 checksum How to use checksums |
4eab00faf62562baeacf22c70c2697dae359ec4152456811fcbd525648d51a34
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 9, 2026.
Transparency logRelease files / tokenizers-1.0.0rc4-cp314-cp314t-manylinux2014_x86_64.manylinux_2_17_x86_64.whl
| Download URL | tokenizers-1.0.0rc4-cp314-cp314t-manylinux2014_x86_64.manylinux_2_17_x86_64.whl |
|---|---|
| Size | 2.0 MB |
| Tags | CPython 3.14 CPython 3.14 free-threading Linux glibc 2.17+ x86-64 |
|
SHA-256 checksum How to use checksums |
d0dc72ccb68e474c494e7f6ea6d23697a40763be1459eef3c20cc98251ec19dd
|
|
BLAKE2b-256 checksum How to use checksums |
e7ed788cf1ef1e3b73b35bd34e3343ef78b164a2bb0edba9e5342ba434ec9b15
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 9, 2026.
Transparency logRelease files / tokenizers-1.0.0rc4-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.whl
| Download URL | tokenizers-1.0.0rc4-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.whl |
|---|---|
| Size | 1.9 MB |
| Tags | CPython 3.14 CPython 3.14 free-threading Linux glibc 2.17+ ARM64 |
|
SHA-256 checksum How to use checksums |
9366f50c24eba33dfbaeddcb6309db45c9a1898805706db911441a56b6f14d9f
|
|
BLAKE2b-256 checksum How to use checksums |
b9c3ace90d1d70ae126aa47b49cebe7ab62f4b307488ef0d05f4dbb15695a95b
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 9, 2026.
Transparency logRelease files / tokenizers-1.0.0rc4-cp314-cp314t-macosx_11_0_arm64.whl
| Download URL | tokenizers-1.0.0rc4-cp314-cp314t-macosx_11_0_arm64.whl |
|---|---|
| Size | 1.8 MB |
| Tags | CPython 3.14 CPython 3.14 free-threading macOS 11.0+ ARM64 |
|
SHA-256 checksum How to use checksums |
db015b12fd5d6a082224b59c456de38b8cb3b27282480c5f01caa2280218e224
|
|
BLAKE2b-256 checksum How to use checksums |
cd0383984cb12bff9a069ae4f1646d09b6344d0722409f4be5bc81849db73500
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 9, 2026.
Transparency logRelease files / tokenizers-1.0.0rc4-cp314-cp314t-macosx_10_12_x86_64.whl
| Download URL | tokenizers-1.0.0rc4-cp314-cp314t-macosx_10_12_x86_64.whl |
|---|---|
| Size | 1.9 MB |
| Tags | CPython 3.14 CPython 3.14 free-threading macOS 10.12+ x86-64 |
|
SHA-256 checksum How to use checksums |
8ffbb5b57fc52cb68fd3553bacf7f2836501c49a3d45da33e10caf094ff554b4
|
|
BLAKE2b-256 checksum How to use checksums |
aa83ef047c1842b86efa03c50bce570b28d3f42b3dd4734c3258e7a034536e5c
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 9, 2026.
Transparency logRelease files / tokenizers-1.0.0rc4-cp310-abi3-win_arm64.whl
| Download URL | tokenizers-1.0.0rc4-cp310-abi3-win_arm64.whl |
|---|---|
| Size | 1.7 MB |
| Tags | CPython 3.10 Windows ARM64 abi3 |
|
SHA-256 checksum How to use checksums |
85d79df8d5a299d84c9fd003315fa580cdfff6d777364d1780f8b4278f26d5be
|
|
BLAKE2b-256 checksum How to use checksums |
7fd24249ba0f4dabe98ed4e6dcb19ade382cf8459a95ee6547917df63f70745d
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 9, 2026.
Transparency logRelease files / tokenizers-1.0.0rc4-cp310-abi3-win_amd64.whl
| Download URL | tokenizers-1.0.0rc4-cp310-abi3-win_amd64.whl |
|---|---|
| Size | 1.8 MB |
| Tags | CPython 3.10 Windows x86-64 abi3 |
|
SHA-256 checksum How to use checksums |
f0ccfeb4713f73d9ebc3bed2ac8d8f764738df87e49ea4cda0f152bd03725310
|
|
BLAKE2b-256 checksum How to use checksums |
be1b08a2c9cc15cc6ec5c04ee5b025d9f063f8672b189cd25964ef8dc1eb7b3b
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 9, 2026.
Transparency logRelease files / tokenizers-1.0.0rc4-cp310-abi3-win32.whl
| Download URL | tokenizers-1.0.0rc4-cp310-abi3-win32.whl |
|---|---|
| Size | 1.7 MB |
| Tags | CPython 3.10 Windows x86-32 abi3 |
|
SHA-256 checksum How to use checksums |
7f48cc6581129581e80ad48053b5b3bb077bb021a89d17f11864a558f84683a4
|
|
BLAKE2b-256 checksum How to use checksums |
7e071dce0ddc558605f2a2d23fcdf065e5c4d73253190ba1ae038f0e3f79f0cd
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 9, 2026.
Transparency logRelease files / tokenizers-1.0.0rc4-cp310-abi3-musllinux_1_2_x86_64.whl
| Download URL | tokenizers-1.0.0rc4-cp310-abi3-musllinux_1_2_x86_64.whl |
|---|---|
| Size | 2.2 MB |
| Tags | CPython 3.10 Linux musl 1.2+ x86-64 abi3 |
|
SHA-256 checksum How to use checksums |
f7244242fdc57b82500f3298e9afe6a936281d31c856a9ff33acb3b9b4ece2df
|
|
BLAKE2b-256 checksum How to use checksums |
5429a04957eaae31008afac7ef3456205b926667215df032172e1abe40ed5597
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 9, 2026.
Transparency logRelease files / tokenizers-1.0.0rc4-cp310-abi3-musllinux_1_2_i686.whl
| Download URL | tokenizers-1.0.0rc4-cp310-abi3-musllinux_1_2_i686.whl |
|---|---|
| Size | 2.2 MB |
| Tags | CPython 3.10 Linux musl 1.2+ x86-32 abi3 |
|
SHA-256 checksum How to use checksums |
19082adca605b42eba4951f07650f3dd073cfcee8f5112fbaedcdd98ad8a4216
|
|
BLAKE2b-256 checksum How to use checksums |
5a1a8d105d6de009a2d41f1a897f8a65d7478ba2ecaad38dc29e997db3e5e274
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 9, 2026.
Transparency logRelease files / tokenizers-1.0.0rc4-cp310-abi3-musllinux_1_2_armv7l.whl
| Download URL | tokenizers-1.0.0rc4-cp310-abi3-musllinux_1_2_armv7l.whl |
|---|---|
| Size | 2.2 MB |
| Tags | CPython 3.10 Linux musl 1.2+ ARMv7l abi3 |
|
SHA-256 checksum How to use checksums |
6197947e02d63d573cf54d6b352bc9d8deee6a3837e5b3dfca17ce20c1466009
|
|
BLAKE2b-256 checksum How to use checksums |
8e2a976f29ad7f1ee2593685dbd3a5fed0391b4ee65b236202dfb68b0b8b6533
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 9, 2026.
Transparency logRelease files / tokenizers-1.0.0rc4-cp310-abi3-musllinux_1_2_aarch64.whl
| Download URL | tokenizers-1.0.0rc4-cp310-abi3-musllinux_1_2_aarch64.whl |
|---|---|
| Size | 2.1 MB |
| Tags | CPython 3.10 Linux musl 1.2+ ARM64 abi3 |
|
SHA-256 checksum How to use checksums |
7c86dcc06fbb9ee33712ce768bc4855974c7532832207c58051da66df18616a3
|
|
BLAKE2b-256 checksum How to use checksums |
79ce6aba66f81b1a2d89219446169aa191ca2f62f75179c7d4af4879bf0ab7e6
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 9, 2026.
Transparency logRelease files / tokenizers-1.0.0rc4-cp310-abi3-manylinux_2_31_riscv64.whl
| Download URL | tokenizers-1.0.0rc4-cp310-abi3-manylinux_2_31_riscv64.whl |
|---|---|
| Size | 2.0 MB |
| Tags | CPython 3.10 Linux glibc 2.31+ RISC-V 64 abi3 |
|
SHA-256 checksum How to use checksums |
9c530874528db232efabb19aa18f8ec27a9c333d8874232327f859bea701b4e8
|
|
BLAKE2b-256 checksum How to use checksums |
20fe4e71ce56b4c9b608cbb893329e987a7417adc6ce1313187c9dceec339a49
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 9, 2026.
Transparency logRelease files / tokenizers-1.0.0rc4-cp310-abi3-manylinux2014_x86_64.manylinux_2_17_x86_64.whl
| Download URL | tokenizers-1.0.0rc4-cp310-abi3-manylinux2014_x86_64.manylinux_2_17_x86_64.whl |
|---|---|
| Size | 2.0 MB |
| Tags | CPython 3.10 Linux glibc 2.17+ x86-64 abi3 |
|
SHA-256 checksum How to use checksums |
ecea8dc0d5d08a9e855931989ca60591f7aa2c16d7573862635c15a3c132b332
|
|
BLAKE2b-256 checksum How to use checksums |
b7b1d5e098ec361bb9fb321d83de3be9b924299b5f974a6bbe01495175e05985
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 9, 2026.
Transparency logRelease files / tokenizers-1.0.0rc4-cp310-abi3-manylinux2014_s390x.manylinux_2_17_s390x.whl
| Download URL | tokenizers-1.0.0rc4-cp310-abi3-manylinux2014_s390x.manylinux_2_17_s390x.whl |
|---|---|
| Size | 2.1 MB |
| Tags | CPython 3.10 Linux glibc 2.17+ IBM System/390x abi3 |
|
SHA-256 checksum How to use checksums |
b44672f36c75b0e2af5ec03fd3b632e51fb90173c889f357e2d464559c87f4cc
|
|
BLAKE2b-256 checksum How to use checksums |
240717c9db989d477c2d49ccf0915e5f3c8dfb815e9cae2f57af389b4fd6fd51
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 9, 2026.
Transparency logRelease files / tokenizers-1.0.0rc4-cp310-abi3-manylinux2014_ppc64le.manylinux_2_17_ppc64le.whl
| Download URL | tokenizers-1.0.0rc4-cp310-abi3-manylinux2014_ppc64le.manylinux_2_17_ppc64le.whl |
|---|---|
| Size | 2.1 MB |
| Tags | CPython 3.10 Linux glibc 2.17+ PowerPC 64-le abi3 |
|
SHA-256 checksum How to use checksums |
6578aa4c450e784ca5ff017ab3fe4b2f850029ee76cd704f81d23dc5c9d1829e
|
|
BLAKE2b-256 checksum How to use checksums |
89fb34c58a1beab17761365f31f0b510392bca9b67c2848922a61e66a7984b63
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 9, 2026.
Transparency logRelease files / tokenizers-1.0.0rc4-cp310-abi3-manylinux2014_armv7l.manylinux_2_17_armv7l.whl
| Download URL | tokenizers-1.0.0rc4-cp310-abi3-manylinux2014_armv7l.manylinux_2_17_armv7l.whl |
|---|---|
| Size | 1.9 MB |
| Tags | CPython 3.10 Linux glibc 2.17+ ARMv7l abi3 |
|
SHA-256 checksum How to use checksums |
44cadd86d5c50b821d4ea993230820487d8db661123a2d69ef925d8e9b09da94
|
|
BLAKE2b-256 checksum How to use checksums |
4a7d3cd315ebe30441a2ba96f1e4d84157931d069708aaabaddac84aa8726165
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 9, 2026.
Transparency logRelease files / tokenizers-1.0.0rc4-cp310-abi3-manylinux2014_aarch64.manylinux_2_17_aarch64.whl
| Download URL | tokenizers-1.0.0rc4-cp310-abi3-manylinux2014_aarch64.manylinux_2_17_aarch64.whl |
|---|---|
| Size | 1.9 MB |
| Tags | CPython 3.10 Linux glibc 2.17+ ARM64 abi3 |
|
SHA-256 checksum How to use checksums |
d56b835638561c40dda2d3fa939836e38f9494d2bff1d052a44355adf5fd3b9d
|
|
BLAKE2b-256 checksum How to use checksums |
ceab5559ab02f0489907e00fa1b7147df8d8e16d3e5cc68104386a15cf9bf690
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 9, 2026.
Transparency logRelease files / tokenizers-1.0.0rc4-cp310-abi3-manylinux2010_i686.manylinux_2_12_i686.whl
| Download URL | tokenizers-1.0.0rc4-cp310-abi3-manylinux2010_i686.manylinux_2_12_i686.whl |
|---|---|
| Size | 2.0 MB |
| Tags | CPython 3.10 Linux glibc 2.12+ x86-32 abi3 |
|
SHA-256 checksum How to use checksums |
2bbf963fc545ae55f4e6e72b5bcda493463c13d9e914576e9903e66342d6ee0a
|
|
BLAKE2b-256 checksum How to use checksums |
1777859d18b92ace6fa5abfd0300bd261f23a347ee082eb38795754e3bf28d95
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 9, 2026.
Transparency logRelease files / tokenizers-1.0.0rc4-cp310-abi3-macosx_11_0_arm64.whl
| Download URL | tokenizers-1.0.0rc4-cp310-abi3-macosx_11_0_arm64.whl |
|---|---|
| Size | 1.8 MB |
| Tags | CPython 3.10 abi3 macOS 11.0+ ARM64 |
|
SHA-256 checksum How to use checksums |
fe236f28c131d8b6971054fc95db308765f4987f69da2433170698bc30010e37
|
|
BLAKE2b-256 checksum How to use checksums |
3fdf87d8cb925ab8579b2a57687efb97fa776df21382df9ae850526dfef5832a
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 9, 2026.
Transparency logRelease files / tokenizers-1.0.0rc4-cp310-abi3-macosx_10_12_x86_64.whl
| Download URL | tokenizers-1.0.0rc4-cp310-abi3-macosx_10_12_x86_64.whl |
|---|---|
| Size | 1.9 MB |
| Tags | CPython 3.10 abi3 macOS 10.12+ x86-64 |
|
SHA-256 checksum How to use checksums |
3a5c158291d37fcb3151d2d0ccf0639b4997c54b77ae14083c289d47ad4ff31d
|
|
BLAKE2b-256 checksum How to use checksums |
b480b8016cbde85715df8f6b46b4f35d1fc81eeb8a2c9c3a1c6d0b8bf4e7f3f7
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 9, 2026.
Transparency logSecurity for tokenizers 1.0.0rc4
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
Full provenance
All 24 files in this release have been uploaded with provenance metadata.
Attestation for tokenizers-1.0.0rc4.tar.gz
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 9, 2026.
Transparency logA note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.
