Tokenizers

Provides an implementation of today's most used tokenizers, with a focus on performance and versatility.

Bindings over the Rust implementation. If you are interested in the High-level design, you can go check it there.

Otherwise, let's dive in!

Main features:

Train new vocabularies and tokenize using 4 pre-made tokenizers (Bert WordPiece and the 3 most common BPE versions).
Extremely fast (both training and tokenization), thanks to the Rust implementation. Takes less than 20 seconds to tokenize a GB of text on a server's CPU.
Easy to use, but also extremely versatile.
Designed for research and production.
Normalization comes with alignments tracking. It's always possible to get the part of the original sentence that corresponds to a given token.
Does all the pre-processing: Truncate, Pad, add the special tokens your model needs.

Installation

With pip:

pip install tokenizers

From sources:

To use this method, you need to have the Rust installed:

# Install with:
curl https://sh.rustup.rs -sSf | sh -s -- -y
export PATH="$HOME/.cargo/bin:$PATH"

Once Rust is installed, you can compile doing the following

git clone https://github.com/huggingface/tokenizers
cd tokenizers/bindings/python

# Create a virtual env (you can use yours as well)
python -m venv .env
source .env/bin/activate

# Install `tokenizers` in the current virtual env
pip install -e .

Load a pretrained tokenizer from the Hub

from tokenizers import Tokenizer

tokenizer = Tokenizer.from_pretrained("bert-base-cased")

Using the provided Tokenizers

We provide some pre-build tokenizers to cover the most common cases. You can easily load one of these using some vocab.json and merges.txt files:

from tokenizers import CharBPETokenizer

# Initialize a tokenizer
vocab = "./path/to/vocab.json"
merges = "./path/to/merges.txt"
tokenizer = CharBPETokenizer(vocab, merges)

# And then encode:
encoded = tokenizer.encode("I can feel the magic, can you?")
print(encoded.ids)
print(encoded.tokens)

And you can train them just as simply:

from tokenizers import CharBPETokenizer

# Initialize a tokenizer
tokenizer = CharBPETokenizer()

# Then train it!
tokenizer.train([ "./path/to/files/1.txt", "./path/to/files/2.txt" ])

# Now, let's use it:
encoded = tokenizer.encode("I can feel the magic, can you?")

# And finally save it somewhere
tokenizer.save("./path/to/directory/my-bpe.tokenizer.json")

Provided Tokenizers

CharBPETokenizer: The original BPE
ByteLevelBPETokenizer: The byte level version of the BPE
SentencePieceBPETokenizer: A BPE implementation compatible with the one used by SentencePiece
BertWordPieceTokenizer: The famous Bert tokenizer, using WordPiece

All of these can be used and trained as explained above!

Build your own

Whenever these provided tokenizers don't give you enough freedom, you can build your own tokenizer, by putting all the different parts you need together. You can check how we implemented the provided tokenizers and adapt them easily to your own needs.

Building a byte-level BPE

Here is an example showing how to build your own byte-level BPE by putting all the different pieces together, and then saving it to a single file:

from tokenizers import Tokenizer, models, pre_tokenizers, decoders, trainers, processors

# Initialize a tokenizer
tokenizer = Tokenizer(models.BPE())

# Customize pre-tokenization and decoding
tokenizer.pre_tokenizer = pre_tokenizers.ByteLevel(add_prefix_space=True)
tokenizer.decoder = decoders.ByteLevel()
tokenizer.post_processor = processors.ByteLevel(trim_offsets=True)

# And then train
trainer = trainers.BpeTrainer(
    vocab_size=20000,
    min_frequency=2,
    initial_alphabet=pre_tokenizers.ByteLevel.alphabet()
)
tokenizer.train([
    "./path/to/dataset/1.txt",
    "./path/to/dataset/2.txt",
    "./path/to/dataset/3.txt"
], trainer=trainer)

# And Save it
tokenizer.save("byte-level-bpe.tokenizer.json", pretty=True)

Now, when you want to use this tokenizer, this is as simple as:

from tokenizers import Tokenizer

tokenizer = Tokenizer.from_file("byte-level-bpe.tokenizer.json")

encoded = tokenizer.encode("I can feel the magic, can you?")

Metadata

Release files for tokenizers 0.20.1

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for tokenizers 0.20.1
File	Size	Uploaded	Provenance
tokenizers-0.20.1.tar.gz	339.6 kB	Oct 10, 2024	Without provenance	Details

Built distributions (wheels)

Table of built distributions (wheels) for tokenizers 0.20.1
File	Interpreter	ABI	Platform	Reset
tokenizers-0.20.1-pp310-pypy310_pp73-musllinux_1_1_x86_64.whl 9.3 MB Oct 10, 2024 Without provenance	PyPy 3.10	PyPy 3.10 7.3	Linux musl 1.1+ x86-64	Details
tokenizers-0.20.1-pp310-pypy310_pp73-musllinux_1_1_aarch64.whl 9.0 MB Oct 10, 2024 Without provenance	PyPy 3.10	PyPy 3.10 7.3	Linux musl 1.1+ ARM64	Details
tokenizers-0.20.1-pp310-pypy310_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl 3.0 MB Oct 10, 2024 Without provenance	PyPy 3.10	PyPy 3.10 7.3	Linux glibc 2.17+ x86-64	Details
tokenizers-0.20.1-pp310-pypy310_pp73-manylinux_2_17_i686.manylinux2014_i686.whl 3.1 MB Oct 10, 2024 Without provenance	PyPy 3.10	PyPy 3.10 7.3	Linux glibc 2.17+ x86-32	Details
tokenizers-0.20.1-pp310-pypy310_pp73-manylinux_2_17_aarch64.manylinux2014_aarch64.whl 2.9 MB Oct 10, 2024 Without provenance	PyPy 3.10	PyPy 3.10 7.3	Linux glibc 2.17+ ARM64	Details
tokenizers-0.20.1-pp310-pypy310_pp73-macosx_11_0_arm64.whl 2.6 MB Oct 10, 2024 Without provenance	PyPy 3.10	PyPy 3.10 7.3	macOS 11.0+ ARM64	Details
tokenizers-0.20.1-pp310-pypy310_pp73-macosx_10_12_x86_64.whl 2.7 MB Oct 10, 2024 Without provenance	PyPy 3.10	PyPy 3.10 7.3	macOS 10.12+ x86-64	Details
tokenizers-0.20.1-pp39-pypy39_pp73-musllinux_1_1_x86_64.whl 9.3 MB Oct 10, 2024 Without provenance	PyPy 3.9	PyPy 3.9 7.3	Linux musl 1.1+ x86-64	Details
tokenizers-0.20.1-pp39-pypy39_pp73-musllinux_1_1_aarch64.whl 9.0 MB Oct 10, 2024 Without provenance	PyPy 3.9	PyPy 3.9 7.3	Linux musl 1.1+ ARM64	Details
tokenizers-0.20.1-pp39-pypy39_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl 3.0 MB Oct 10, 2024 Without provenance	PyPy 3.9	PyPy 3.9 7.3	Linux glibc 2.17+ x86-64	Details
tokenizers-0.20.1-pp39-pypy39_pp73-manylinux_2_17_i686.manylinux2014_i686.whl 3.1 MB Oct 10, 2024 Without provenance	PyPy 3.9	PyPy 3.9 7.3	Linux glibc 2.17+ x86-32	Details
tokenizers-0.20.1-pp39-pypy39_pp73-manylinux_2_17_aarch64.manylinux2014_aarch64.whl 2.9 MB Oct 10, 2024 Without provenance	PyPy 3.9	PyPy 3.9 7.3	Linux glibc 2.17+ ARM64	Details
tokenizers-0.20.1-pp39-pypy39_pp73-macosx_11_0_arm64.whl 2.6 MB Oct 10, 2024 Without provenance	PyPy 3.9	PyPy 3.9 7.3	macOS 11.0+ ARM64	Details
tokenizers-0.20.1-pp39-pypy39_pp73-macosx_10_12_x86_64.whl 2.7 MB Oct 10, 2024 Without provenance	PyPy 3.9	PyPy 3.9 7.3	macOS 10.12+ x86-64	Details
tokenizers-0.20.1-pp38-pypy38_pp73-musllinux_1_1_x86_64.whl 9.3 MB Oct 10, 2024 Without provenance	PyPy 3.8	PyPy 3.8 7.3	Linux musl 1.1+ x86-64	Details
tokenizers-0.20.1-pp38-pypy38_pp73-musllinux_1_1_aarch64.whl 9.0 MB Oct 10, 2024 Without provenance	PyPy 3.8	PyPy 3.8 7.3	Linux musl 1.1+ ARM64	Details
tokenizers-0.20.1-pp38-pypy38_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl 3.0 MB Oct 10, 2024 Without provenance	PyPy 3.8	PyPy 3.8 7.3	Linux glibc 2.17+ x86-64	Details
tokenizers-0.20.1-pp38-pypy38_pp73-manylinux_2_17_i686.manylinux2014_i686.whl 3.1 MB Oct 10, 2024 Without provenance	PyPy 3.8	PyPy 3.8 7.3	Linux glibc 2.17+ x86-32	Details
tokenizers-0.20.1-pp38-pypy38_pp73-manylinux_2_17_aarch64.manylinux2014_aarch64.whl 2.9 MB Oct 10, 2024 Without provenance	PyPy 3.8	PyPy 3.8 7.3	Linux glibc 2.17+ ARM64	Details
tokenizers-0.20.1-pp38-pypy38_pp73-macosx_11_0_arm64.whl 2.6 MB Oct 10, 2024 Without provenance	PyPy 3.8	PyPy 3.8 7.3	macOS 11.0+ ARM64	Details
tokenizers-0.20.1-pp38-pypy38_pp73-macosx_10_12_x86_64.whl 2.7 MB Oct 10, 2024 Without provenance	PyPy 3.8	PyPy 3.8 7.3	macOS 10.12+ x86-64	Details
tokenizers-0.20.1-pp37-pypy37_pp73-musllinux_1_1_x86_64.whl 9.3 MB Oct 10, 2024 Without provenance	PyPy 3.7	PyPy 3.7 7.3	Linux musl 1.1+ x86-64	Details
tokenizers-0.20.1-pp37-pypy37_pp73-musllinux_1_1_aarch64.whl 9.0 MB Oct 10, 2024 Without provenance	PyPy 3.7	PyPy 3.7 7.3	Linux musl 1.1+ ARM64	Details
tokenizers-0.20.1-pp37-pypy37_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl 3.0 MB Oct 10, 2024 Without provenance	PyPy 3.7	PyPy 3.7 7.3	Linux glibc 2.17+ x86-64	Details
tokenizers-0.20.1-pp37-pypy37_pp73-manylinux_2_17_i686.manylinux2014_i686.whl 3.1 MB Oct 10, 2024 Without provenance	PyPy 3.7	PyPy 3.7 7.3	Linux glibc 2.17+ x86-32	Details
tokenizers-0.20.1-pp37-pypy37_pp73-manylinux_2_17_aarch64.manylinux2014_aarch64.whl 2.9 MB Oct 10, 2024 Without provenance	PyPy 3.7	PyPy 3.7 7.3	Linux glibc 2.17+ ARM64	Details
tokenizers-0.20.1-pp37-pypy37_pp73-macosx_10_12_x86_64.whl 2.7 MB Oct 10, 2024 Without provenance	PyPy 3.7	PyPy 3.7 7.3	macOS 10.12+ x86-64	Details
tokenizers-0.20.1-cp312-none-win_amd64.whl 2.4 MB Oct 10, 2024 Without provenance	CPython 3.12	none	Windows x86-64	Details
tokenizers-0.20.1-cp312-none-win32.whl 2.2 MB Oct 10, 2024 Without provenance	CPython 3.12	none	Windows x86-32	Details
tokenizers-0.20.1-cp312-cp312-musllinux_1_1_x86_64.whl 9.3 MB Oct 10, 2024 Without provenance	CPython 3.12	CPython 3.12	Linux musl 1.1+ x86-64	Details
tokenizers-0.20.1-cp312-cp312-musllinux_1_1_aarch64.whl 9.0 MB Oct 10, 2024 Without provenance	CPython 3.12	CPython 3.12	Linux musl 1.1+ ARM64	Details
tokenizers-0.20.1-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl 3.0 MB Oct 10, 2024 Without provenance	CPython 3.12	CPython 3.12	Linux glibc 2.17+ x86-64	Details
tokenizers-0.20.1-cp312-cp312-manylinux_2_17_s390x.manylinux2014_s390x.whl 3.4 MB Oct 10, 2024 Without provenance	CPython 3.12	CPython 3.12	Linux glibc 2.17+ IBM System/390x	Details
tokenizers-0.20.1-cp312-cp312-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl 3.1 MB Oct 10, 2024 Without provenance	CPython 3.12	CPython 3.12	Linux glibc 2.17+ PowerPC 64-le	Details
tokenizers-0.20.1-cp312-cp312-manylinux_2_17_i686.manylinux2014_i686.whl 3.1 MB Oct 10, 2024 Without provenance	CPython 3.12	CPython 3.12	Linux glibc 2.17+ x86-32	Details
tokenizers-0.20.1-cp312-cp312-manylinux_2_17_armv7l.manylinux2014_armv7l.whl 2.8 MB Oct 10, 2024 Without provenance	CPython 3.12	CPython 3.12	Linux glibc 2.17+ ARMv7l	Details
tokenizers-0.20.1-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl 2.9 MB Oct 10, 2024 Without provenance	CPython 3.12	CPython 3.12	Linux glibc 2.17+ ARM64	Details
tokenizers-0.20.1-cp312-cp312-macosx_11_0_arm64.whl 2.6 MB Oct 10, 2024 Without provenance	CPython 3.12	CPython 3.12	macOS 11.0+ ARM64	Details
tokenizers-0.20.1-cp312-cp312-macosx_10_12_x86_64.whl 2.7 MB Oct 10, 2024 Without provenance	CPython 3.12	CPython 3.12	macOS 10.12+ x86-64	Details
tokenizers-0.20.1-cp311-none-win_amd64.whl 2.4 MB Oct 10, 2024 Without provenance	CPython 3.11	none	Windows x86-64	Details
tokenizers-0.20.1-cp311-none-win32.whl 2.2 MB Oct 10, 2024 Without provenance	CPython 3.11	none	Windows x86-32	Details
tokenizers-0.20.1-cp311-cp311-musllinux_1_1_x86_64.whl 9.3 MB Oct 10, 2024 Without provenance	CPython 3.11	CPython 3.11	Linux musl 1.1+ x86-64	Details
tokenizers-0.20.1-cp311-cp311-musllinux_1_1_aarch64.whl 9.0 MB Oct 10, 2024 Without provenance	CPython 3.11	CPython 3.11	Linux musl 1.1+ ARM64	Details
tokenizers-0.20.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl 3.0 MB Oct 10, 2024 Without provenance	CPython 3.11	CPython 3.11	Linux glibc 2.17+ x86-64	Details
tokenizers-0.20.1-cp311-cp311-manylinux_2_17_s390x.manylinux2014_s390x.whl 3.4 MB Oct 10, 2024 Without provenance	CPython 3.11	CPython 3.11	Linux glibc 2.17+ IBM System/390x	Details
tokenizers-0.20.1-cp311-cp311-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl 3.1 MB Oct 10, 2024 Without provenance	CPython 3.11	CPython 3.11	Linux glibc 2.17+ PowerPC 64-le	Details
tokenizers-0.20.1-cp311-cp311-manylinux_2_17_i686.manylinux2014_i686.whl 3.1 MB Oct 10, 2024 Without provenance	CPython 3.11	CPython 3.11	Linux glibc 2.17+ x86-32	Details
tokenizers-0.20.1-cp311-cp311-manylinux_2_17_armv7l.manylinux2014_armv7l.whl 2.8 MB Oct 10, 2024 Without provenance	CPython 3.11	CPython 3.11	Linux glibc 2.17+ ARMv7l	Details
tokenizers-0.20.1-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl 2.9 MB Oct 10, 2024 Without provenance	CPython 3.11	CPython 3.11	Linux glibc 2.17+ ARM64	Details
tokenizers-0.20.1-cp311-cp311-macosx_11_0_arm64.whl 2.6 MB Oct 10, 2024 Without provenance	CPython 3.11	CPython 3.11	macOS 11.0+ ARM64	Details
tokenizers-0.20.1-cp311-cp311-macosx_10_12_x86_64.whl 2.7 MB Oct 10, 2024 Without provenance	CPython 3.11	CPython 3.11	macOS 10.12+ x86-64	Details
tokenizers-0.20.1-cp310-none-win_amd64.whl 2.4 MB Oct 10, 2024 Without provenance	CPython 3.10	none	Windows x86-64	Details
tokenizers-0.20.1-cp310-none-win32.whl 2.2 MB Oct 10, 2024 Without provenance	CPython 3.10	none	Windows x86-32	Details
tokenizers-0.20.1-cp310-cp310-musllinux_1_1_x86_64.whl 9.3 MB Oct 10, 2024 Without provenance	CPython 3.10	CPython 3.10	Linux musl 1.1+ x86-64	Details
tokenizers-0.20.1-cp310-cp310-musllinux_1_1_aarch64.whl 9.0 MB Oct 10, 2024 Without provenance	CPython 3.10	CPython 3.10	Linux musl 1.1+ ARM64	Details
tokenizers-0.20.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl 3.0 MB Oct 10, 2024 Without provenance	CPython 3.10	CPython 3.10	Linux glibc 2.17+ x86-64	Details
tokenizers-0.20.1-cp310-cp310-manylinux_2_17_s390x.manylinux2014_s390x.whl 3.4 MB Oct 10, 2024 Without provenance	CPython 3.10	CPython 3.10	Linux glibc 2.17+ IBM System/390x	Details
tokenizers-0.20.1-cp310-cp310-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl 3.1 MB Oct 10, 2024 Without provenance	CPython 3.10	CPython 3.10	Linux glibc 2.17+ PowerPC 64-le	Details
tokenizers-0.20.1-cp310-cp310-manylinux_2_17_i686.manylinux2014_i686.whl 3.1 MB Oct 10, 2024 Without provenance	CPython 3.10	CPython 3.10	Linux glibc 2.17+ x86-32	Details
tokenizers-0.20.1-cp310-cp310-manylinux_2_17_armv7l.manylinux2014_armv7l.whl 2.8 MB Oct 10, 2024 Without provenance	CPython 3.10	CPython 3.10	Linux glibc 2.17+ ARMv7l	Details
tokenizers-0.20.1-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl 2.9 MB Oct 10, 2024 Without provenance	CPython 3.10	CPython 3.10	Linux glibc 2.17+ ARM64	Details
tokenizers-0.20.1-cp310-cp310-macosx_11_0_arm64.whl 2.6 MB Oct 10, 2024 Without provenance	CPython 3.10	CPython 3.10	macOS 11.0+ ARM64	Details
tokenizers-0.20.1-cp310-cp310-macosx_10_12_x86_64.whl 2.7 MB Oct 10, 2024 Without provenance	CPython 3.10	CPython 3.10	macOS 10.12+ x86-64	Details
tokenizers-0.20.1-cp39-none-win_amd64.whl 2.4 MB Oct 10, 2024 Without provenance	CPython 3.9	none	Windows x86-64	Details
tokenizers-0.20.1-cp39-none-win32.whl 2.2 MB Oct 10, 2024 Without provenance	CPython 3.9	none	Windows x86-32	Details
tokenizers-0.20.1-cp39-cp39-musllinux_1_1_x86_64.whl 9.3 MB Oct 10, 2024 Without provenance	CPython 3.9	CPython 3.9	Linux musl 1.1+ x86-64	Details
tokenizers-0.20.1-cp39-cp39-musllinux_1_1_aarch64.whl 9.0 MB Oct 10, 2024 Without provenance	CPython 3.9	CPython 3.9	Linux musl 1.1+ ARM64	Details
tokenizers-0.20.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl 3.0 MB Oct 10, 2024 Without provenance	CPython 3.9	CPython 3.9	Linux glibc 2.17+ x86-64	Details
tokenizers-0.20.1-cp39-cp39-manylinux_2_17_s390x.manylinux2014_s390x.whl 3.4 MB Oct 10, 2024 Without provenance	CPython 3.9	CPython 3.9	Linux glibc 2.17+ IBM System/390x	Details
tokenizers-0.20.1-cp39-cp39-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl 3.1 MB Oct 10, 2024 Without provenance	CPython 3.9	CPython 3.9	Linux glibc 2.17+ PowerPC 64-le	Details
tokenizers-0.20.1-cp39-cp39-manylinux_2_17_i686.manylinux2014_i686.whl 3.1 MB Oct 10, 2024 Without provenance	CPython 3.9	CPython 3.9	Linux glibc 2.17+ x86-32	Details
tokenizers-0.20.1-cp39-cp39-manylinux_2_17_armv7l.manylinux2014_armv7l.whl 2.8 MB Oct 10, 2024 Without provenance	CPython 3.9	CPython 3.9	Linux glibc 2.17+ ARMv7l	Details
tokenizers-0.20.1-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl 2.9 MB Oct 10, 2024 Without provenance	CPython 3.9	CPython 3.9	Linux glibc 2.17+ ARM64	Details
tokenizers-0.20.1-cp39-cp39-macosx_11_0_arm64.whl 2.6 MB Oct 10, 2024 Without provenance	CPython 3.9	CPython 3.9	macOS 11.0+ ARM64	Details
tokenizers-0.20.1-cp39-cp39-macosx_10_12_x86_64.whl 2.7 MB Oct 10, 2024 Without provenance	CPython 3.9	CPython 3.9	macOS 10.12+ x86-64	Details
tokenizers-0.20.1-cp38-none-win_amd64.whl 2.4 MB Oct 10, 2024 Without provenance	CPython 3.8	none	Windows x86-64	Details
tokenizers-0.20.1-cp38-none-win32.whl 2.2 MB Oct 10, 2024 Without provenance	CPython 3.8	none	Windows x86-32	Details
tokenizers-0.20.1-cp38-cp38-musllinux_1_1_x86_64.whl 9.3 MB Oct 10, 2024 Without provenance	CPython 3.8	CPython 3.8	Linux musl 1.1+ x86-64	Details
tokenizers-0.20.1-cp38-cp38-musllinux_1_1_aarch64.whl 9.0 MB Oct 10, 2024 Without provenance	CPython 3.8	CPython 3.8	Linux musl 1.1+ ARM64	Details
tokenizers-0.20.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl 3.0 MB Oct 10, 2024 Without provenance	CPython 3.8	CPython 3.8	Linux glibc 2.17+ x86-64	Details
tokenizers-0.20.1-cp38-cp38-manylinux_2_17_s390x.manylinux2014_s390x.whl 3.4 MB Oct 10, 2024 Without provenance	CPython 3.8	CPython 3.8	Linux glibc 2.17+ IBM System/390x	Details
tokenizers-0.20.1-cp38-cp38-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl 3.1 MB Oct 10, 2024 Without provenance	CPython 3.8	CPython 3.8	Linux glibc 2.17+ PowerPC 64-le	Details
tokenizers-0.20.1-cp38-cp38-manylinux_2_17_i686.manylinux2014_i686.whl 3.1 MB Oct 10, 2024 Without provenance	CPython 3.8	CPython 3.8	Linux glibc 2.17+ x86-32	Details
tokenizers-0.20.1-cp38-cp38-manylinux_2_17_armv7l.manylinux2014_armv7l.whl 2.8 MB Oct 10, 2024 Without provenance	CPython 3.8	CPython 3.8	Linux glibc 2.17+ ARMv7l	Details
tokenizers-0.20.1-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl 2.9 MB Oct 10, 2024 Without provenance	CPython 3.8	CPython 3.8	Linux glibc 2.17+ ARM64	Details
tokenizers-0.20.1-cp38-cp38-macosx_11_0_arm64.whl 2.6 MB Oct 10, 2024 Without provenance	CPython 3.8	CPython 3.8	macOS 11.0+ ARM64	Details
tokenizers-0.20.1-cp38-cp38-macosx_10_12_x86_64.whl 2.7 MB Oct 10, 2024 Without provenance	CPython 3.8	CPython 3.8	macOS 10.12+ x86-64	Details
tokenizers-0.20.1-cp37-none-win_amd64.whl 2.4 MB Oct 10, 2024 Without provenance	CPython 3.7	none	Windows x86-64	Details
tokenizers-0.20.1-cp37-none-win32.whl 2.2 MB Oct 10, 2024 Without provenance	CPython 3.7	none	Windows x86-32	Details
tokenizers-0.20.1-cp37-cp37m-musllinux_1_1_x86_64.whl 9.3 MB Oct 10, 2024 Without provenance	CPython 3.7	CPython 3.7 pymalloc	Linux musl 1.1+ x86-64	Details
tokenizers-0.20.1-cp37-cp37m-musllinux_1_1_aarch64.whl 9.0 MB Oct 10, 2024 Without provenance	CPython 3.7	CPython 3.7 pymalloc	Linux musl 1.1+ ARM64	Details
tokenizers-0.20.1-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl 3.0 MB Oct 10, 2024 Without provenance	CPython 3.7	CPython 3.7 pymalloc	Linux glibc 2.17+ x86-64	Details
tokenizers-0.20.1-cp37-cp37m-manylinux_2_17_s390x.manylinux2014_s390x.whl 3.4 MB Oct 10, 2024 Without provenance	CPython 3.7	CPython 3.7 pymalloc	Linux glibc 2.17+ IBM System/390x	Details
tokenizers-0.20.1-cp37-cp37m-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl 3.1 MB Oct 10, 2024 Without provenance	CPython 3.7	CPython 3.7 pymalloc	Linux glibc 2.17+ PowerPC 64-le	Details
tokenizers-0.20.1-cp37-cp37m-manylinux_2_17_i686.manylinux2014_i686.whl 3.1 MB Oct 10, 2024 Without provenance	CPython 3.7	CPython 3.7 pymalloc	Linux glibc 2.17+ x86-32	Details
tokenizers-0.20.1-cp37-cp37m-manylinux_2_17_armv7l.manylinux2014_armv7l.whl 2.8 MB Oct 10, 2024 Without provenance	CPython 3.7	CPython 3.7 pymalloc	Linux glibc 2.17+ ARMv7l	Details
tokenizers-0.20.1-cp37-cp37m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl 2.9 MB Oct 10, 2024 Without provenance	CPython 3.7	CPython 3.7 pymalloc	Linux glibc 2.17+ ARM64	Details
tokenizers-0.20.1-cp37-cp37m-macosx_11_0_arm64.whl 2.6 MB Oct 10, 2024 Without provenance	CPython 3.7	CPython 3.7 pymalloc	macOS 11.0+ ARM64	Details
tokenizers-0.20.1-cp37-cp37m-macosx_10_12_x86_64.whl 2.7 MB Oct 10, 2024 Without provenance	CPython 3.7	CPython 3.7 pymalloc	macOS 10.12+ x86-64	Details

Total release size: 406.7 MB

Release files / tokenizers-0.20.1.tar.gz

Download URL	tokenizers-0.20.1.tar.gz
Size	339.6 kB
Tags	Source
SHA-256 checksum How to use checksums	`84edcc7cdeeee45ceedb65d518fffb77aec69311c9c8e30f77ad84da3025f002`
BLAKE2b-256 checksum How to use checksums	`d7fb373b66ba58cbf5eda371480e4e051d8892ea1433a73f1f92c48657a699a6`
Upload date	Oct 10, 2024
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/5.1.1 CPython/3.12.7`

Release files / tokenizers-0.20.1-pp310-pypy310_pp73-musllinux_1_1_x86_64.whl

Download URL	tokenizers-0.20.1-pp310-pypy310_pp73-musllinux_1_1_x86_64.whl
Size	9.3 MB
Tags	Linux musl 1.1+ x86-64 PyPy 3.10 PyPy 3.10 7.3
SHA-256 checksum How to use checksums	`f40df5e0294a95131cc5f0e0eb91fe86d88837abfbee46b9b3610b09860195a7`
BLAKE2b-256 checksum How to use checksums	`d4f2ea998aaf69966a87f92e31db7cba887125994bb9cd9a4dfcc83ac202d446`
Upload date	Oct 10, 2024
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/5.1.1 CPython/3.12.7`

Release files / tokenizers-0.20.1-pp310-pypy310_pp73-musllinux_1_1_aarch64.whl

Download URL	tokenizers-0.20.1-pp310-pypy310_pp73-musllinux_1_1_aarch64.whl
Size	9.0 MB
Tags	Linux musl 1.1+ ARM64 PyPy 3.10 PyPy 3.10 7.3
SHA-256 checksum How to use checksums	`218e5a3561561ea0f0ef1559c6d95b825308dbec23fb55b70b92589e7ff2e1e8`
BLAKE2b-256 checksum How to use checksums	`cbf779a74f8c54d1232ddbd68967ce56a00cc9589a31b94bee4cf9f34af91ace`
Upload date	Oct 10, 2024
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/5.1.1 CPython/3.12.7`

Release files / tokenizers-0.20.1-pp310-pypy310_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl

Download URL	tokenizers-0.20.1-pp310-pypy310_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Size	3.0 MB
Tags	Linux glibc 2.17+ x86-64 PyPy 3.10 PyPy 3.10 7.3
SHA-256 checksum How to use checksums	`03b03cf8b9a32254b1bf8a305fb95c6daf1baae0c1f93b27f2b08c9759f41dee`
BLAKE2b-256 checksum How to use checksums	`9c2c9f04aa030ba8994d478ab35464f8c541aad264556811f12afce9369cc0d3`
Upload date	Oct 10, 2024
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/5.1.1 CPython/3.12.7`

Release files / tokenizers-0.20.1-pp310-pypy310_pp73-manylinux_2_17_i686.manylinux2014_i686.whl

Download URL	tokenizers-0.20.1-pp310-pypy310_pp73-manylinux_2_17_i686.manylinux2014_i686.whl
Size	3.1 MB
Tags	Linux glibc 2.17+ x86-32 PyPy 3.10 PyPy 3.10 7.3
SHA-256 checksum How to use checksums	`0d3caf244ce89d24c87545aafc3448be15870096e796c703a0d68547187192e1`
BLAKE2b-256 checksum How to use checksums	`c1e6ec76a7761eb7ba3cf95e2485cb2e7999a8eb0900d771616c0efa61beb1cd`
Upload date	Oct 10, 2024
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/5.1.1 CPython/3.12.7`

Release files / tokenizers-0.20.1-pp310-pypy310_pp73-manylinux_2_17_aarch64.manylinux2014_aarch64.whl

Download URL	tokenizers-0.20.1-pp310-pypy310_pp73-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Size	2.9 MB
Tags	Linux glibc 2.17+ ARM64 PyPy 3.10 PyPy 3.10 7.3
SHA-256 checksum How to use checksums	`359eceb6a620c965988fc559cebc0a98db26713758ec4df43fb76d41486a8ed5`
BLAKE2b-256 checksum How to use checksums	`97e3167ca1981b3f512030a28f591b8ef786585b625d45f0fbf1c42723474ecd`
Upload date	Oct 10, 2024
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/5.1.1 CPython/3.12.7`

Release files / tokenizers-0.20.1-pp310-pypy310_pp73-macosx_11_0_arm64.whl

Download URL	tokenizers-0.20.1-pp310-pypy310_pp73-macosx_11_0_arm64.whl
Size	2.6 MB
Tags	PyPy 3.10 PyPy 3.10 7.3 macOS 11.0+ ARM64
SHA-256 checksum How to use checksums	`712f90ea33f9bd2586b4a90d697c26d56d0a22fd3c91104c5858c4b5b6489a79`
BLAKE2b-256 checksum How to use checksums	`37988221a62aed679aefcbc1793ed8bb33f1e060f8b7d95bb20809db1b5c0e0e`
Upload date	Oct 10, 2024
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/5.1.1 CPython/3.12.7`

Release files / tokenizers-0.20.1-pp310-pypy310_pp73-macosx_10_12_x86_64.whl

Download URL	tokenizers-0.20.1-pp310-pypy310_pp73-macosx_10_12_x86_64.whl
Size	2.7 MB
Tags	PyPy 3.10 PyPy 3.10 7.3 macOS 10.12+ x86-64
SHA-256 checksum How to use checksums	`48689da7a395df41114f516208d6550e3e905e1239cc5ad386686d9358e9cef0`
BLAKE2b-256 checksum How to use checksums	`4b9ecf0911565ae302e4e4ed3d53bba28f2db75a9418f4e89e2434246723f01a`
Upload date	Oct 10, 2024
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/5.1.1 CPython/3.12.7`

Release files / tokenizers-0.20.1-pp39-pypy39_pp73-musllinux_1_1_x86_64.whl

Download URL	tokenizers-0.20.1-pp39-pypy39_pp73-musllinux_1_1_x86_64.whl
Size	9.3 MB
Tags	Linux musl 1.1+ x86-64 PyPy 3.9 PyPy 3.9 7.3
SHA-256 checksum How to use checksums	`de291633fb9303555793cc544d4a86e858da529b7d0b752bcaf721ae1d74b2c9`
BLAKE2b-256 checksum How to use checksums	`da961603264112ce0fb5da7dae215de037d21dcd9b7c6947ed575b9e81a0d05a`
Upload date	Oct 10, 2024
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/5.1.1 CPython/3.12.7`

Release files / tokenizers-0.20.1-pp39-pypy39_pp73-musllinux_1_1_aarch64.whl

Download URL	tokenizers-0.20.1-pp39-pypy39_pp73-musllinux_1_1_aarch64.whl
Size	9.0 MB
Tags	Linux musl 1.1+ ARM64 PyPy 3.9 PyPy 3.9 7.3
SHA-256 checksum How to use checksums	`e7edb8ec12c100d5458d15b1e47c0eb30ad606a05641f19af7563bc3d1608c14`
BLAKE2b-256 checksum How to use checksums	`2a63a640fc3e011bb0392e05c9e5f7054c039d3542426dec46f6b7c8e753d999`
Upload date	Oct 10, 2024
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/5.1.1 CPython/3.12.7`

Release files / tokenizers-0.20.1-pp39-pypy39_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl

Download URL	tokenizers-0.20.1-pp39-pypy39_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Size	3.0 MB
Tags	Linux glibc 2.17+ x86-64 PyPy 3.9 PyPy 3.9 7.3
SHA-256 checksum How to use checksums	`c5ffe0d7f7bfcfa3b2585776ecf11da2e01c317027c8573c78ebcb8985279e23`
BLAKE2b-256 checksum How to use checksums	`24e820d71cb85d3f3f39e01246f7827b42b6f9862332c500433f20307addc212`
Upload date	Oct 10, 2024
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/5.1.1 CPython/3.12.7`

Release files / tokenizers-0.20.1-pp39-pypy39_pp73-manylinux_2_17_i686.manylinux2014_i686.whl

Download URL	tokenizers-0.20.1-pp39-pypy39_pp73-manylinux_2_17_i686.manylinux2014_i686.whl
Size	3.1 MB
Tags	Linux glibc 2.17+ x86-32 PyPy 3.9 PyPy 3.9 7.3
SHA-256 checksum How to use checksums	`5dc611e6ac0fa00a41de19c3bf6391a05ea201d2d22b757d63f5491ec0e67faa`
BLAKE2b-256 checksum How to use checksums	`f638977e4ca1dc955fb368fa900050d08b9bf67f8d5de3bd364f0d04ed4eedfd`
Upload date	Oct 10, 2024
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/5.1.1 CPython/3.12.7`

Release files / tokenizers-0.20.1-pp39-pypy39_pp73-manylinux_2_17_aarch64.manylinux2014_aarch64.whl

Download URL	tokenizers-0.20.1-pp39-pypy39_pp73-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Size	2.9 MB
Tags	Linux glibc 2.17+ ARM64 PyPy 3.9 PyPy 3.9 7.3
SHA-256 checksum How to use checksums	`3e0305fc1ec6b1e5052d30d9c1d5c807081a7bd0cae46a33d03117082e91908c`
BLAKE2b-256 checksum How to use checksums	`f19fffcf52c1d0f27778ec2e5c526ee7c727c24441254ef5732d0b391583ff7b`
Upload date	Oct 10, 2024
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/5.1.1 CPython/3.12.7`

Release files / tokenizers-0.20.1-pp39-pypy39_pp73-macosx_11_0_arm64.whl

Download URL	tokenizers-0.20.1-pp39-pypy39_pp73-macosx_11_0_arm64.whl
Size	2.6 MB
Tags	PyPy 3.9 PyPy 3.9 7.3 macOS 11.0+ ARM64
SHA-256 checksum How to use checksums	`0b7f515c83397e73292accdbbbedc62264e070bae9682f06061e2ddce67cacaf`
BLAKE2b-256 checksum How to use checksums	`8f35643727d36c409f71ddd9b914513ed793dcaa56d92e4310a5f3a0d6b84138`
Upload date	Oct 10, 2024
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/5.1.1 CPython/3.12.7`

Release files / tokenizers-0.20.1-pp39-pypy39_pp73-macosx_10_12_x86_64.whl

Download URL	tokenizers-0.20.1-pp39-pypy39_pp73-macosx_10_12_x86_64.whl
Size	2.7 MB
Tags	PyPy 3.9 PyPy 3.9 7.3 macOS 10.12+ x86-64
SHA-256 checksum How to use checksums	`89d5c337d74ea6e5e7dc8af124cf177be843bbb9ca6e58c01f75ea103c12c8a9`
BLAKE2b-256 checksum How to use checksums	`3c89969b427a3882c029f968e44d5bb1cff14606f878fd7c7d5b6010767881d6`
Upload date	Oct 10, 2024
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/5.1.1 CPython/3.12.7`

Release files / tokenizers-0.20.1-pp38-pypy38_pp73-musllinux_1_1_x86_64.whl

Download URL	tokenizers-0.20.1-pp38-pypy38_pp73-musllinux_1_1_x86_64.whl
Size	9.3 MB
Tags	Linux musl 1.1+ x86-64 PyPy 3.8 PyPy 3.8 7.3
SHA-256 checksum How to use checksums	`f861889707b54a9ab1204030b65fd6c22bdd4a95205deec7994dc22a8baa2ea4`
BLAKE2b-256 checksum How to use checksums	`22e39038cf20799bd4832190325810329c1ea63984be0486b2c1934a22ff8327`
Upload date	Oct 10, 2024
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/5.1.1 CPython/3.12.7`

Release files / tokenizers-0.20.1-pp38-pypy38_pp73-musllinux_1_1_aarch64.whl

Download URL	tokenizers-0.20.1-pp38-pypy38_pp73-musllinux_1_1_aarch64.whl
Size	9.0 MB
Tags	Linux musl 1.1+ ARM64 PyPy 3.8 PyPy 3.8 7.3
SHA-256 checksum How to use checksums	`62eb9daea2a2c06bcd8113a5824af8ef8ee7405d3a71123ba4d52c79bb3d9f1a`
BLAKE2b-256 checksum How to use checksums	`2cd01dadd1b37ac18a8056a88d88f5944eecb1d98717d8f3d74d500f1f4b9f9e`
Upload date	Oct 10, 2024
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/5.1.1 CPython/3.12.7`

Release files / tokenizers-0.20.1-pp38-pypy38_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl

Download URL	tokenizers-0.20.1-pp38-pypy38_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Size	3.0 MB
Tags	Linux glibc 2.17+ x86-64 PyPy 3.8 PyPy 3.8 7.3
SHA-256 checksum How to use checksums	`9041ee665d0fa7f5c4ccf0f81f5e6b7087f797f85b143c094126fc2611fec9d0`
BLAKE2b-256 checksum How to use checksums	`6c9d8001ae42cf4ad907a97d86b5d3d43db6f9d66338d8ba2dc803855bd0005f`
Upload date	Oct 10, 2024
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/5.1.1 CPython/3.12.7`

Release files / tokenizers-0.20.1-pp38-pypy38_pp73-manylinux_2_17_i686.manylinux2014_i686.whl

Download URL	tokenizers-0.20.1-pp38-pypy38_pp73-manylinux_2_17_i686.manylinux2014_i686.whl
Size	3.1 MB
Tags	Linux glibc 2.17+ x86-32 PyPy 3.8 PyPy 3.8 7.3
SHA-256 checksum How to use checksums	`cd28a8614f5c82a54ab2463554e84ad79526c5184cf4573bbac2efbbbcead457`
BLAKE2b-256 checksum How to use checksums	`7e1f897845e8ccf1cbfce83d0a3b5a467d56301ff461006bcf807104959ad552`
Upload date	Oct 10, 2024
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/5.1.1 CPython/3.12.7`

Release files / tokenizers-0.20.1-pp38-pypy38_pp73-manylinux_2_17_aarch64.manylinux2014_aarch64.whl

Download URL	tokenizers-0.20.1-pp38-pypy38_pp73-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Size	2.9 MB
Tags	Linux glibc 2.17+ ARM64 PyPy 3.8 PyPy 3.8 7.3
SHA-256 checksum How to use checksums	`130e35e76f9337ed6c31be386e75d4925ea807055acf18ca1a9b0eec03d8fe23`
BLAKE2b-256 checksum How to use checksums	`459c4f105c42cd29653e2f3c5c819b33f6dba067313d0430f4825a3c1c77b350`
Upload date	Oct 10, 2024
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/5.1.1 CPython/3.12.7`

Release files / tokenizers-0.20.1-pp38-pypy38_pp73-macosx_11_0_arm64.whl

Download URL	tokenizers-0.20.1-pp38-pypy38_pp73-macosx_11_0_arm64.whl
Size	2.6 MB
Tags	PyPy 3.8 PyPy 3.8 7.3 macOS 11.0+ ARM64
SHA-256 checksum How to use checksums	`81970b80b8ac126910295f8aab2d7ef962009ea39e0d86d304769493f69aaa1e`
BLAKE2b-256 checksum How to use checksums	`4d4fe7a4ea2221a7e007d9844364bc8fd7ea89b35c3c5fbc42ed7d03b12c1a86`
Upload date	Oct 10, 2024
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/5.1.1 CPython/3.12.7`

Release files / tokenizers-0.20.1-pp38-pypy38_pp73-macosx_10_12_x86_64.whl

Download URL	tokenizers-0.20.1-pp38-pypy38_pp73-macosx_10_12_x86_64.whl
Size	2.7 MB
Tags	PyPy 3.8 PyPy 3.8 7.3 macOS 10.12+ x86-64
SHA-256 checksum How to use checksums	`ebe63e31f9c1a970c53866d814e35ec2ec26fda03097c486f82f3891cee60830`
BLAKE2b-256 checksum How to use checksums	`e9495e2df67ea57e92317175abc60dc81d2031dea02020dd77c216eab2a48d97`
Upload date	Oct 10, 2024
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/5.1.1 CPython/3.12.7`

Release files / tokenizers-0.20.1-pp37-pypy37_pp73-musllinux_1_1_x86_64.whl

Download URL	tokenizers-0.20.1-pp37-pypy37_pp73-musllinux_1_1_x86_64.whl
Size	9.3 MB
Tags	Linux musl 1.1+ x86-64 PyPy 3.7 PyPy 3.7 7.3
SHA-256 checksum How to use checksums	`1fbbaf17a393c78d8aedb6a334097c91cb4119a9ced4764ab8cfdc8d254dc9f9`
BLAKE2b-256 checksum How to use checksums	`b5b2d13823d3b44a8cf7b3f702c2a36e7d4de57bda623b5f4d1109ffab8adebf`
Upload date	Oct 10, 2024
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/5.1.1 CPython/3.12.7`

Release files / tokenizers-0.20.1-pp37-pypy37_pp73-musllinux_1_1_aarch64.whl

Download URL	tokenizers-0.20.1-pp37-pypy37_pp73-musllinux_1_1_aarch64.whl
Size	9.0 MB
Tags	Linux musl 1.1+ ARM64 PyPy 3.7 PyPy 3.7 7.3
SHA-256 checksum How to use checksums	`956f21d359ae29dd51ca5726d2c9a44ffafa041c623f5aa33749da87cfa809b9`
BLAKE2b-256 checksum How to use checksums	`4e506bb7f79034e4043873320aadbc36fbdd8e97748bfa7e0b18f41dc26c7296`
Upload date	Oct 10, 2024
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/5.1.1 CPython/3.12.7`

Release files / tokenizers-0.20.1-pp37-pypy37_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl

Download URL	tokenizers-0.20.1-pp37-pypy37_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Size	3.0 MB
Tags	Linux glibc 2.17+ x86-64 PyPy 3.7 PyPy 3.7 7.3
SHA-256 checksum How to use checksums	`31e87fca4f6bbf5cc67481b562147fe932f73d5602734de7dd18a8f2eee9c6dd`
BLAKE2b-256 checksum How to use checksums	`26bdb531886b73f79ab299fd190fcce4f7d0fdf678da0cb688dc6d29d4964a70`
Upload date	Oct 10, 2024
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/5.1.1 CPython/3.12.7`

Release files / tokenizers-0.20.1-pp37-pypy37_pp73-manylinux_2_17_i686.manylinux2014_i686.whl

Download URL	tokenizers-0.20.1-pp37-pypy37_pp73-manylinux_2_17_i686.manylinux2014_i686.whl
Size	3.1 MB
Tags	Linux glibc 2.17+ x86-32 PyPy 3.7 PyPy 3.7 7.3
SHA-256 checksum How to use checksums	`62f7fbd3c2c38b179556d879edae442b45f68312019c3a6013e56c3947a4e648`
BLAKE2b-256 checksum How to use checksums	`3d6ead7f9c604fcc191cdd114d6e7d48e26a57deaadf5e9ef52245c2c57b1fc0`
Upload date	Oct 10, 2024
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/5.1.1 CPython/3.12.7`

Release files / tokenizers-0.20.1-pp37-pypy37_pp73-manylinux_2_17_aarch64.manylinux2014_aarch64.whl

Download URL	tokenizers-0.20.1-pp37-pypy37_pp73-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Size	2.9 MB
Tags	Linux glibc 2.17+ ARM64 PyPy 3.7 PyPy 3.7 7.3
SHA-256 checksum How to use checksums	`998700177b45f70afeb206ad22c08d9e5f3a80639dae1032bf41e8cbc4dada4b`
BLAKE2b-256 checksum How to use checksums	`dc33994f6977400c0c7991ff40157dc3663ce35286acb89c5d8997cfd1aab9da`
Upload date	Oct 10, 2024
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/5.1.1 CPython/3.12.7`

Release files / tokenizers-0.20.1-pp37-pypy37_pp73-macosx_10_12_x86_64.whl

Download URL	tokenizers-0.20.1-pp37-pypy37_pp73-macosx_10_12_x86_64.whl
Size	2.7 MB
Tags	PyPy 3.7 PyPy 3.7 7.3 macOS 10.12+ x86-64
SHA-256 checksum How to use checksums	`08aaa0d72bb65058e8c4b0455f61b840b156c557e2aca57627056624c3a93976`
BLAKE2b-256 checksum How to use checksums	`2cae10d10f93b2e9fba24d038954eef5cf22a29774f65b449d23c68f642525d0`
Upload date	Oct 10, 2024
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/5.1.1 CPython/3.12.7`

Release files / tokenizers-0.20.1-cp312-none-win_amd64.whl

Download URL	tokenizers-0.20.1-cp312-none-win_amd64.whl
Size	2.4 MB
Tags	CPython 3.12 Windows x86-64
SHA-256 checksum How to use checksums	`b0874481aea54a178f2bccc45aa2d0c99cd3f79143a0948af6a9a21dcc49173b`
BLAKE2b-256 checksum How to use checksums	`7eba18bf6a7ad04f8225b71aa862b57188748d1d81e268de4a9aac1aed237246`
Upload date	Oct 10, 2024
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/5.1.1 CPython/3.12.7`

Release files / tokenizers-0.20.1-cp312-none-win32.whl

Download URL	tokenizers-0.20.1-cp312-none-win32.whl
Size	2.2 MB
Tags	CPython 3.12 Windows x86-32
SHA-256 checksum How to use checksums	`6d3ac5c1f48358ffe20086bf065e843c0d0a9fce0d7f0f45d5f2f9fba3609ca5`
BLAKE2b-256 checksum How to use checksums	`41ff4aeb924d09f6561209b57af9123a0a28fa69472cc71ee40415f036253203`
Upload date	Oct 10, 2024
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/5.1.1 CPython/3.12.7`

Release files / tokenizers-0.20.1-cp312-cp312-musllinux_1_1_x86_64.whl

Download URL	tokenizers-0.20.1-cp312-cp312-musllinux_1_1_x86_64.whl
Size	9.3 MB
Tags	CPython 3.12 Linux musl 1.1+ x86-64
SHA-256 checksum How to use checksums	`3ea919687aa7001a8ff1ba36ac64f165c4e89035f57998fa6cedcfd877be619d`
BLAKE2b-256 checksum How to use checksums	`bb37eaa072b848471d31ae3df6e6d5be5ae594ed5fe39ca921e65cabf193dbde`
Upload date	Oct 10, 2024
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/5.1.1 CPython/3.12.7`

Release files / tokenizers-0.20.1-cp312-cp312-musllinux_1_1_aarch64.whl

Download URL	tokenizers-0.20.1-cp312-cp312-musllinux_1_1_aarch64.whl
Size	9.0 MB
Tags	CPython 3.12 Linux musl 1.1+ ARM64
SHA-256 checksum How to use checksums	`0c6a796ddcd9a19ad13cf146997cd5895a421fe6aec8fd970d69f9117bddb45c`
BLAKE2b-256 checksum How to use checksums	`d4abceb7bdb3394431e92b18123faef9862877009f61377bfa45ffe5135747a5`
Upload date	Oct 10, 2024
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/5.1.1 CPython/3.12.7`

Release files / tokenizers-0.20.1-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl

Download URL	tokenizers-0.20.1-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Size	3.0 MB
Tags	CPython 3.12 Linux glibc 2.17+ x86-64
SHA-256 checksum How to use checksums	`689b93d2e26d04da337ac407acec8b5d081d8d135e3e5066a88edd5bdb5aff89`
BLAKE2b-256 checksum How to use checksums	`2459664121cb41b4f738479e2e1271013a2a7c9160955922536fb723a9c690b7`
Upload date	Oct 10, 2024
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/5.1.1 CPython/3.12.7`

Release files / tokenizers-0.20.1-cp312-cp312-manylinux_2_17_s390x.manylinux2014_s390x.whl

Download URL	tokenizers-0.20.1-cp312-cp312-manylinux_2_17_s390x.manylinux2014_s390x.whl
Size	3.4 MB
Tags	CPython 3.12 Linux glibc 2.17+ IBM System/390x
SHA-256 checksum How to use checksums	`bdd67a0e3503a9a7cf8bc5a4a49cdde5fa5bada09a51e4c7e1c73900297539bd`
BLAKE2b-256 checksum How to use checksums	`cf7dc895f076e552cb39ea0491f62ff6551cb3e60323a7496017182bd57cc314`
Upload date	Oct 10, 2024
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/5.1.1 CPython/3.12.7`

Release files / tokenizers-0.20.1-cp312-cp312-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl

Download URL	tokenizers-0.20.1-cp312-cp312-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl
Size	3.1 MB
Tags	CPython 3.12 Linux glibc 2.17+ PowerPC 64-le
SHA-256 checksum How to use checksums	`e2e2d47a819d2954f2c1cd0ad51bb58ffac6f53a872d5d82d65d79bf76b9896d`
BLAKE2b-256 checksum How to use checksums	`a42ada72c32446ad7f3e6e5cb3c625222a5b9b0bc10b50456f6cb79f6230ae1f`
Upload date	Oct 10, 2024
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/5.1.1 CPython/3.12.7`

Release files / tokenizers-0.20.1-cp312-cp312-manylinux_2_17_i686.manylinux2014_i686.whl

Download URL	tokenizers-0.20.1-cp312-cp312-manylinux_2_17_i686.manylinux2014_i686.whl
Size	3.1 MB
Tags	CPython 3.12 Linux glibc 2.17+ x86-32
SHA-256 checksum How to use checksums	`128c1110e950534426e2274837fc06b118ab5f2fa61c3436e60e0aada0ccfd67`
BLAKE2b-256 checksum How to use checksums	`f759185ff0bb35d46d88613e87bd76b03989ef8537ebf4f39876bddf9bed2fc1`
Upload date	Oct 10, 2024
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/5.1.1 CPython/3.12.7`

Release files / tokenizers-0.20.1-cp312-cp312-manylinux_2_17_armv7l.manylinux2014_armv7l.whl

Download URL	tokenizers-0.20.1-cp312-cp312-manylinux_2_17_armv7l.manylinux2014_armv7l.whl
Size	2.8 MB
Tags	CPython 3.12 Linux glibc 2.17+ ARMv7l
SHA-256 checksum How to use checksums	`47c1bcdd61e61136087459cb9e0b069ff23b5568b008265e5cbc927eae3387ce`
BLAKE2b-256 checksum How to use checksums	`285192e3b25eb41be7fd65219c832c4ff61bf5c8cc1c3d0543e9a117d63a0876`
Upload date	Oct 10, 2024
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/5.1.1 CPython/3.12.7`

Release files / tokenizers-0.20.1-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl

Download URL	tokenizers-0.20.1-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Size	2.9 MB
Tags	CPython 3.12 Linux glibc 2.17+ ARM64
SHA-256 checksum How to use checksums	`e98eee4dca22849fbb56a80acaa899eec5b72055d79637dd6aa15d5e4b8628c9`
BLAKE2b-256 checksum How to use checksums	`f46e9dfd1afcfd38fcc5b3a84bca54c33025561f7cab8ea375fa88f03407adc1`
Upload date	Oct 10, 2024
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/5.1.1 CPython/3.12.7`

Release files / tokenizers-0.20.1-cp312-cp312-macosx_11_0_arm64.whl

Download URL	tokenizers-0.20.1-cp312-cp312-macosx_11_0_arm64.whl
Size	2.6 MB
Tags	CPython 3.12 macOS 11.0+ ARM64
SHA-256 checksum How to use checksums	`2f13a2d16032ebc8bd812eb8099b035ac65887d8f0c207261472803b9633cf3e`
BLAKE2b-256 checksum How to use checksums	`99c32132487ca51148392f0d1ed7f35c23179f67d66fd64c233ff50f091258b4`
Upload date	Oct 10, 2024
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/5.1.1 CPython/3.12.7`

Release files / tokenizers-0.20.1-cp312-cp312-macosx_10_12_x86_64.whl

Download URL	tokenizers-0.20.1-cp312-cp312-macosx_10_12_x86_64.whl
Size	2.7 MB
Tags	CPython 3.12 macOS 10.12+ x86-64
SHA-256 checksum How to use checksums	`407ab666b38e02228fa785e81f7cf79ef929f104bcccf68a64525a54a93ceac9`
BLAKE2b-256 checksum How to use checksums	`8e8da051f979f955c6717099718054d7f51fea0a92d807a7d078a48f2684e54f`
Upload date	Oct 10, 2024
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/5.1.1 CPython/3.12.7`

Release files / tokenizers-0.20.1-cp311-none-win_amd64.whl

Download URL	tokenizers-0.20.1-cp311-none-win_amd64.whl
Size	2.4 MB
Tags	CPython 3.11 Windows x86-64
SHA-256 checksum How to use checksums	`899152a78b095559c287b4c6d0099469573bb2055347bb8154db106651296f39`
BLAKE2b-256 checksum How to use checksums	`f195f1b56f4b1fbd54bd7f170aa64258d0650500e9f45de217ffe4d4663809b6`
Upload date	Oct 10, 2024
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/5.1.1 CPython/3.12.7`

Release files / tokenizers-0.20.1-cp311-none-win32.whl

Download URL	tokenizers-0.20.1-cp311-none-win32.whl
Size	2.2 MB
Tags	CPython 3.11 Windows x86-32
SHA-256 checksum How to use checksums	`9af2dc4ee97d037bc6b05fa4429ddc87532c706316c5e11ce2f0596dfcfa77af`
BLAKE2b-256 checksum How to use checksums	`ea487d4ac79588b5b1c3651b753b0a1bdd1343d81af57be18138dfdb304a710a`
Upload date	Oct 10, 2024
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/5.1.1 CPython/3.12.7`

Release files / tokenizers-0.20.1-cp311-cp311-musllinux_1_1_x86_64.whl

Download URL	tokenizers-0.20.1-cp311-cp311-musllinux_1_1_x86_64.whl
Size	9.3 MB
Tags	CPython 3.11 Linux musl 1.1+ x86-64
SHA-256 checksum How to use checksums	`86dcd08da163912e17b27bbaba5efdc71b4fbffb841530fdb74c5707f3c49216`
BLAKE2b-256 checksum How to use checksums	`d042c287d28ebcb3ba4f712e7a58d8f170a7b569528acf2d2a8fd1f684c24c0c`
Upload date	Oct 10, 2024
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/5.1.1 CPython/3.12.7`

Release files / tokenizers-0.20.1-cp311-cp311-musllinux_1_1_aarch64.whl

Download URL	tokenizers-0.20.1-cp311-cp311-musllinux_1_1_aarch64.whl
Size	9.0 MB
Tags	CPython 3.11 Linux musl 1.1+ ARM64
SHA-256 checksum How to use checksums	`ee86d4095d3542d73579e953c2e5e07d9321af2ffea6ecc097d16d538a2dea16`
BLAKE2b-256 checksum How to use checksums	`d392e5b80e42c24e564ac892c9135e4b9ec34bbcd6cdf0cc7a04735c44fe2ced`
Upload date	Oct 10, 2024
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/5.1.1 CPython/3.12.7`

Release files / tokenizers-0.20.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl

Download URL	tokenizers-0.20.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Size	3.0 MB
Tags	CPython 3.11 Linux glibc 2.17+ x86-64
SHA-256 checksum How to use checksums	`ef3f1ae08fa9aea5891cbd69df29913e11d3841798e0bfb1ff78b78e4e7ea0a4`
BLAKE2b-256 checksum How to use checksums	`096c1b573998fe3f0e18ac5d434e43966de2d225d6837f099ce0df7df4274c87`
Upload date	Oct 10, 2024
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/5.1.1 CPython/3.12.7`

Release files / tokenizers-0.20.1-cp311-cp311-manylinux_2_17_s390x.manylinux2014_s390x.whl

Download URL	tokenizers-0.20.1-cp311-cp311-manylinux_2_17_s390x.manylinux2014_s390x.whl
Size	3.4 MB
Tags	CPython 3.11 Linux glibc 2.17+ IBM System/390x
SHA-256 checksum How to use checksums	`5170be9ec942f3d1d317817ced8d749b3e1202670865e4fd465e35d8c259de83`
BLAKE2b-256 checksum How to use checksums	`7e0f9136bc0ea492d29f1d72217c6231dc584bccd3ba41dde12d4a85c75eb12a`
Upload date	Oct 10, 2024
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/5.1.1 CPython/3.12.7`

Release files / tokenizers-0.20.1-cp311-cp311-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl

Download URL	tokenizers-0.20.1-cp311-cp311-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl
Size	3.1 MB
Tags	CPython 3.11 Linux glibc 2.17+ PowerPC 64-le
SHA-256 checksum How to use checksums	`0ecaf7b0e39caeb1aa6dd6e0975c405716c82c1312b55ac4f716ef563a906969`
BLAKE2b-256 checksum How to use checksums	`fef44302dce958ce0e7f2d85a4725cebe6b02161c2d82990a89317580e17469a`
Upload date	Oct 10, 2024
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/5.1.1 CPython/3.12.7`

Release files / tokenizers-0.20.1-cp311-cp311-manylinux_2_17_i686.manylinux2014_i686.whl

Download URL	tokenizers-0.20.1-cp311-cp311-manylinux_2_17_i686.manylinux2014_i686.whl
Size	3.1 MB
Tags	CPython 3.11 Linux glibc 2.17+ x86-32
SHA-256 checksum How to use checksums	`9300fac73ddc7e4b0330acbdda4efaabf74929a4a61e119a32a181f534a11b47`
BLAKE2b-256 checksum How to use checksums	`ff71b220deba78e42e483e2856c9cc83a8352c7c5d7322dad61eed4e1ca09c49`
Upload date	Oct 10, 2024
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/5.1.1 CPython/3.12.7`

Release files / tokenizers-0.20.1-cp311-cp311-manylinux_2_17_armv7l.manylinux2014_armv7l.whl

Download URL	tokenizers-0.20.1-cp311-cp311-manylinux_2_17_armv7l.manylinux2014_armv7l.whl
Size	2.8 MB
Tags	CPython 3.11 Linux glibc 2.17+ ARMv7l
SHA-256 checksum How to use checksums	`d10766473954397e2d370f215ebed1cc46dcf6fd3906a2a116aa1d6219bfedc3`
BLAKE2b-256 checksum How to use checksums	`3188740a6a069e997dc3e96941083fe3264162f4d198a5e5841acb625f84adbd`
Upload date	Oct 10, 2024
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/5.1.1 CPython/3.12.7`

Release files / tokenizers-0.20.1-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl

Download URL	tokenizers-0.20.1-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Size	2.9 MB
Tags	CPython 3.11 Linux glibc 2.17+ ARM64
SHA-256 checksum How to use checksums	`929c8f3afa16a5130a81ab5079c589226273ec618949cce79b46d96e59a84f61`
BLAKE2b-256 checksum How to use checksums	`2dcaf3a294ed89f2a1b900fba072ef4cb5331d4f156e2d5ea2d34f60160ef5bd`
Upload date	Oct 10, 2024
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/5.1.1 CPython/3.12.7`

Release files / tokenizers-0.20.1-cp311-cp311-macosx_11_0_arm64.whl

Download URL	tokenizers-0.20.1-cp311-cp311-macosx_11_0_arm64.whl
Size	2.6 MB
Tags	CPython 3.11 macOS 11.0+ ARM64
SHA-256 checksum How to use checksums	`3f84dad1ff1863c648d80628b1b55353d16303431283e4efbb6ab1af56a75832`
BLAKE2b-256 checksum How to use checksums	`26a292af8a5f19d0e8bc480759a9975489ebd429b94a81ad46e1422c7927f246`
Upload date	Oct 10, 2024
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/5.1.1 CPython/3.12.7`

Release files / tokenizers-0.20.1-cp311-cp311-macosx_10_12_x86_64.whl

Download URL	tokenizers-0.20.1-cp311-cp311-macosx_10_12_x86_64.whl
Size	2.7 MB
Tags	CPython 3.11 macOS 10.12+ x86-64
SHA-256 checksum How to use checksums	`4a717dcb08f2dabbf27ae4b6b20cbbb2ad7ed78ce05a829fae100ff4b3c7ff15`
BLAKE2b-256 checksum How to use checksums	`619abe5f00cd37ad4fab0e5d1dbf31404a66ac2c1c33973beda9fc8e248a37ab`
Upload date	Oct 10, 2024
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/5.1.1 CPython/3.12.7`

Release files / tokenizers-0.20.1-cp310-none-win_amd64.whl

Download URL	tokenizers-0.20.1-cp310-none-win_amd64.whl
Size	2.4 MB
Tags	CPython 3.10 Windows x86-64
SHA-256 checksum How to use checksums	`f9aa93eacd865f2798b9e62f7ce4533cfff4f5fbd50c02926a78e81c74e432cd`
BLAKE2b-256 checksum How to use checksums	`7adac7728bb6be0ccfbd5662f054ee28d8ba7883558cc9fcd102e6cdce07bbbf`
Upload date	Oct 10, 2024
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/5.1.1 CPython/3.12.7`

Release files / tokenizers-0.20.1-cp310-none-win32.whl

Download URL	tokenizers-0.20.1-cp310-none-win32.whl
Size	2.2 MB
Tags	CPython 3.10 Windows x86-32
SHA-256 checksum How to use checksums	`2847843c53f445e0f19ea842a4e48b89dd0db4e62ba6e1e47a2749d6ec11f50d`
BLAKE2b-256 checksum How to use checksums	`e85422825bc3d00ae8a801314a6d96e7e83c180b626a40299179073364c7eac7`
Upload date	Oct 10, 2024
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/5.1.1 CPython/3.12.7`

Release files / tokenizers-0.20.1-cp310-cp310-musllinux_1_1_x86_64.whl

Download URL	tokenizers-0.20.1-cp310-cp310-musllinux_1_1_x86_64.whl
Size	9.3 MB
Tags	CPython 3.10 Linux musl 1.1+ x86-64
SHA-256 checksum How to use checksums	`a2ffd9a8895575ac636d44500c66dffaef133823b6b25067604fa73bbc5ec09d`
BLAKE2b-256 checksum How to use checksums	`dff78c0ec102f0a723d09347ff6cd617c7e5e8d44efd342305f52a7fcd3e30e2`
Upload date	Oct 10, 2024
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/5.1.1 CPython/3.12.7`

Release files / tokenizers-0.20.1-cp310-cp310-musllinux_1_1_aarch64.whl

Download URL	tokenizers-0.20.1-cp310-cp310-musllinux_1_1_aarch64.whl
Size	9.0 MB
Tags	CPython 3.10 Linux musl 1.1+ ARM64
SHA-256 checksum How to use checksums	`f22dee205329a636148c325921c73cf3e412e87d31f4d9c3153b302a0200057b`
BLAKE2b-256 checksum How to use checksums	`dbb25e45632799d816291de4d04149decf19cf6c2faf42bb99574d80050c87bd`
Upload date	Oct 10, 2024
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/5.1.1 CPython/3.12.7`

Release files / tokenizers-0.20.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl

Download URL	tokenizers-0.20.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Size	3.0 MB
Tags	CPython 3.10 Linux glibc 2.17+ x86-64
SHA-256 checksum How to use checksums	`fc9e95ad49c932b80abfbfeaf63b155761e695ad9f8a58c52a47d962d76e310f`
BLAKE2b-256 checksum How to use checksums	`4b35326b9642307a53b3d9ae145b5c7f157aae9ecaa930888f920124412e0bd2`
Upload date	Oct 10, 2024
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/5.1.1 CPython/3.12.7`

Release files / tokenizers-0.20.1-cp310-cp310-manylinux_2_17_s390x.manylinux2014_s390x.whl

Download URL	tokenizers-0.20.1-cp310-cp310-manylinux_2_17_s390x.manylinux2014_s390x.whl
Size	3.4 MB
Tags	CPython 3.10 Linux glibc 2.17+ IBM System/390x
SHA-256 checksum How to use checksums	`e96f6c14c9752bb82145636b614d5a78e9cde95edfbe0a85dad0dd5ddd6ec95c`
BLAKE2b-256 checksum How to use checksums	`2a393d11780b82d9ba4d8fda093daa48622ed5f2616d6ac8cb638ac290d39d95`
Upload date	Oct 10, 2024
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/5.1.1 CPython/3.12.7`

Release files / tokenizers-0.20.1-cp310-cp310-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl

Download URL	tokenizers-0.20.1-cp310-cp310-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl
Size	3.1 MB
Tags	CPython 3.10 Linux glibc 2.17+ PowerPC 64-le
SHA-256 checksum How to use checksums	`299c85c1d21135bc01542237979bf25c32efa0d66595dd0069ae259b97fb2dbe`
BLAKE2b-256 checksum How to use checksums	`cf03801e91d41e2134a32089af2d382a6c40b3d8b932b42fa96443d77258ab28`
Upload date	Oct 10, 2024
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/5.1.1 CPython/3.12.7`

Release files / tokenizers-0.20.1-cp310-cp310-manylinux_2_17_i686.manylinux2014_i686.whl

Download URL	tokenizers-0.20.1-cp310-cp310-manylinux_2_17_i686.manylinux2014_i686.whl
Size	3.1 MB
Tags	CPython 3.10 Linux glibc 2.17+ x86-32
SHA-256 checksum How to use checksums	`d388d1ea8b7447da784e32e3b86a75cce55887e3b22b31c19d0b186b1c677800`
BLAKE2b-256 checksum How to use checksums	`22096e0a378a35f215b40ae1c04b4d0fe43e9ddfaf3a08a2b7d7fab8953a6587`
Upload date	Oct 10, 2024
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/5.1.1 CPython/3.12.7`

Release files / tokenizers-0.20.1-cp310-cp310-manylinux_2_17_armv7l.manylinux2014_armv7l.whl

Download URL	tokenizers-0.20.1-cp310-cp310-manylinux_2_17_armv7l.manylinux2014_armv7l.whl
Size	2.8 MB
Tags	CPython 3.10 Linux glibc 2.17+ ARMv7l
SHA-256 checksum How to use checksums	`ec870fce1ee5248a10be69f7a8408a234d6f2109f8ea827b4f7ecdbf08c9fd15`
BLAKE2b-256 checksum How to use checksums	`5ff0c1ed45ff90088eba4f15eca9763b5e439cb86b71fc9e66a827318b61e44d`
Upload date	Oct 10, 2024
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/5.1.1 CPython/3.12.7`

Release files / tokenizers-0.20.1-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl

Download URL	tokenizers-0.20.1-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Size	2.9 MB
Tags	CPython 3.10 Linux glibc 2.17+ ARM64
SHA-256 checksum How to use checksums	`b61f561f329ffe4b28367798b89d60c4abf3f815d37413b6352bc6412a359867`
BLAKE2b-256 checksum How to use checksums	`25e25046ad3b0426548b37c96cc4262a7f2ba6ac9593ee10be69effc78a91764`
Upload date	Oct 10, 2024
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/5.1.1 CPython/3.12.7`

Release files / tokenizers-0.20.1-cp310-cp310-macosx_11_0_arm64.whl

Download URL	tokenizers-0.20.1-cp310-cp310-macosx_11_0_arm64.whl
Size	2.6 MB
Tags	CPython 3.10 macOS 11.0+ ARM64
SHA-256 checksum How to use checksums	`03dae629d99068b1ea5416d50de0fea13008f04129cc79af77a2a6392792d93c`
BLAKE2b-256 checksum How to use checksums	`24d4a529aa06db71600c1688210ce035cbff637ece919dcaca599c9235ad832d`
Upload date	Oct 10, 2024
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/5.1.1 CPython/3.12.7`

Release files / tokenizers-0.20.1-cp310-cp310-macosx_10_12_x86_64.whl

Download URL	tokenizers-0.20.1-cp310-cp310-macosx_10_12_x86_64.whl
Size	2.7 MB
Tags	CPython 3.10 macOS 10.12+ x86-64
SHA-256 checksum How to use checksums	`439261da7c0a5c88bda97acb284d49fbdaf67e9d3b623c0bfd107512d22787a9`
BLAKE2b-256 checksum How to use checksums	`72d23c05efeeccefa833b82038ce49ee736756eed10ab66fc723ce423a747b0e`
Upload date	Oct 10, 2024
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/5.1.1 CPython/3.12.7`

Release files / tokenizers-0.20.1-cp39-none-win_amd64.whl

Download URL	tokenizers-0.20.1-cp39-none-win_amd64.whl
Size	2.4 MB
Tags	CPython 3.9 Windows x86-64
SHA-256 checksum How to use checksums	`37d1e6f616c84fceefa7c6484a01df05caf1e207669121c66213cb5b2911d653`
BLAKE2b-256 checksum How to use checksums	`3195ba6461cbb58e38d725564a74ce1ec56c63cb4abf13131ec400522d4f7ce8`
Upload date	Oct 10, 2024
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/5.1.1 CPython/3.12.7`

Release files / tokenizers-0.20.1-cp39-none-win32.whl

Download URL	tokenizers-0.20.1-cp39-none-win32.whl
Size	2.2 MB
Tags	CPython 3.9 Windows x86-32
SHA-256 checksum How to use checksums	`3d4d218573a3d8b121a1f8c801029d70444ffb6d8f129d4cca1c7b672ee4a24c`
BLAKE2b-256 checksum How to use checksums	`68bf911ca80c5928bf8728c7f4d052ab8487512dc34f5ce872dcad0f561734bc`
Upload date	Oct 10, 2024
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/5.1.1 CPython/3.12.7`

Release files / tokenizers-0.20.1-cp39-cp39-musllinux_1_1_x86_64.whl

Download URL	tokenizers-0.20.1-cp39-cp39-musllinux_1_1_x86_64.whl
Size	9.3 MB
Tags	CPython 3.9 Linux musl 1.1+ x86-64
SHA-256 checksum How to use checksums	`42c097390e2f0ed0a5c5d569e6669dd4e9fff7b31c6a5ce6e9c66a61687197de`
BLAKE2b-256 checksum How to use checksums	`8f9ebce3fecf89901de85c12b27430aa61ada6388a0df6b11e8b1a6793399d9c`
Upload date	Oct 10, 2024
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/5.1.1 CPython/3.12.7`

Release files / tokenizers-0.20.1-cp39-cp39-musllinux_1_1_aarch64.whl

Download URL	tokenizers-0.20.1-cp39-cp39-musllinux_1_1_aarch64.whl
Size	9.0 MB
Tags	CPython 3.9 Linux musl 1.1+ ARM64
SHA-256 checksum How to use checksums	`da1001aa46f4490099c82e2facc4fbc06a6a32bf7de3918ba798010954b775e0`
BLAKE2b-256 checksum How to use checksums	`00fd805bbbabe081de0f36120206e1d8c5d408336ff914f3a3b9fa1e77928794`
Upload date	Oct 10, 2024
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/5.1.1 CPython/3.12.7`

Release files / tokenizers-0.20.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl

Download URL	tokenizers-0.20.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Size	3.0 MB
Tags	CPython 3.9 Linux glibc 2.17+ x86-64
SHA-256 checksum How to use checksums	`a908c69c2897a68f412aa05ba38bfa87a02980df70f5a72fa8490479308b1f2d`
BLAKE2b-256 checksum How to use checksums	`63041ce87a3eae86aa4e301e863d42d6907deaf61f3e9178210d9ebe653e948c`
Upload date	Oct 10, 2024
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/5.1.1 CPython/3.12.7`

Release files / tokenizers-0.20.1-cp39-cp39-manylinux_2_17_s390x.manylinux2014_s390x.whl

Download URL	tokenizers-0.20.1-cp39-cp39-manylinux_2_17_s390x.manylinux2014_s390x.whl
Size	3.4 MB
Tags	CPython 3.9 Linux glibc 2.17+ IBM System/390x
SHA-256 checksum How to use checksums	`57b7a8880b208866508b06ce365dc631e7a2472a3faa24daa430d046fb56c885`
BLAKE2b-256 checksum How to use checksums	`9b5f9bfd63eb6e36a42af0b3f9313e66b13ec48c348ddb2e9398560daddd0206`
Upload date	Oct 10, 2024
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/5.1.1 CPython/3.12.7`

Release files / tokenizers-0.20.1-cp39-cp39-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl

Download URL	tokenizers-0.20.1-cp39-cp39-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl
Size	3.1 MB
Tags	CPython 3.9 Linux glibc 2.17+ PowerPC 64-le
SHA-256 checksum How to use checksums	`ce6238a3311bb8e4c15b12600927d35c267b92a52c881ef5717a900ca14793f7`
BLAKE2b-256 checksum How to use checksums	`ba6bf8d3f5eb484ba4ca92d3ce0fd050ed796aec23b28f2582e85e48c76d14d2`
Upload date	Oct 10, 2024
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/5.1.1 CPython/3.12.7`

Release files / tokenizers-0.20.1-cp39-cp39-manylinux_2_17_i686.manylinux2014_i686.whl

Download URL	tokenizers-0.20.1-cp39-cp39-manylinux_2_17_i686.manylinux2014_i686.whl
Size	3.1 MB
Tags	CPython 3.9 Linux glibc 2.17+ x86-32
SHA-256 checksum How to use checksums	`0b4872647ea6f25224e2833b044b0b19084e39400e8ead3cfe751238b0802140`
BLAKE2b-256 checksum How to use checksums	`149d2eb09e02be5c581535efb550ff3920db067526d380a154cdacf3e82a3cb2`
Upload date	Oct 10, 2024
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/5.1.1 CPython/3.12.7`

Release files / tokenizers-0.20.1-cp39-cp39-manylinux_2_17_armv7l.manylinux2014_armv7l.whl

Download URL	tokenizers-0.20.1-cp39-cp39-manylinux_2_17_armv7l.manylinux2014_armv7l.whl
Size	2.8 MB
Tags	CPython 3.9 Linux glibc 2.17+ ARMv7l
SHA-256 checksum How to use checksums	`f326a1ac51ae909b9760e34671c26cd0dfe15662f447302a9d5bb2d872bab8ab`
BLAKE2b-256 checksum How to use checksums	`06b9029fbb476bda0e51b16e8c0bbbef759eb11d4cdbfca7d48b6b971e849270`
Upload date	Oct 10, 2024
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/5.1.1 CPython/3.12.7`

Release files / tokenizers-0.20.1-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl

Download URL	tokenizers-0.20.1-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Size	2.9 MB
Tags	CPython 3.9 Linux glibc 2.17+ ARM64
SHA-256 checksum How to use checksums	`02e18da58cf115b7c40de973609c35bde95856012ba42a41ee919c77935af251`
BLAKE2b-256 checksum How to use checksums	`220c18437327596682446be04597ffb6296cc472886857a057b726bbe7fc18c6`
Upload date	Oct 10, 2024
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/5.1.1 CPython/3.12.7`

Release files / tokenizers-0.20.1-cp39-cp39-macosx_11_0_arm64.whl

Download URL	tokenizers-0.20.1-cp39-cp39-macosx_11_0_arm64.whl
Size	2.6 MB
Tags	CPython 3.9 macOS 11.0+ ARM64
SHA-256 checksum How to use checksums	`a12c3cebb8c92e9c35a23ab10d3852aee522f385c28d0b4fe48c0b7527d59762`
BLAKE2b-256 checksum How to use checksums	`80eb95af53d2072c01c8a6e14114434099370d45344875b53be8df1dfaea9b6d`
Upload date	Oct 10, 2024
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/5.1.1 CPython/3.12.7`

Release files / tokenizers-0.20.1-cp39-cp39-macosx_10_12_x86_64.whl

Download URL	tokenizers-0.20.1-cp39-cp39-macosx_10_12_x86_64.whl
Size	2.7 MB
Tags	CPython 3.9 macOS 10.12+ x86-64
SHA-256 checksum How to use checksums	`a25dcb2f41a0a6aac31999e6c96a75e9152fa0127af8ece46c2f784f23b8197a`
BLAKE2b-256 checksum How to use checksums	`2c7f63333364e618d60d6f82da364e6ea28e09e98f0c479ca4baf09591067cd0`
Upload date	Oct 10, 2024
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/5.1.1 CPython/3.12.7`

Release files / tokenizers-0.20.1-cp38-none-win_amd64.whl

Download URL	tokenizers-0.20.1-cp38-none-win_amd64.whl
Size	2.4 MB
Tags	CPython 3.8 Windows x86-64
SHA-256 checksum How to use checksums	`d412a74cf5b3f68a90c615611a5aa4478bb303d1c65961d22db45001df68afcb`
BLAKE2b-256 checksum How to use checksums	`d2a95a52a6a10064a3a0891e68fe743ee7bbf1f1389311a8b682e4ae50482f4e`
Upload date	Oct 10, 2024
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/5.1.1 CPython/3.12.7`

Release files / tokenizers-0.20.1-cp38-none-win32.whl

Download URL	tokenizers-0.20.1-cp38-none-win32.whl
Size	2.2 MB
Tags	CPython 3.8 Windows x86-32
SHA-256 checksum How to use checksums	`88b3bc76ab4db1ab95ead623d49c95205411e26302cf9f74203e762ac7e85685`
BLAKE2b-256 checksum How to use checksums	`7a2ebb12fc4db04e7b1a5ed4e54846ca582d3ce6c51e653b46492d549be6779e`
Upload date	Oct 10, 2024
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/5.1.1 CPython/3.12.7`

Release files / tokenizers-0.20.1-cp38-cp38-musllinux_1_1_x86_64.whl

Download URL	tokenizers-0.20.1-cp38-cp38-musllinux_1_1_x86_64.whl
Size	9.3 MB
Tags	CPython 3.8 Linux musl 1.1+ x86-64
SHA-256 checksum How to use checksums	`b605c540753e62199bf15cf69c333e934077ef2350262af2ccada46026f83d1c`
BLAKE2b-256 checksum How to use checksums	`74db403bd11c40083572a785c9a4b8dd22a5e67526dba36cc5fc2c277ddcd87d`
Upload date	Oct 10, 2024
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/5.1.1 CPython/3.12.7`

Release files / tokenizers-0.20.1-cp38-cp38-musllinux_1_1_aarch64.whl

Download URL	tokenizers-0.20.1-cp38-cp38-musllinux_1_1_aarch64.whl
Size	9.0 MB
Tags	CPython 3.8 Linux musl 1.1+ ARM64
SHA-256 checksum How to use checksums	`07c4b7be58da142b0730cc4e5fd66bb7bf6f57f4986ddda73833cd39efef8a01`
BLAKE2b-256 checksum How to use checksums	`9a0f0fc165aa7521fccd5130df526657024a39777a19d30b73435053a7b65d7e`
Upload date	Oct 10, 2024
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/5.1.1 CPython/3.12.7`

Release files / tokenizers-0.20.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl

Download URL	tokenizers-0.20.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Size	3.0 MB
Tags	CPython 3.8 Linux glibc 2.17+ x86-64
SHA-256 checksum How to use checksums	`e53975a6694428a0586534cc1354b2408d4e010a3103117f617cbb550299797c`
BLAKE2b-256 checksum How to use checksums	`15e867dc15b6ed227358476fcfa6c2c71e4c508ff86cb6a5996c72e316ebac5b`
Upload date	Oct 10, 2024
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/5.1.1 CPython/3.12.7`

Release files / tokenizers-0.20.1-cp38-cp38-manylinux_2_17_s390x.manylinux2014_s390x.whl

Download URL	tokenizers-0.20.1-cp38-cp38-manylinux_2_17_s390x.manylinux2014_s390x.whl
Size	3.4 MB
Tags	CPython 3.8 Linux glibc 2.17+ IBM System/390x
SHA-256 checksum How to use checksums	`910b96ed87316e4277b23c7bcaf667ce849c7cc379a453fa179e7e09290eeb25`
BLAKE2b-256 checksum How to use checksums	`520a70cbf45dd79f94ebf56580edb9c6d0cd04f402bba9bf8f19b2d316bea79b`
Upload date	Oct 10, 2024
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/5.1.1 CPython/3.12.7`

Release files / tokenizers-0.20.1-cp38-cp38-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl

Download URL	tokenizers-0.20.1-cp38-cp38-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl
Size	3.1 MB
Tags	CPython 3.8 Linux glibc 2.17+ PowerPC 64-le
SHA-256 checksum How to use checksums	`1ba72260449e16c4c2f6f3252823b059fbf2d31b32617e582003f2b18b415c39`
BLAKE2b-256 checksum How to use checksums	`3bcc19e94d7309b0a160014f60e3c2015c3282a1abf9e9e49e91c2b3db110aeb`
Upload date	Oct 10, 2024
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/5.1.1 CPython/3.12.7`

Release files / tokenizers-0.20.1-cp38-cp38-manylinux_2_17_i686.manylinux2014_i686.whl

Download URL	tokenizers-0.20.1-cp38-cp38-manylinux_2_17_i686.manylinux2014_i686.whl
Size	3.1 MB
Tags	CPython 3.8 Linux glibc 2.17+ x86-32
SHA-256 checksum How to use checksums	`7cdf379219e1e1dd432091058dab325a2e6235ebb23e0aec8d0508567c90cd01`
BLAKE2b-256 checksum How to use checksums	`733ed8e5630a032b7d2f81b3bd3527a37747529d68505663e75cdf725b32ebaf`
Upload date	Oct 10, 2024
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/5.1.1 CPython/3.12.7`

Release files / tokenizers-0.20.1-cp38-cp38-manylinux_2_17_armv7l.manylinux2014_armv7l.whl

Download URL	tokenizers-0.20.1-cp38-cp38-manylinux_2_17_armv7l.manylinux2014_armv7l.whl
Size	2.8 MB
Tags	CPython 3.8 Linux glibc 2.17+ ARMv7l
SHA-256 checksum How to use checksums	`a647c5b7cb896d6430cf3e01b4e9a2d77f719c84cefcef825d404830c2071da2`
BLAKE2b-256 checksum How to use checksums	`6dea753829488d580b5784852c2ad56bc73f3c10b13b72483a508d13e006fb9c`
Upload date	Oct 10, 2024
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/5.1.1 CPython/3.12.7`

Release files / tokenizers-0.20.1-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl

Download URL	tokenizers-0.20.1-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Size	2.9 MB
Tags	CPython 3.8 Linux glibc 2.17+ ARM64
SHA-256 checksum How to use checksums	`514cf279b22fa1ae0bc08e143458c74ad3b56cd078b319464959685a35c53d5e`
BLAKE2b-256 checksum How to use checksums	`0f6deb69f3c0fc3ee699794fbd56860926489e8a616aad402760f10a7eec99a4`
Upload date	Oct 10, 2024
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/5.1.1 CPython/3.12.7`

Release files / tokenizers-0.20.1-cp38-cp38-macosx_11_0_arm64.whl

Download URL	tokenizers-0.20.1-cp38-cp38-macosx_11_0_arm64.whl
Size	2.6 MB
Tags	CPython 3.8 macOS 11.0+ ARM64
SHA-256 checksum How to use checksums	`10be14ebd8082086a342d969e17fc2d6edc856c59dbdbddd25f158fa40eaf043`
BLAKE2b-256 checksum How to use checksums	`619b2786d23aa897f186f930205db836425196284784068d0fc5ca1a40ffbf36`
Upload date	Oct 10, 2024
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/5.1.1 CPython/3.12.7`

Release files / tokenizers-0.20.1-cp38-cp38-macosx_10_12_x86_64.whl

Download URL	tokenizers-0.20.1-cp38-cp38-macosx_10_12_x86_64.whl
Size	2.7 MB
Tags	CPython 3.8 macOS 10.12+ x86-64
SHA-256 checksum How to use checksums	`741fb22788482d09d68e73ece1495cfc6d9b29a06c37b3df90564a9cfa688e6d`
BLAKE2b-256 checksum How to use checksums	`f3f961e8c8836d5cbf0402630d1e9eeccaef46b91afbc71b44c0eca2391c13b7`
Upload date	Oct 10, 2024
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/5.1.1 CPython/3.12.7`

Release files / tokenizers-0.20.1-cp37-none-win_amd64.whl

Download URL	tokenizers-0.20.1-cp37-none-win_amd64.whl
Size	2.4 MB
Tags	CPython 3.7 Windows x86-64
SHA-256 checksum How to use checksums	`896195eb9dfdc85c8c052e29947169c1fcbe75a254c4b5792cdbd451587bce85`
BLAKE2b-256 checksum How to use checksums	`1409812a484d7cb22a043446469dfd4f81b57c5c3dd09fb8d8481993301337d7`
Upload date	Oct 10, 2024
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/5.1.1 CPython/3.12.7`

Release files / tokenizers-0.20.1-cp37-none-win32.whl

Download URL	tokenizers-0.20.1-cp37-none-win32.whl
Size	2.2 MB
Tags	CPython 3.7 Windows x86-32
SHA-256 checksum How to use checksums	`212231ab7dfcdc879baf4892ca87c726259fa7c887e1688e3f3cead384d8c305`
BLAKE2b-256 checksum How to use checksums	`9df57d669fb93872795d663736fba03182b3ffe602d68a9e9cf3f37b7a3a75c5`
Upload date	Oct 10, 2024
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/5.1.1 CPython/3.12.7`

Release files / tokenizers-0.20.1-cp37-cp37m-musllinux_1_1_x86_64.whl

Download URL	tokenizers-0.20.1-cp37-cp37m-musllinux_1_1_x86_64.whl
Size	9.3 MB
Tags	CPython 3.7 CPython 3.7 pymalloc Linux musl 1.1+ x86-64
SHA-256 checksum How to use checksums	`05e41e302c315bd2ed86c02e917bf03a6cf7d2f652c9cee1a0eb0d0f1ca0d32c`
BLAKE2b-256 checksum How to use checksums	`b659e24ede3f75425a48974a275193367e79e2b0cfc03c677880e5bca539c768`
Upload date	Oct 10, 2024
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/5.1.1 CPython/3.12.7`

Release files / tokenizers-0.20.1-cp37-cp37m-musllinux_1_1_aarch64.whl

Download URL	tokenizers-0.20.1-cp37-cp37m-musllinux_1_1_aarch64.whl
Size	9.0 MB
Tags	CPython 3.7 CPython 3.7 pymalloc Linux musl 1.1+ ARM64
SHA-256 checksum How to use checksums	`9310951c92c9fb91660de0c19a923c432f110dbfad1a2d429fbc44fa956bf64f`
BLAKE2b-256 checksum How to use checksums	`f1fee373a8d79f5a4c933b254e54e0e12cd3fb5375745f8eb4861cc57e4cef82`
Upload date	Oct 10, 2024
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/5.1.1 CPython/3.12.7`

Release files / tokenizers-0.20.1-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl

Download URL	tokenizers-0.20.1-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Size	3.0 MB
Tags	CPython 3.7 CPython 3.7 pymalloc Linux glibc 2.17+ x86-64
SHA-256 checksum How to use checksums	`14e4cf033a2aa207d7ac790e91adca598b679999710a632c4a494aab0fc3a1b2`
BLAKE2b-256 checksum How to use checksums	`27db80c8bf72f2c40e693c753645149bc1f93100d0c2f62ce16195d7e136c35b`
Upload date	Oct 10, 2024
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/5.1.1 CPython/3.12.7`

Release files / tokenizers-0.20.1-cp37-cp37m-manylinux_2_17_s390x.manylinux2014_s390x.whl

Download URL	tokenizers-0.20.1-cp37-cp37m-manylinux_2_17_s390x.manylinux2014_s390x.whl
Size	3.4 MB
Tags	CPython 3.7 CPython 3.7 pymalloc Linux glibc 2.17+ IBM System/390x
SHA-256 checksum How to use checksums	`094663dd0e85ee2e573126918747bdb40044a848fde388efb5b09d57bc74c680`
BLAKE2b-256 checksum How to use checksums	`aff778cf4837f7282dd2ef4dd45ec988308865fa420439b6d3e6bf3612400897`
Upload date	Oct 10, 2024
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/5.1.1 CPython/3.12.7`

Release files / tokenizers-0.20.1-cp37-cp37m-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl

Download URL	tokenizers-0.20.1-cp37-cp37m-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl
Size	3.1 MB
Tags	CPython 3.7 CPython 3.7 pymalloc Linux glibc 2.17+ PowerPC 64-le
SHA-256 checksum How to use checksums	`bfdad27b0e50544f6b838895a373db6114b85112ba5c0cefadffa78d6daae563`
BLAKE2b-256 checksum How to use checksums	`6064750ab25115c80d58f91d71041213e9fb0e62e1723982c1fa7f0dc36dd4e4`
Upload date	Oct 10, 2024
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/5.1.1 CPython/3.12.7`

Release files / tokenizers-0.20.1-cp37-cp37m-manylinux_2_17_i686.manylinux2014_i686.whl

Download URL	tokenizers-0.20.1-cp37-cp37m-manylinux_2_17_i686.manylinux2014_i686.whl
Size	3.1 MB
Tags	CPython 3.7 CPython 3.7 pymalloc Linux glibc 2.17+ x86-32
SHA-256 checksum How to use checksums	`4b39356df4575d37f9b187bb623aab5abb7b62c8cb702867a1768002f814800c`
BLAKE2b-256 checksum How to use checksums	`6ae30ac76753e00dfbada1dd9cb0cea7a45d1b4951346c26037e267b4f6971d0`
Upload date	Oct 10, 2024
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/5.1.1 CPython/3.12.7`

Release files / tokenizers-0.20.1-cp37-cp37m-manylinux_2_17_armv7l.manylinux2014_armv7l.whl

Download URL	tokenizers-0.20.1-cp37-cp37m-manylinux_2_17_armv7l.manylinux2014_armv7l.whl
Size	2.8 MB
Tags	CPython 3.7 CPython 3.7 pymalloc Linux glibc 2.17+ ARMv7l
SHA-256 checksum How to use checksums	`b8c0fc3542cf9370bf92c932eb71bdeb33d2d4aeeb4126d9fd567b60bd04cb30`
BLAKE2b-256 checksum How to use checksums	`f2f4a8e86da74392855494b0f2fcf5fc92ed59f99e3359cb2ec203cdf8038922`
Upload date	Oct 10, 2024
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/5.1.1 CPython/3.12.7`

Release files / tokenizers-0.20.1-cp37-cp37m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl

Download URL	tokenizers-0.20.1-cp37-cp37m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Size	2.9 MB
Tags	CPython 3.7 CPython 3.7 pymalloc Linux glibc 2.17+ ARM64
SHA-256 checksum How to use checksums	`17f98fccb5c12ab1ce1f471731a9cd86df5d4bd2cf2880c5a66b229802d96145`
BLAKE2b-256 checksum How to use checksums	`500b05742acad333e816c4ffe401226f6259adb6b385286a7cadec76abbf835d`
Upload date	Oct 10, 2024
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/5.1.1 CPython/3.12.7`

Release files / tokenizers-0.20.1-cp37-cp37m-macosx_11_0_arm64.whl

Download URL	tokenizers-0.20.1-cp37-cp37m-macosx_11_0_arm64.whl
Size	2.6 MB
Tags	CPython 3.7 CPython 3.7 pymalloc macOS 11.0+ ARM64
SHA-256 checksum How to use checksums	`65f34e5b731a262dfa562820818533c38ce32a45864437f3d9c82f26c139ca7f`
BLAKE2b-256 checksum How to use checksums	`27eac628b6a5884d2d580cfcc53f613143d8aba1563737ba93f7460b4315f33e`
Upload date	Oct 10, 2024
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/5.1.1 CPython/3.12.7`

Release files / tokenizers-0.20.1-cp37-cp37m-macosx_10_12_x86_64.whl

Download URL	tokenizers-0.20.1-cp37-cp37m-macosx_10_12_x86_64.whl
Size	2.7 MB
Tags	CPython 3.7 CPython 3.7 pymalloc macOS 10.12+ x86-64
SHA-256 checksum How to use checksums	`96af92e833bd44760fb17f23f402e07a66339c1dcbe17d79a9b55bb0cc4f038e`
BLAKE2b-256 checksum How to use checksums	`2a2e33be045c394900cebf33233921b64d5d2a613b52493cedaa0b6f53dd786e`
Upload date	Oct 10, 2024
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/5.1.1 CPython/3.12.7`

Security for tokenizers 0.20.1

Report project as malware

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.