Tokenizers

Provides an implementation of today's most used tokenizers, with a focus on performance and versatility.

Bindings over the Rust implementation. If you are interested in the High-level design, you can go check it there.

Otherwise, let's dive in!

Main features:

Train new vocabularies and tokenize using 4 pre-made tokenizers (Bert WordPiece and the 3 most common BPE versions).
Extremely fast (both training and tokenization), thanks to the Rust implementation. Takes less than 20 seconds to tokenize a GB of text on a server's CPU.
Easy to use, but also extremely versatile.
Designed for research and production.
Normalization comes with alignments tracking. It's always possible to get the part of the original sentence that corresponds to a given token.
Does all the pre-processing: Truncate, Pad, add the special tokens your model needs.

Installation

With pip:

pip install tokenizers

From sources:

To use this method, you need to have the Rust installed:

# Install with:
curl https://sh.rustup.rs -sSf | sh -s -- -y
export PATH="$HOME/.cargo/bin:$PATH"

Once Rust is installed, you can compile doing the following

git clone https://github.com/huggingface/tokenizers
cd tokenizers/bindings/python

# Create a virtual env (you can use yours as well)
python -m venv .env
source .env/bin/activate

# Install `tokenizers` in the current virtual env
pip install -e .

Free-threaded Python (3.14t)

tokenizers ships dedicated wheels for the free-threaded build of CPython (python3.14t). These wheels declare Py_MOD_GIL_NOT_USED, so importing tokenizers does not force the GIL back on — multi-threaded code stays GIL-free.

The full mutable API works on 3.14t — the same as on regular CPython. Setters are thread-safe: the inner tokenizer state is wrapped in a std::sync::RwLock, so concurrent tokenizer.X = … from multiple threads serialize correctly and concurrent encode operations take a read guard that blocks writers only briefly.

from tokenizers import Tokenizer
from tokenizers.models import BPE
from tokenizers.pre_tokenizers import Whitespace
from tokenizers.processors import ByteLevel

tok = Tokenizer(BPE())
tok.pre_tokenizer = Whitespace()                 # ✅ thread-safe on 3.14t
tok.post_processor = ByteLevel(trim_offsets=True)

Caveat — compound mutations are not atomic. Statements like tokenizer.post_processor.special_tokens = X evaluate in two steps from Python's point of view (read attribute → set attribute on the result). If another thread swaps tokenizer.post_processor between those steps, the mutation lands on an orphaned component. This is the same class of race as dict[k] = v interleaved with dict.clear() — coordinate with a Python lock if you need the compound to be atomic.

For the full thread-safety analysis, see docs/free-threading-audit.md.

Load a pretrained tokenizer from the Hub

from tokenizers import Tokenizer

tokenizer = Tokenizer.from_pretrained("bert-base-cased")

Using the provided Tokenizers

We provide some pre-build tokenizers to cover the most common cases. You can easily load one of these using some vocab.json and merges.txt files:

from tokenizers import CharBPETokenizer

# Initialize a tokenizer
vocab = "./path/to/vocab.json"
merges = "./path/to/merges.txt"
tokenizer = CharBPETokenizer(vocab, merges)

# And then encode:
encoded = tokenizer.encode("I can feel the magic, can you?")
print(encoded.ids)
print(encoded.tokens)

And you can train them just as simply:

from tokenizers import CharBPETokenizer

# Initialize a tokenizer
tokenizer = CharBPETokenizer()

# Then train it!
tokenizer.train([ "./path/to/files/1.txt", "./path/to/files/2.txt" ])

# Now, let's use it:
encoded = tokenizer.encode("I can feel the magic, can you?")

# And finally save it somewhere
tokenizer.save("./path/to/directory/my-bpe.tokenizer.json")

Provided Tokenizers

CharBPETokenizer: The original BPE
ByteLevelBPETokenizer: The byte level version of the BPE
SentencePieceBPETokenizer: A BPE implementation compatible with the one used by SentencePiece
BertWordPieceTokenizer: The famous Bert tokenizer, using WordPiece

All of these can be used and trained as explained above!

Build your own

Whenever these provided tokenizers don't give you enough freedom, you can build your own tokenizer, by putting all the different parts you need together. You can check how we implemented the provided tokenizers and adapt them easily to your own needs.

Building a byte-level BPE

Here is an example showing how to build your own byte-level BPE by putting all the different pieces together, and then saving it to a single file:

from tokenizers import Tokenizer, models, pre_tokenizers, decoders, trainers, processors

# Initialize a tokenizer
tokenizer = Tokenizer(models.BPE())

# Customize pre-tokenization and decoding
tokenizer.pre_tokenizer = pre_tokenizers.ByteLevel(add_prefix_space=True)
tokenizer.decoder = decoders.ByteLevel()
tokenizer.post_processor = processors.ByteLevel(trim_offsets=True)

# And then train
trainer = trainers.BpeTrainer(
    vocab_size=20000,
    min_frequency=2,
    initial_alphabet=pre_tokenizers.ByteLevel.alphabet()
)
tokenizer.train([
    "./path/to/dataset/1.txt",
    "./path/to/dataset/2.txt",
    "./path/to/dataset/3.txt"
], trainer=trainer)

# And Save it
tokenizer.save("byte-level-bpe.tokenizer.json", pretty=True)

Now, when you want to use this tokenizer, this is as simple as:

from tokenizers import Tokenizer

tokenizer = Tokenizer.from_file("byte-level-bpe.tokenizer.json")

encoded = tokenizer.encode("I can feel the magic, can you?")

Typing support and stub generation

The compiled PyO3 extension does not expose type annotations, so editors and type checkers would otherwise see most objects as Any. To provide full typing support, we use a two-step stub generation process:

Rust introspection (tools/stub-gen/): Uses pyo3-introspection to analyze the compiled extension and generate .pyi stub files
Python enrichment (stub.py): Adds docstrings from the runtime module and generates forwarding __init__.py shims

Running stub generation

The easiest way to regenerate stubs is via make style:

cd bindings/python
make style

This will:

Build the extension with maturin develop --release
Run introspection to generate .pyi files
Enrich stubs with docstrings via stub.py
Format with ruff

Running manually

To run the stub generator directly:

cd bindings/python
cargo run --manifest-path tools/stub-gen/Cargo.toml
python stub.py

The stub generator automatically:

Builds the extension using maturin
Copies the built .so to the project root for introspection
Detects and sets PYTHONHOME for embedded Python (handles uv/venv environments)
Generates stubs to py_src/tokenizers/

Troubleshooting

If you encounter Python initialization errors, you can manually set PYTHONHOME:

export PYTHONHOME=$(python3 -c 'import sys; print(sys.base_prefix)')
cargo run --manifest-path tools/stub-gen/Cargo.toml

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

tokenizers-0.23.2.tar.gz (385.7 kB view details)

Uploaded Sep 3, 2026 Source

Built Distributions

If you're not sure about the file name format, learn more about wheel file names.

The dropdown lists show the available interpreters, ABIs, and platforms. Enable javascript to be able to filter the list of wheel files.

tokenizers-0.23.2-cp310-abi3-win_arm64.whl (2.7 MB view details)

Uploaded Sep 3, 2026 CPython 3.10+Windows ARM64

tokenizers-0.23.2-cp310-abi3-win_amd64.whl (2.9 MB view details)

Uploaded Sep 3, 2026 CPython 3.10+Windows x86-64

tokenizers-0.23.2-cp310-abi3-win32.whl (2.6 MB view details)

Uploaded Sep 3, 2026 CPython 3.10+Windows x86

tokenizers-0.23.2-cp310-abi3-musllinux_1_2_x86_64.whl (10.3 MB view details)

Uploaded Sep 3, 2026 CPython 3.10+musllinux: musl 1.2+ x86-64

tokenizers-0.23.2-cp310-abi3-musllinux_1_2_i686.whl (10.1 MB view details)

Uploaded Sep 3, 2026 CPython 3.10+musllinux: musl 1.2+ i686

tokenizers-0.23.2-cp310-abi3-musllinux_1_2_armv7l.whl (9.8 MB view details)

Uploaded Sep 3, 2026 CPython 3.10+musllinux: musl 1.2+ ARMv7l

tokenizers-0.23.2-cp310-abi3-musllinux_1_2_aarch64.whl (10.0 MB view details)

Uploaded Sep 3, 2026 CPython 3.10+musllinux: musl 1.2+ ARM64

tokenizers-0.23.2-cp310-abi3-manylinux_2_31_riscv64.whl (3.6 MB view details)

Uploaded Sep 3, 2026 CPython 3.10+manylinux: glibc 2.31+ riscv64

tokenizers-0.23.2-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.4 MB view details)

Uploaded Sep 3, 2026 CPython 3.10+manylinux: glibc 2.17+ x86-64

tokenizers-0.23.2-cp310-abi3-manylinux_2_17_s390x.manylinux2014_s390x.whl (3.6 MB view details)

Uploaded Sep 3, 2026 CPython 3.10+manylinux: glibc 2.17+ s390x

tokenizers-0.23.2-cp310-abi3-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl (4.0 MB view details)

Uploaded Sep 3, 2026 CPython 3.10+manylinux: glibc 2.17+ ppc64le

tokenizers-0.23.2-cp310-abi3-manylinux_2_17_i686.manylinux2014_i686.whl (3.7 MB view details)

Uploaded Sep 3, 2026 CPython 3.10+manylinux: glibc 2.17+ i686

tokenizers-0.23.2-cp310-abi3-manylinux_2_17_armv7l.manylinux2014_armv7l.whl (3.4 MB view details)

Uploaded Sep 3, 2026 CPython 3.10+manylinux: glibc 2.17+ ARMv7l

tokenizers-0.23.2-cp310-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (3.5 MB view details)

Uploaded Sep 3, 2026 CPython 3.10+manylinux: glibc 2.17+ ARM64

tokenizers-0.23.2-cp310-abi3-macosx_11_0_arm64.whl (3.1 MB view details)

Uploaded Sep 3, 2026 CPython 3.10+macOS 11.0+ ARM64

tokenizers-0.23.2-cp310-abi3-macosx_10_12_x86_64.whl (3.1 MB view details)

Uploaded Sep 3, 2026 CPython 3.10+macOS 10.12+ x86-64

File details

Details for the file tokenizers-0.23.2.tar.gz.

File metadata

Download URL: tokenizers-0.23.2.tar.gz
Upload date: Sep 3, 2026
Size: 385.7 kB
Tags: Source
Uploaded using Trusted Publishing? No
Uploaded via: maturin/1.15.0

File hashes

Hashes for tokenizers-0.23.2.tar.gz
Algorithm	Hash digest
SHA256	`7f0f085686b9de0d0079e6f874ae053600db64c5d13049e0bbc0119926d25aac`
MD5	`7a9fe40e6e796ccb9e35723dac910abf`
BLAKE2b-256	`181ebc6587c5ab643b2e17776cace9070a2ae73549c86bffac9934a600bf3c31`

See more details on using hashes here.

File details

Details for the file tokenizers-0.23.2-cp310-abi3-win_arm64.whl.

File metadata

Download URL: tokenizers-0.23.2-cp310-abi3-win_arm64.whl
Upload date: Sep 3, 2026
Size: 2.7 MB
Tags: CPython 3.10+, Windows ARM64
Uploaded using Trusted Publishing? No
Uploaded via: maturin/1.15.0

File hashes

Hashes for tokenizers-0.23.2-cp310-abi3-win_arm64.whl
Algorithm	Hash digest
SHA256	`e49c394456dd9985787fec76132438ba3fb8911f857b1bf3d40119f9292d41aa`
MD5	`715ea528e48f45cdc1994c7e8e7e3588`
BLAKE2b-256	`d7b0dee84cb44175be1b4c35bd2f770727494e78f0bb38e571a623ade94dbebb`

See more details on using hashes here.

File details

Details for the file tokenizers-0.23.2-cp310-abi3-win_amd64.whl.

File metadata

Download URL: tokenizers-0.23.2-cp310-abi3-win_amd64.whl
Upload date: Sep 3, 2026
Size: 2.9 MB
Tags: CPython 3.10+, Windows x86-64
Uploaded using Trusted Publishing? No
Uploaded via: maturin/1.15.0

File hashes

Hashes for tokenizers-0.23.2-cp310-abi3-win_amd64.whl
Algorithm	Hash digest
SHA256	`2e96f5699d5249c9c64aa8412e044f727aae3a4098cf830f9901ec1afc361cde`
MD5	`4cb546e21ad5491a136167abbda01137`
BLAKE2b-256	`dbf70a69ac6b82dbccf3f71add938a161c497952749294b8dd6dfe03a819dc40`

See more details on using hashes here.

File details

Details for the file tokenizers-0.23.2-cp310-abi3-win32.whl.

File metadata

Download URL: tokenizers-0.23.2-cp310-abi3-win32.whl
Upload date: Sep 3, 2026
Size: 2.6 MB
Tags: CPython 3.10+, Windows x86
Uploaded using Trusted Publishing? No
Uploaded via: maturin/1.15.0

File hashes

Hashes for tokenizers-0.23.2-cp310-abi3-win32.whl
Algorithm	Hash digest
SHA256	`debf978920d93ba9c219bd67cc4bbfaf912c9039e41e7a28b91ec15e3728c95a`
MD5	`9849022a31cf7c12be56f6a2a334982b`
BLAKE2b-256	`f31fc79a01f671a49728ebb0b61f7ff9ea45663b66cab40bc0858e9859b25c16`

See more details on using hashes here.

File details

Details for the file tokenizers-0.23.2-cp310-abi3-musllinux_1_2_x86_64.whl.

File metadata

Download URL: tokenizers-0.23.2-cp310-abi3-musllinux_1_2_x86_64.whl
Upload date: Sep 3, 2026
Size: 10.3 MB
Tags: CPython 3.10+, musllinux: musl 1.2+ x86-64
Uploaded using Trusted Publishing? No
Uploaded via: maturin/1.15.0

File hashes

Hashes for tokenizers-0.23.2-cp310-abi3-musllinux_1_2_x86_64.whl
Algorithm	Hash digest
SHA256	`5c56bda1511921587789163e524d196ed8284174ac23abd7685d5ea8da6c4718`
MD5	`6bd3ad090eeef505574cd9ab031c574e`
BLAKE2b-256	`b5d88e9e4e0b287a338d8f88976729628c9d22e8a54cfaf9777018a7f7cb58a0`

See more details on using hashes here.

File details

Details for the file tokenizers-0.23.2-cp310-abi3-musllinux_1_2_i686.whl.

File metadata

Download URL: tokenizers-0.23.2-cp310-abi3-musllinux_1_2_i686.whl
Upload date: Sep 3, 2026
Size: 10.1 MB
Tags: CPython 3.10+, musllinux: musl 1.2+ i686
Uploaded using Trusted Publishing? No
Uploaded via: maturin/1.15.0

File hashes

Hashes for tokenizers-0.23.2-cp310-abi3-musllinux_1_2_i686.whl
Algorithm	Hash digest
SHA256	`bef235815a067b2648caf6dcc7a71091b0b0fff9ee8057f6451eb9335fae52ef`
MD5	`3dfd7bb61ba92fb6ff4c36d4e249d860`
BLAKE2b-256	`fa737038e612d48bda1599457f712f6bd3854eae1a9dc9c13aa47f835349db48`

See more details on using hashes here.

File details

Details for the file tokenizers-0.23.2-cp310-abi3-musllinux_1_2_armv7l.whl.

File metadata

Download URL: tokenizers-0.23.2-cp310-abi3-musllinux_1_2_armv7l.whl
Upload date: Sep 3, 2026
Size: 9.8 MB
Tags: CPython 3.10+, musllinux: musl 1.2+ ARMv7l
Uploaded using Trusted Publishing? No
Uploaded via: maturin/1.15.0

File hashes

Hashes for tokenizers-0.23.2-cp310-abi3-musllinux_1_2_armv7l.whl
Algorithm	Hash digest
SHA256	`f486f402f6f9abee5bb032553736813af0c710a86b2e0ca592634c55cea1f835`
MD5	`f28aff509a814d42042090796dcdaef5`
BLAKE2b-256	`06013ccb3a956c7528b2507b8a9714155c4baf86af593039db6ea375dd0c96c3`

See more details on using hashes here.

File details

Details for the file tokenizers-0.23.2-cp310-abi3-musllinux_1_2_aarch64.whl.

File metadata

Download URL: tokenizers-0.23.2-cp310-abi3-musllinux_1_2_aarch64.whl
Upload date: Sep 3, 2026
Size: 10.0 MB
Tags: CPython 3.10+, musllinux: musl 1.2+ ARM64
Uploaded using Trusted Publishing? No
Uploaded via: maturin/1.15.0

File hashes

Hashes for tokenizers-0.23.2-cp310-abi3-musllinux_1_2_aarch64.whl
Algorithm	Hash digest
SHA256	`eb2f9c8a24da020ea8c11a01a19c1c2547912d92121ae4a01cfbca46125dee40`
MD5	`26dd59044c09fa50ec1e5718c41c0181`
BLAKE2b-256	`8d6a1552b70fb0d9ab074fd3fc961435d01364e79c9058481822c3af6e8d402c`

See more details on using hashes here.

File details

Details for the file tokenizers-0.23.2-cp310-abi3-manylinux_2_31_riscv64.whl.

File metadata

Download URL: tokenizers-0.23.2-cp310-abi3-manylinux_2_31_riscv64.whl
Upload date: Sep 3, 2026
Size: 3.6 MB
Tags: CPython 3.10+, manylinux: glibc 2.31+ riscv64
Uploaded using Trusted Publishing? No
Uploaded via: maturin/1.15.0

File hashes

Hashes for tokenizers-0.23.2-cp310-abi3-manylinux_2_31_riscv64.whl
Algorithm	Hash digest
SHA256	`12f0835dc2ee694746a76adf7b1567d4346a4a502ebe93fb1f5f80ea49799b78`
MD5	`b2454b49bd928b2198fa91a517980555`
BLAKE2b-256	`e9a44f9106d317b14a80aefea9f0e3a8d07ef25f856a7607eb7f5ab894281fcb`

See more details on using hashes here.

File details

Details for the file tokenizers-0.23.2-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

Download URL: tokenizers-0.23.2-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Upload date: Sep 3, 2026
Size: 3.4 MB
Tags: CPython 3.10+, manylinux: glibc 2.17+ x86-64
Uploaded using Trusted Publishing? No
Uploaded via: maturin/1.15.0

File hashes

Hashes for tokenizers-0.23.2-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm	Hash digest
SHA256	`41c2f84d172449b4dadb9cdc508e3e364076613c35b16e76ecfe47a60d1e3305`
MD5	`363676fcb8abb2d2eed1652c6d71b0f9`
BLAKE2b-256	`2ccaca6b93c7820df123b2662a9469e8facc826ccc94e98fdd0d615f6431e73a`

See more details on using hashes here.

File details

Details for the file tokenizers-0.23.2-cp310-abi3-manylinux_2_17_s390x.manylinux2014_s390x.whl.

File metadata

Download URL: tokenizers-0.23.2-cp310-abi3-manylinux_2_17_s390x.manylinux2014_s390x.whl
Upload date: Sep 3, 2026
Size: 3.6 MB
Tags: CPython 3.10+, manylinux: glibc 2.17+ s390x
Uploaded using Trusted Publishing? No
Uploaded via: maturin/1.15.0

File hashes

Hashes for tokenizers-0.23.2-cp310-abi3-manylinux_2_17_s390x.manylinux2014_s390x.whl
Algorithm	Hash digest
SHA256	`950d7c9426fa72406a0ffeacdbc0bb9985f5db20eb8b263f29c79aaf83105703`
MD5	`aa6a53a0cace7610aefc78880eaf5865`
BLAKE2b-256	`9b8a0175e216f005c2fe08238292663aa41e4c802b216e71047a69a0e9fc6fa3`

See more details on using hashes here.

File details

Details for the file tokenizers-0.23.2-cp310-abi3-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl.

File metadata

Download URL: tokenizers-0.23.2-cp310-abi3-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl
Upload date: Sep 3, 2026
Size: 4.0 MB
Tags: CPython 3.10+, manylinux: glibc 2.17+ ppc64le
Uploaded using Trusted Publishing? No
Uploaded via: maturin/1.15.0

File hashes

Hashes for tokenizers-0.23.2-cp310-abi3-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl
Algorithm	Hash digest
SHA256	`325fee2e0418a9dc6c9ecf736a5f5f0db7875183ace9549ae339da76f7a1fbb7`
MD5	`69bc6a17c7bf2b260c6892ced8b92c21`
BLAKE2b-256	`2db756b84b80bc96942bba8eb23751a9e8a1fce4faaf4390425e7083f721c98c`

See more details on using hashes here.

File details

Details for the file tokenizers-0.23.2-cp310-abi3-manylinux_2_17_i686.manylinux2014_i686.whl.

File metadata

Download URL: tokenizers-0.23.2-cp310-abi3-manylinux_2_17_i686.manylinux2014_i686.whl
Upload date: Sep 3, 2026
Size: 3.7 MB
Tags: CPython 3.10+, manylinux: glibc 2.17+ i686
Uploaded using Trusted Publishing? No
Uploaded via: maturin/1.15.0

File hashes

Hashes for tokenizers-0.23.2-cp310-abi3-manylinux_2_17_i686.manylinux2014_i686.whl
Algorithm	Hash digest
SHA256	`43e4f2071e3cc8d5d86421c874aebc82659bb51a68bcdef5a0da75ee89511ccb`
MD5	`c4ec116066b7778bbc6baa314957ee18`
BLAKE2b-256	`c5bd93c69152d02ef06ce47aed8b2bf4952dcf733c935a62791873932b2934d9`

See more details on using hashes here.

File details

Details for the file tokenizers-0.23.2-cp310-abi3-manylinux_2_17_armv7l.manylinux2014_armv7l.whl.

File metadata

Download URL: tokenizers-0.23.2-cp310-abi3-manylinux_2_17_armv7l.manylinux2014_armv7l.whl
Upload date: Sep 3, 2026
Size: 3.4 MB
Tags: CPython 3.10+, manylinux: glibc 2.17+ ARMv7l
Uploaded using Trusted Publishing? No
Uploaded via: maturin/1.15.0

File hashes

Hashes for tokenizers-0.23.2-cp310-abi3-manylinux_2_17_armv7l.manylinux2014_armv7l.whl
Algorithm	Hash digest
SHA256	`7b7e37ba198f24150f523e1242e83c4970de4a525480586be5dcc24d9add32c5`
MD5	`b731c14dc00e4e171b73569d12f43761`
BLAKE2b-256	`2adee2f14c8919d5bf51874051d00d6c7b7e0e8bde6c6a2dbeddda7f642896ff`

See more details on using hashes here.

File details

Details for the file tokenizers-0.23.2-cp310-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

Download URL: tokenizers-0.23.2-cp310-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Upload date: Sep 3, 2026
Size: 3.5 MB
Tags: CPython 3.10+, manylinux: glibc 2.17+ ARM64
Uploaded using Trusted Publishing? No
Uploaded via: maturin/1.15.0

File hashes

Hashes for tokenizers-0.23.2-cp310-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm	Hash digest
SHA256	`a37039b5dfc4af84eb3ef0a92f4307e28936c8f9adccba2629d36f652e9bf7a2`
MD5	`d195586b6ba1e6aa0dcbd1224c529be1`
BLAKE2b-256	`2e4d8f569ed49372a3ed8e57099bd515055fd48d7c95912c4307cda6973c2168`

See more details on using hashes here.

File details

Details for the file tokenizers-0.23.2-cp310-abi3-macosx_11_0_arm64.whl.

File metadata

Download URL: tokenizers-0.23.2-cp310-abi3-macosx_11_0_arm64.whl
Upload date: Sep 3, 2026
Size: 3.1 MB
Tags: CPython 3.10+, macOS 11.0+ ARM64
Uploaded using Trusted Publishing? No
Uploaded via: maturin/1.15.0

File hashes

Hashes for tokenizers-0.23.2-cp310-abi3-macosx_11_0_arm64.whl
Algorithm	Hash digest
SHA256	`986670e43691469dcee610ea0f846f91a8f84e91fc6f7a48d4c064414c0ec2bf`
MD5	`a187aac18a42c461994823c99cec8547`
BLAKE2b-256	`674922da045a91732384d3a3771816bf188dc5a1f702c32e635afa7c679c0bef`

See more details on using hashes here.

File details

Details for the file tokenizers-0.23.2-cp310-abi3-macosx_10_12_x86_64.whl.

File metadata

Download URL: tokenizers-0.23.2-cp310-abi3-macosx_10_12_x86_64.whl
Upload date: Sep 3, 2026
Size: 3.1 MB
Tags: CPython 3.10+, macOS 10.12+ x86-64
Uploaded using Trusted Publishing? No
Uploaded via: maturin/1.15.0

File hashes

Hashes for tokenizers-0.23.2-cp310-abi3-macosx_10_12_x86_64.whl
Algorithm	Hash digest
SHA256	`85a9a357a3764aecc904ee76bdaf8cf1ad8e5a67a1b929a487c4a39b49ed0e90`
MD5	`3052dc967fe25a0405c3d756dc5485db`
BLAKE2b-256	`4ded8a443528baa6fac8dfe8c3b75b038c63ac92bb539bcabe311e227c718173`