tokenizers

C++ implementations for various tokenizers (sentencepiece, tiktoken etc). Useful for other PyTorch repos such as torchchat, ExecuTorch to build LLM runners using ExecuTorch stack or AOT Inductor stack.

Installation (from source)

git clone git@github.com:meta-pytorch/tokenizers.git
cd ~/tokenizers
git submodule update --init --recursive
pip install -e .

SentencePiece tokenizer

Depend on https://github.com/google/sentencepiece from Google.

Tiktoken tokenizer

Adapted from https://github.com/sewenew/tokenizer.

Huggingface tokenizer

Compatible with https://github.com/huggingface/tokenizers/.

Llama2.c tokenizer

Adapted from https://github.com/karpathy/llama2.c.

Tekken tokenizer

Mistral's Tekken tokenizer (v7) with full support for special tokens, multilingual text, and instruction-tuned conversations. Provides significant efficiency gains for AI workloads:

  • Special token recognition: [INST], [/INST], [AVAILABLE_TOOLS], etc. as single tokens
  • Multilingual support: Complete Unicode handling including emojis and complex scripts
  • Production-ready: 100% decode accuracy with comprehensive test coverage
  • Python bindings: Full compatibility with mistral-common ecosystem

License

tokenizers is released under the BSD 3 license. (Additional code in this distribution is covered by the MIT and Apache Open Source licenses.) However you may have other legal obligations that govern your use of content, such as the terms of service for third-party models.

Project links

Key dates

PyPI data
Data sourced directly from PyPI's database.
  • Released:
Newer release available (1.4.1)

3 maintainers

PyPI data
Data sourced directly from PyPI's database.
Avatar for jacobszwejbka from gravatar.com jacobszwejbka Avatar for larryliu0820 from gravatar.com larryliu0820 Avatar for manuelcandales from gravatar.com manuelcandales

Credits

Author: PyTorch Team

License

BSD License (BSD 3-Clause License)

Requires

Python >=3.10

Tags

pytorch machine learning llm

Classifiers

Report project as malware

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distributions

No source distribution files available for this release.See tutorial on generating distribution archives.

Built Distributions

If you're not sure about the file name format, learn more about wheel file names.

pytorch_tokenizers-1.4.0-cp313-cp313-win_amd64.whl (857.9 kB view details)

Uploaded CPython 3.13Windows x86-64

pytorch_tokenizers-1.4.0-cp313-cp313-manylinux_2_28_x86_64.whl (1.6 MB view details)

Uploaded CPython 3.13manylinux: glibc 2.28+ x86-64

pytorch_tokenizers-1.4.0-cp313-cp313-manylinux_2_28_aarch64.whl (1.5 MB view details)

Uploaded CPython 3.13manylinux: glibc 2.28+ ARM64

pytorch_tokenizers-1.4.0-cp313-cp313-macosx_14_0_arm64.whl (1.1 MB view details)

Uploaded CPython 3.13macOS 14.0+ ARM64

pytorch_tokenizers-1.4.0-cp312-cp312-win_amd64.whl (857.9 kB view details)

Uploaded CPython 3.12Windows x86-64

pytorch_tokenizers-1.4.0-cp312-cp312-manylinux_2_28_x86_64.whl (1.6 MB view details)

Uploaded CPython 3.12manylinux: glibc 2.28+ x86-64

pytorch_tokenizers-1.4.0-cp312-cp312-manylinux_2_28_aarch64.whl (1.5 MB view details)

Uploaded CPython 3.12manylinux: glibc 2.28+ ARM64

pytorch_tokenizers-1.4.0-cp312-cp312-macosx_14_0_arm64.whl (1.1 MB view details)

Uploaded CPython 3.12macOS 14.0+ ARM64

pytorch_tokenizers-1.4.0-cp311-cp311-win_amd64.whl (858.5 kB view details)

Uploaded CPython 3.11Windows x86-64

pytorch_tokenizers-1.4.0-cp311-cp311-manylinux_2_28_x86_64.whl (1.6 MB view details)

Uploaded CPython 3.11manylinux: glibc 2.28+ x86-64

pytorch_tokenizers-1.4.0-cp311-cp311-manylinux_2_28_aarch64.whl (1.5 MB view details)

Uploaded CPython 3.11manylinux: glibc 2.28+ ARM64

pytorch_tokenizers-1.4.0-cp311-cp311-macosx_14_0_arm64.whl (1.1 MB view details)

Uploaded CPython 3.11macOS 14.0+ ARM64

pytorch_tokenizers-1.4.0-cp310-cp310-win_amd64.whl (856.2 kB view details)

Uploaded CPython 3.10Windows x86-64

pytorch_tokenizers-1.4.0-cp310-cp310-manylinux_2_28_x86_64.whl (1.6 MB view details)

Uploaded CPython 3.10manylinux: glibc 2.28+ x86-64

pytorch_tokenizers-1.4.0-cp310-cp310-manylinux_2_28_aarch64.whl (1.5 MB view details)

Uploaded CPython 3.10manylinux: glibc 2.28+ ARM64

pytorch_tokenizers-1.4.0-cp310-cp310-macosx_14_0_arm64.whl (1.1 MB view details)

Uploaded CPython 3.10macOS 14.0+ ARM64

File details

Details for the file pytorch_tokenizers-1.4.0-cp313-cp313-win_amd64.whl.

File metadata

File hashes

Hashes for pytorch_tokenizers-1.4.0-cp313-cp313-win_amd64.whl
Algorithm Hash digest
SHA256 6a1c458af1c3a0c0238d9c229f3e48b1d3d3fbcfc43bc485b9abf9538b16a35f
MD5 400b62692644baf8e191d8140eff98d8
BLAKE2b-256 90876633a147443d261fcfd0264a3f28cd3bfc94efbddf28fe6eae5f9ff1a698

See more details on using hashes here.

File details

Details for the file pytorch_tokenizers-1.4.0-cp313-cp313-manylinux_2_28_x86_64.whl.

File metadata

File hashes

Hashes for pytorch_tokenizers-1.4.0-cp313-cp313-manylinux_2_28_x86_64.whl
Algorithm Hash digest
SHA256 25dc646a8e3198833d8013e1a3c338ec2366c534b53a4bbdb020a27348df5242
MD5 8811ecc6afcc4f5ad56a983c75df968a
BLAKE2b-256 7cdda486cbfcde2256c58d6d5d688b8dc48ef3817439553dc61e9a299c46c809

See more details on using hashes here.

File details

Details for the file pytorch_tokenizers-1.4.0-cp313-cp313-manylinux_2_28_aarch64.whl.

File metadata

File hashes

Hashes for pytorch_tokenizers-1.4.0-cp313-cp313-manylinux_2_28_aarch64.whl
Algorithm Hash digest
SHA256 d161ba1d9d82514ab1b81b25645c6386b83988ba3d4e130d059b0ad7af07ff02
MD5 1ae53d174dd32b8c3777c1813a8aa1f2
BLAKE2b-256 b560aa405671bfefaa9dfc830b36f55df4ea5dc4a44de425d391f6af90f1110a

See more details on using hashes here.

File details

Details for the file pytorch_tokenizers-1.4.0-cp313-cp313-macosx_14_0_arm64.whl.

File metadata

File hashes

Hashes for pytorch_tokenizers-1.4.0-cp313-cp313-macosx_14_0_arm64.whl
Algorithm Hash digest
SHA256 b669db8d2c130a595746e9ea8e780eb39d4e6629b8abc35414c4f654551323ee
MD5 6ecb0c1da8a1676ecac43b04ac702555
BLAKE2b-256 f38f6ede9ece1ecd984bedf8fcc2b64547315c9ec5808f064c0878c85dedf672

See more details on using hashes here.

File details

Details for the file pytorch_tokenizers-1.4.0-cp312-cp312-win_amd64.whl.

File metadata

File hashes

Hashes for pytorch_tokenizers-1.4.0-cp312-cp312-win_amd64.whl
Algorithm Hash digest
SHA256 4ecacf913291452b037254da027acea95350d7413ca40c0ef4f2e7d140d1e4ca
MD5 7268601820636ce735b35efb74daa3b3
BLAKE2b-256 4d222aac59fa6340bb6e43b13a4d547d2baec2789078bc0bef3964b8176bbca4

See more details on using hashes here.

File details

Details for the file pytorch_tokenizers-1.4.0-cp312-cp312-manylinux_2_28_x86_64.whl.

File metadata

File hashes

Hashes for pytorch_tokenizers-1.4.0-cp312-cp312-manylinux_2_28_x86_64.whl
Algorithm Hash digest
SHA256 49242ed7c4fb8e0e64db1b521f1cef1125891e3a77c0c08696ca0bfe9317eaa9
MD5 aedd3816bf9bfc71d2b1b5d46709d17e
BLAKE2b-256 ccf0f8f27e0f14bcca5470ee14fcb4c83fd51da5af1169e26d91d31ba4aa14fd

See more details on using hashes here.

File details

Details for the file pytorch_tokenizers-1.4.0-cp312-cp312-manylinux_2_28_aarch64.whl.

File metadata

File hashes

Hashes for pytorch_tokenizers-1.4.0-cp312-cp312-manylinux_2_28_aarch64.whl
Algorithm Hash digest
SHA256 c70ee197d6c050e8479aeaf3d786fed750f5e5e8b7bff1dd861e68baa08f793c
MD5 80e1e674fb1b0a658d6486080cea4a4b
BLAKE2b-256 3c25b042f28bf32f36c9e3ab01b1fd77a0dba967fc7b0a276c4613924d87c413

See more details on using hashes here.

File details

Details for the file pytorch_tokenizers-1.4.0-cp312-cp312-macosx_14_0_arm64.whl.

File metadata

File hashes

Hashes for pytorch_tokenizers-1.4.0-cp312-cp312-macosx_14_0_arm64.whl
Algorithm Hash digest
SHA256 926e52fbe8ff9a4ed37dca5acaca373433068e3acd0656352452b19abbc20455
MD5 e7f33cdcd0f0f5e53a7acefb3636c05c
BLAKE2b-256 04079a6e9ecf7dc1473c26b53af9cddf23a946b3e9ccaa3a226db3b08c5a9471

See more details on using hashes here.

File details

Details for the file pytorch_tokenizers-1.4.0-cp311-cp311-win_amd64.whl.

File metadata

File hashes

Hashes for pytorch_tokenizers-1.4.0-cp311-cp311-win_amd64.whl
Algorithm Hash digest
SHA256 9ccc1d5a6ef0db3a4d9b354dc1baf75405a7b0aa8b105f50b16db6ef6c119bc5
MD5 e89f6251e50614a377b771c68d59015c
BLAKE2b-256 335d3c492de5b63b8a7460ce7e0491c713524bd409f27599feb04d23b6aa1bb8

See more details on using hashes here.

File details

Details for the file pytorch_tokenizers-1.4.0-cp311-cp311-manylinux_2_28_x86_64.whl.

File metadata

File hashes

Hashes for pytorch_tokenizers-1.4.0-cp311-cp311-manylinux_2_28_x86_64.whl
Algorithm Hash digest
SHA256 489173fcf813dc83308da5e7721168aca39f34e24727ff0c9ae607aa9093d6dd
MD5 88ea7a094db41952eb26ebe5044fed78
BLAKE2b-256 1ef154405861661cf926b65b325f58e0e289fa48a8e60c04933e37fd2ba7e42f

See more details on using hashes here.

File details

Details for the file pytorch_tokenizers-1.4.0-cp311-cp311-manylinux_2_28_aarch64.whl.

File metadata

File hashes

Hashes for pytorch_tokenizers-1.4.0-cp311-cp311-manylinux_2_28_aarch64.whl
Algorithm Hash digest
SHA256 115bde1bf68fa700a3d48d2e9ba806db6cda07586a5e2714155c38f372100700
MD5 3727721ec886ed5dac1b16a4d9a8bf09
BLAKE2b-256 8fc32fe6c3f69e21fa406263e9a7c25a8d1dc08c2876cc8fdbb2b982a2078f35

See more details on using hashes here.

File details

Details for the file pytorch_tokenizers-1.4.0-cp311-cp311-macosx_14_0_arm64.whl.

File metadata

File hashes

Hashes for pytorch_tokenizers-1.4.0-cp311-cp311-macosx_14_0_arm64.whl
Algorithm Hash digest
SHA256 ecd267865ef6c52276ab33cf2682ae05a31b36d34a6b1288b1e5bf211f21f1ba
MD5 0ab532a67b84e31ead340446585e46c2
BLAKE2b-256 21948d623015a92cbb81db94e4db55199b4260331eefd25cce00c673f79db0f5

See more details on using hashes here.

File details

Details for the file pytorch_tokenizers-1.4.0-cp310-cp310-win_amd64.whl.

File metadata

File hashes

Hashes for pytorch_tokenizers-1.4.0-cp310-cp310-win_amd64.whl
Algorithm Hash digest
SHA256 f9ffaf996940dd4551c2044175ea3b3b9b448bbdd5ea64ad650e6ebec4fb6a8e
MD5 861eb89a7fac9a821c87b6a3ce4d5568
BLAKE2b-256 00caf8f3fec24ba3720e9c7aa5b5286e25f5b9b6c7e883fe13e67b1676751e6d

See more details on using hashes here.

File details

Details for the file pytorch_tokenizers-1.4.0-cp310-cp310-manylinux_2_28_x86_64.whl.

File metadata

File hashes

Hashes for pytorch_tokenizers-1.4.0-cp310-cp310-manylinux_2_28_x86_64.whl
Algorithm Hash digest
SHA256 bceabd2db2f0ab090d1c7c882a0bc15cac32f290d7dd67da5114da187089a4e4
MD5 cca40f6fb2e1a7a7c49bf5824e048ccc
BLAKE2b-256 55bb197cfdcbfbb5955770a74d377c7c3267faf0baf310a8eeab461b4014a5f7

See more details on using hashes here.

File details

Details for the file pytorch_tokenizers-1.4.0-cp310-cp310-manylinux_2_28_aarch64.whl.

File metadata

File hashes

Hashes for pytorch_tokenizers-1.4.0-cp310-cp310-manylinux_2_28_aarch64.whl
Algorithm Hash digest
SHA256 2562a504380310d7c593d87a661b4def7aedf76547b41146d455a4fdd6a2736c
MD5 c1ae56650a8728db042641a35ef1a8dd
BLAKE2b-256 36cc9e6bb2f993e69bbe551c50e212de1f3347922705e05d7f4f0dba9eed3c7b

See more details on using hashes here.

File details

Details for the file pytorch_tokenizers-1.4.0-cp310-cp310-macosx_14_0_arm64.whl.

File metadata

File hashes

Hashes for pytorch_tokenizers-1.4.0-cp310-cp310-macosx_14_0_arm64.whl
Algorithm Hash digest
SHA256 590281f7d57d642d70e3b22a919d31534e1931d686f07fce909b90902beb0e48
MD5 e69bea3af79a3369d4c2d4e45cd9406a
BLAKE2b-256 e68ae213602833e4a09ac730cae1869861e2c53f086272e58dfd691d6542ba93

See more details on using hashes here.

Release history Release notifications | RSS feed

1.4.1

20 files

This release

1.4.0 This release

16 files

1.3.0

16 files

1.2.0

16 files

1.1.0

24 files

1.0.1

23 files

Yanked

1.0.0

6 files

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Depot Continuous Integration Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page