Skip to main content

⏳ tiktoken

tiktoken is a fast BPE tokeniser for use with OpenAI's models.

import tiktoken
enc = tiktoken.get_encoding("cl100k_base")
assert enc.decode(enc.encode("hello world")) == "hello world"

# To get the tokeniser corresponding to a specific model in the OpenAI API:
enc = tiktoken.encoding_for_model("gpt-4")

The open source version of tiktoken can be installed from PyPI:

pip install tiktoken

The tokeniser API is documented in tiktoken/core.py.

Example code using tiktoken can be found in the OpenAI Cookbook.

Performance

tiktoken is between 3-6x faster than a comparable open source tokeniser:

image

Performance measured on 1GB of text using the GPT-2 tokeniser, using GPT2TokenizerFast from tokenizers==0.13.2, transformers==4.24.0 and tiktoken==0.2.0.

Getting help

Please post questions in the issue tracker.

If you work at OpenAI, make sure to check the internal documentation or feel free to contact @shantanu.

Extending tiktoken

You may wish to extend tiktoken to support new encodings. There are two ways to do this.

Create your Encoding object exactly the way you want and simply pass it around.

cl100k_base = tiktoken.get_encoding("cl100k_base")

# In production, load the arguments directly instead of accessing private attributes
# See openai_public.py for examples of arguments for specific encodings
enc = tiktoken.Encoding(
    # If you're changing the set of special tokens, make sure to use a different name
    # It should be clear from the name what behaviour to expect.
    name="cl100k_im",
    pat_str=cl100k_base._pat_str,
    mergeable_ranks=cl100k_base._mergeable_ranks,
    special_tokens={
        **cl100k_base._special_tokens,
        "<|im_start|>": 100264,
        "<|im_end|>": 100265,
    }
)

Use the tiktoken_ext plugin mechanism to register your Encoding objects with tiktoken.

This is only useful if you need tiktoken.get_encoding to find your encoding, otherwise prefer option 1.

To do this, you'll need to create a namespace package under tiktoken_ext.

Layout your project like this, making sure to omit the tiktoken_ext/__init__.py file:

my_tiktoken_extension
├── tiktoken_ext
│   └── my_encodings.py
└── setup.py

my_encodings.py should be a module that contains a variable named ENCODING_CONSTRUCTORS. This is a dictionary from an encoding name to a function that takes no arguments and returns arguments that can be passed to tiktoken.Encoding to construct that encoding. For an example, see tiktoken_ext/openai_public.py. For precise details, see tiktoken/registry.py.

Your setup.py should look something like this:

from setuptools import setup, find_namespace_packages

setup(
    name="my_tiktoken_extension",
    packages=find_namespace_packages(include=['tiktoken_ext*']),
    install_requires=["tiktoken"],
    ...
)

Then simply pip install ./my_tiktoken_extension and you should be able to use your custom encodings! Make sure not to use an editable install.

Metadata

Release files for tiktoken 0.3.2

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for tiktoken 0.3.2
File Size Uploaded Provenance Actions
tiktoken-0.3.2.tar.gz 24.9 kB Without provenance Details for tiktoken-0.3.2.tar.gz

Built distributions (wheels)

Table of built distributions (wheels) for tiktoken 0.3.2
File Actions
tiktoken-0.3.2-cp311-cp311-win_amd64.whl CPython 3.11 CPython 3.11 Windows x86-64 Details for tiktoken-0.3.2-cp311-cp311-win_amd64.whl
tiktoken-0.3.2-cp311-cp311-musllinux_1_1_x86_64.whl CPython 3.11 CPython 3.11 Linux musl 1.1+ x86-64 Details for tiktoken-0.3.2-cp311-cp311-musllinux_1_1_x86_64.whl
tiktoken-0.3.2-cp311-cp311-musllinux_1_1_aarch64.whl CPython 3.11 CPython 3.11 Linux musl 1.1+ ARM64 Details for tiktoken-0.3.2-cp311-cp311-musllinux_1_1_aarch64.whl
tiktoken-0.3.2-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl CPython 3.11 CPython 3.11 Linux glibc 2.17+ x86-64 Details for tiktoken-0.3.2-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
tiktoken-0.3.2-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl CPython 3.11 CPython 3.11 Linux glibc 2.17+ ARM64 Details for tiktoken-0.3.2-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
tiktoken-0.3.2-cp311-cp311-macosx_11_0_arm64.whl CPython 3.11 CPython 3.11 macOS 11.0+ ARM64 Details for tiktoken-0.3.2-cp311-cp311-macosx_11_0_arm64.whl
tiktoken-0.3.2-cp311-cp311-macosx_10_9_x86_64.whl CPython 3.11 CPython 3.11 macOS 10.9+ x86-64 Details for tiktoken-0.3.2-cp311-cp311-macosx_10_9_x86_64.whl
tiktoken-0.3.2-cp310-cp310-win_amd64.whl CPython 3.10 CPython 3.10 Windows x86-64 Details for tiktoken-0.3.2-cp310-cp310-win_amd64.whl
tiktoken-0.3.2-cp310-cp310-musllinux_1_1_x86_64.whl CPython 3.10 CPython 3.10 Linux musl 1.1+ x86-64 Details for tiktoken-0.3.2-cp310-cp310-musllinux_1_1_x86_64.whl
tiktoken-0.3.2-cp310-cp310-musllinux_1_1_aarch64.whl CPython 3.10 CPython 3.10 Linux musl 1.1+ ARM64 Details for tiktoken-0.3.2-cp310-cp310-musllinux_1_1_aarch64.whl
tiktoken-0.3.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl CPython 3.10 CPython 3.10 Linux glibc 2.17+ x86-64 Details for tiktoken-0.3.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
tiktoken-0.3.2-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl CPython 3.10 CPython 3.10 Linux glibc 2.17+ ARM64 Details for tiktoken-0.3.2-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
tiktoken-0.3.2-cp310-cp310-macosx_11_0_arm64.whl CPython 3.10 CPython 3.10 macOS 11.0+ ARM64 Details for tiktoken-0.3.2-cp310-cp310-macosx_11_0_arm64.whl
tiktoken-0.3.2-cp310-cp310-macosx_10_9_x86_64.whl CPython 3.10 CPython 3.10 macOS 10.9+ x86-64 Details for tiktoken-0.3.2-cp310-cp310-macosx_10_9_x86_64.whl
tiktoken-0.3.2-cp39-cp39-win_amd64.whl CPython 3.9 CPython 3.9 Windows x86-64 Details for tiktoken-0.3.2-cp39-cp39-win_amd64.whl
tiktoken-0.3.2-cp39-cp39-musllinux_1_1_x86_64.whl CPython 3.9 CPython 3.9 Linux musl 1.1+ x86-64 Details for tiktoken-0.3.2-cp39-cp39-musllinux_1_1_x86_64.whl
tiktoken-0.3.2-cp39-cp39-musllinux_1_1_aarch64.whl CPython 3.9 CPython 3.9 Linux musl 1.1+ ARM64 Details for tiktoken-0.3.2-cp39-cp39-musllinux_1_1_aarch64.whl
tiktoken-0.3.2-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl CPython 3.9 CPython 3.9 Linux glibc 2.17+ x86-64 Details for tiktoken-0.3.2-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
tiktoken-0.3.2-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl CPython 3.9 CPython 3.9 Linux glibc 2.17+ ARM64 Details for tiktoken-0.3.2-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
tiktoken-0.3.2-cp39-cp39-macosx_11_0_arm64.whl CPython 3.9 CPython 3.9 macOS 11.0+ ARM64 Details for tiktoken-0.3.2-cp39-cp39-macosx_11_0_arm64.whl
tiktoken-0.3.2-cp39-cp39-macosx_10_9_x86_64.whl CPython 3.9 CPython 3.9 macOS 10.9+ x86-64 Details for tiktoken-0.3.2-cp39-cp39-macosx_10_9_x86_64.whl
tiktoken-0.3.2-cp38-cp38-win_amd64.whl CPython 3.8 CPython 3.8 Windows x86-64 Details for tiktoken-0.3.2-cp38-cp38-win_amd64.whl
tiktoken-0.3.2-cp38-cp38-musllinux_1_1_x86_64.whl CPython 3.8 CPython 3.8 Linux musl 1.1+ x86-64 Details for tiktoken-0.3.2-cp38-cp38-musllinux_1_1_x86_64.whl
tiktoken-0.3.2-cp38-cp38-musllinux_1_1_aarch64.whl CPython 3.8 CPython 3.8 Linux musl 1.1+ ARM64 Details for tiktoken-0.3.2-cp38-cp38-musllinux_1_1_aarch64.whl
tiktoken-0.3.2-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl CPython 3.8 CPython 3.8 Linux glibc 2.17+ x86-64 Details for tiktoken-0.3.2-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
tiktoken-0.3.2-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl CPython 3.8 CPython 3.8 Linux glibc 2.17+ ARM64 Details for tiktoken-0.3.2-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
tiktoken-0.3.2-cp38-cp38-macosx_11_0_arm64.whl CPython 3.8 CPython 3.8 macOS 11.0+ ARM64 Details for tiktoken-0.3.2-cp38-cp38-macosx_11_0_arm64.whl
tiktoken-0.3.2-cp38-cp38-macosx_10_9_x86_64.whl CPython 3.8 CPython 3.8 macOS 10.9+ x86-64 Details for tiktoken-0.3.2-cp38-cp38-macosx_10_9_x86_64.whl

Total release size: 34.5 MB

Release files / tiktoken-0.3.2.tar.gz

Download URL tiktoken-0.3.2.tar.gz
Size 24.9 kB
Tags Source
SHA-256 checksum
How to use checksums
a51b5449e883e409cf2f4a846a6a97962d5656a354a5532c330811c833ac3b37
BLAKE2b-256 checksum
How to use checksums
91edc6596b42188c03b671ea4d8b6912c1f280ac2991664235039d3a18069d6a
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/4.0.2 CPython/3.10.9

Release files / tiktoken-0.3.2-cp311-cp311-win_amd64.whl

Download URL tiktoken-0.3.2-cp311-cp311-win_amd64.whl
Size 579.0 kB
Tags CPython 3.11 Windows x86-64
SHA-256 checksum
How to use checksums
2f81ed783bbdf00a0646e02f29cfcfccb21ad355bdbb6ed8e67f89dc6e203255
BLAKE2b-256 checksum
How to use checksums
72fab21ed16e1b5219a5bed64fcc95e884f04407a3a4517cfeca201bbc95c8a8
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/4.0.2 CPython/3.10.9

Release files / tiktoken-0.3.2-cp311-cp311-musllinux_1_1_x86_64.whl

Download URL tiktoken-0.3.2-cp311-cp311-musllinux_1_1_x86_64.whl
Size 1.7 MB
Tags CPython 3.11 Linux musl 1.1+ x86-64
SHA-256 checksum
How to use checksums
162fddfa2e8166c42e91cb4cfda0712a4e17311258f865e9cc83c63736ce07fe
BLAKE2b-256 checksum
How to use checksums
fe415dad573f393e4fc21b07a9bde8d80a421a7cb8a4a24920ae014bc5f30765
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/4.0.2 CPython/3.10.9

Release files / tiktoken-0.3.2-cp311-cp311-musllinux_1_1_aarch64.whl

Download URL tiktoken-0.3.2-cp311-cp311-musllinux_1_1_aarch64.whl
Size 1.6 MB
Tags CPython 3.11 Linux musl 1.1+ ARM64
SHA-256 checksum
How to use checksums
1e00bcd088e5c1edf0f86df24097d23dc05dbdafc8a4f5f572ef9051f6a98627
BLAKE2b-256 checksum
How to use checksums
a07bedd9a93f0ece5f5ee0b90c9e1671fae23cc62be7b27868b78f6ac57a3c81
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/4.0.2 CPython/3.10.9

Release files / tiktoken-0.3.2-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl

Download URL tiktoken-0.3.2-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Size 1.7 MB
Tags CPython 3.11 Linux glibc 2.17+ x86-64
SHA-256 checksum
How to use checksums
f577e0ee8fe9d95a8223ffd88c4cdc2617f4b964a306d3a8303fa151a0b9930b
BLAKE2b-256 checksum
How to use checksums
4257a126c01646910746bf4aab29607c6f5cace681ea8cbc96a4b25fff3e6182
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/4.0.2 CPython/3.10.9

Release files / tiktoken-0.3.2-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl

Download URL tiktoken-0.3.2-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Size 1.6 MB
Tags CPython 3.11 Linux glibc 2.17+ ARM64
SHA-256 checksum
How to use checksums
c283730a986fcfee6141196a52a75883e49e45355e2bf3009e9fc530bee798ce
BLAKE2b-256 checksum
How to use checksums
7a792d1edea0aa53df6db5dff4d886107094f9d32d34cc4a4b64cbe2d84cfa3e
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/4.0.2 CPython/3.10.9

Release files / tiktoken-0.3.2-cp311-cp311-macosx_11_0_arm64.whl

Download URL tiktoken-0.3.2-cp311-cp311-macosx_11_0_arm64.whl
Size 701.9 kB
Tags CPython 3.11 macOS 11.0+ ARM64
SHA-256 checksum
How to use checksums
7fbb048a1d01c8b88bbaba882e6d990ae76e61afc25babc84bb38494bd88f6e3
BLAKE2b-256 checksum
How to use checksums
6d72e612936ce12359f263669e32e2b8a9144f8fdae816f111c1ab2c26a9a71c
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/4.0.2 CPython/3.10.9

Release files / tiktoken-0.3.2-cp311-cp311-macosx_10_9_x86_64.whl

Download URL tiktoken-0.3.2-cp311-cp311-macosx_10_9_x86_64.whl
Size 735.2 kB
Tags CPython 3.11 macOS 10.9+ x86-64
SHA-256 checksum
How to use checksums
05942949039fa9347dab08ff6e179912b332ba75539ab17916d52334ff7951f3
BLAKE2b-256 checksum
How to use checksums
5a36694dd635a1b894c2f27c2b4d3786418ac3d63a924cdb52ba8bc4ac2195f7
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/4.0.2 CPython/3.10.9

Release files / tiktoken-0.3.2-cp310-cp310-win_amd64.whl

Download URL tiktoken-0.3.2-cp310-cp310-win_amd64.whl
Size 579.0 kB
Tags CPython 3.10 Windows x86-64
SHA-256 checksum
How to use checksums
40f3f807f5efbeaeb1ef302265954a4f0e8fd00d370ecb5c7560da5a99e98eff
BLAKE2b-256 checksum
How to use checksums
b7d1699eb95dd1ab097ed3e2706219fd5763c8e723e1f5fa2645efe415fe4874
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/4.0.2 CPython/3.10.9

Release files / tiktoken-0.3.2-cp310-cp310-musllinux_1_1_x86_64.whl

Download URL tiktoken-0.3.2-cp310-cp310-musllinux_1_1_x86_64.whl
Size 1.7 MB
Tags CPython 3.10 Linux musl 1.1+ x86-64
SHA-256 checksum
How to use checksums
b0de454c05f19c9d32258194aaf890bf152007b2da8b4b249a10d140c59158cf
BLAKE2b-256 checksum
How to use checksums
426d616dddc3d02229bbc2131c032c2a00ff0332156513b95bc96974c37f0e13
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/4.0.2 CPython/3.10.9

Release files / tiktoken-0.3.2-cp310-cp310-musllinux_1_1_aarch64.whl

Download URL tiktoken-0.3.2-cp310-cp310-musllinux_1_1_aarch64.whl
Size 1.6 MB
Tags CPython 3.10 Linux musl 1.1+ ARM64
SHA-256 checksum
How to use checksums
39ac56147d61ff31cda9853197b0a99cb9c44d63bd53de8fdaad724707b378fb
BLAKE2b-256 checksum
How to use checksums
4b0793fd336923b9e37b3e8618a12ed86928bdf15b041236a35708e638e59277
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/4.0.2 CPython/3.10.9

Release files / tiktoken-0.3.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl

Download URL tiktoken-0.3.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Size 1.7 MB
Tags CPython 3.10 Linux glibc 2.17+ x86-64
SHA-256 checksum
How to use checksums
c05147ec16ecc7cddea4298b69a8e4da171639fe057b2d88f52ebdae5a47b001
BLAKE2b-256 checksum
How to use checksums
3d942537d8f16a980c6f1d92d71ace6d280c6a82f9be4b74ebb0a0f3e2243dbb
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/4.0.2 CPython/3.10.9

Release files / tiktoken-0.3.2-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl

Download URL tiktoken-0.3.2-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Size 1.6 MB
Tags CPython 3.10 Linux glibc 2.17+ ARM64
SHA-256 checksum
How to use checksums
093b3d81adc8e7669b66478a2f68ac609f2c96cc598bdeca58e476677b90b71c
BLAKE2b-256 checksum
How to use checksums
d0cc47d0337a4640985fcaf6cf82e90026f9ad864bfa8c75db75117c13f801d8
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/4.0.2 CPython/3.10.9

Release files / tiktoken-0.3.2-cp310-cp310-macosx_11_0_arm64.whl

Download URL tiktoken-0.3.2-cp310-cp310-macosx_11_0_arm64.whl
Size 701.9 kB
Tags CPython 3.10 macOS 11.0+ ARM64
SHA-256 checksum
How to use checksums
6fffa2e0baf80608eb05f961f5a577272e6d5dc630907eca8fed7781d726fbb3
BLAKE2b-256 checksum
How to use checksums
2776dcb8e376db50c0b715d4ab8c33f63737474a93068596d0ddf23c8c796590
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/4.0.2 CPython/3.10.9

Release files / tiktoken-0.3.2-cp310-cp310-macosx_10_9_x86_64.whl

Download URL tiktoken-0.3.2-cp310-cp310-macosx_10_9_x86_64.whl
Size 735.2 kB
Tags CPython 3.10 macOS 10.9+ x86-64
SHA-256 checksum
How to use checksums
90e6c209421ffc42e309703b5cbcfe3e56fccc3ff31ab4b10e70ff9041268c7b
BLAKE2b-256 checksum
How to use checksums
def2803d434ebe7e89388d609025ee6c85607c71392b7a83e5fc73aef79b7d67
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/4.0.2 CPython/3.10.9

Release files / tiktoken-0.3.2-cp39-cp39-win_amd64.whl

Download URL tiktoken-0.3.2-cp39-cp39-win_amd64.whl
Size 579.3 kB
Tags CPython 3.9 Windows x86-64
SHA-256 checksum
How to use checksums
2d113493c1e7b6db24b194c19992da7bdb1574237a02f8289147e9a69907b6c9
BLAKE2b-256 checksum
How to use checksums
75fc3957ac696ea1d9c4b6ad108ca083a19548f3b120c7ce1f43341b01225c86
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/4.0.2 CPython/3.10.9

Release files / tiktoken-0.3.2-cp39-cp39-musllinux_1_1_x86_64.whl

Download URL tiktoken-0.3.2-cp39-cp39-musllinux_1_1_x86_64.whl
Size 1.7 MB
Tags CPython 3.9 Linux musl 1.1+ x86-64
SHA-256 checksum
How to use checksums
cff38728823adf05d9600cf494d3ccf87d0087286675d5baed8286481da85696
BLAKE2b-256 checksum
How to use checksums
0bbb34cab1cf22c676871f58a684445f5f5061bf2d075a58f9f9515802135065
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/4.0.2 CPython/3.10.9

Release files / tiktoken-0.3.2-cp39-cp39-musllinux_1_1_aarch64.whl

Download URL tiktoken-0.3.2-cp39-cp39-musllinux_1_1_aarch64.whl
Size 1.6 MB
Tags CPython 3.9 Linux musl 1.1+ ARM64
SHA-256 checksum
How to use checksums
9530978b499af8b77291087782673994a32cf80d4bd408c5d928d91631fcede1
BLAKE2b-256 checksum
How to use checksums
98914e7696e6cfe01efe2591c8a24aa40495c1edbf598bee9c05d7db16bd2859
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/4.0.2 CPython/3.10.9

Release files / tiktoken-0.3.2-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl

Download URL tiktoken-0.3.2-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Size 1.7 MB
Tags CPython 3.9 Linux glibc 2.17+ x86-64
SHA-256 checksum
How to use checksums
d76be63a9c10ddca548383d443846e54fe004df544a390424110bf5ec89a2e8f
BLAKE2b-256 checksum
How to use checksums
9ccf9d088e7e653473f2f22a68d125982ba9d6edece84f887c6f4f9c5c90ff11
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/4.0.2 CPython/3.10.9

Release files / tiktoken-0.3.2-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl

Download URL tiktoken-0.3.2-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Size 1.6 MB
Tags CPython 3.9 Linux glibc 2.17+ ARM64
SHA-256 checksum
How to use checksums
55dbb736cd4fcc8f3a78e8c2c53e5ce02f45c706291e60e8663a982e00ad7198
BLAKE2b-256 checksum
How to use checksums
0c034b35cd9ae3c316580d63d4b29dc89b657c760570acee4886a492cc319871
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/4.0.2 CPython/3.10.9

Release files / tiktoken-0.3.2-cp39-cp39-macosx_11_0_arm64.whl

Download URL tiktoken-0.3.2-cp39-cp39-macosx_11_0_arm64.whl
Size 702.8 kB
Tags CPython 3.9 macOS 11.0+ ARM64
SHA-256 checksum
How to use checksums
ecbb54133c232de82715a2a07ae0874fb5d9ea084e0b8034971b7d4cf9260546
BLAKE2b-256 checksum
How to use checksums
91132bb653cc7ba7b39faae709e3f7ea588b505fb0dc43e21af1f3e0bd9a0a34
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/4.0.2 CPython/3.10.9

Release files / tiktoken-0.3.2-cp39-cp39-macosx_10_9_x86_64.whl

Download URL tiktoken-0.3.2-cp39-cp39-macosx_10_9_x86_64.whl
Size 735.2 kB
Tags CPython 3.9 macOS 10.9+ x86-64
SHA-256 checksum
How to use checksums
e56506517aa0d2a57ffd6c1089908776f13404d435e92cc3a21fd031e364eeee
BLAKE2b-256 checksum
How to use checksums
edc1c2a79c55cccdaad6c2070842a4473833018a7aaa0d315b2a9b321867b279
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/4.0.2 CPython/3.10.9

Release files / tiktoken-0.3.2-cp38-cp38-win_amd64.whl

Download URL tiktoken-0.3.2-cp38-cp38-win_amd64.whl
Size 579.3 kB
Tags CPython 3.8 Windows x86-64
SHA-256 checksum
How to use checksums
7654feff0a6b90907016416a91312dc93d19122d9de92d66a7b6ad2308fb6679
BLAKE2b-256 checksum
How to use checksums
bb3700ccaaac85ddf637c1b9982246de8eccad8b6ba41f7a6aefe30dd840d045
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/4.0.2 CPython/3.10.9

Release files / tiktoken-0.3.2-cp38-cp38-musllinux_1_1_x86_64.whl

Download URL tiktoken-0.3.2-cp38-cp38-musllinux_1_1_x86_64.whl
Size 1.7 MB
Tags CPython 3.8 Linux musl 1.1+ x86-64
SHA-256 checksum
How to use checksums
8a93400c84852acde066d66acb89a1ef1f3ce55dbf5f7bbcb70464d4e822c35a
BLAKE2b-256 checksum
How to use checksums
cfd853aca4d97bfa8cbfdecf2b3755baa5c2ebce9439e89b08ac075a05d42e83
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/4.0.2 CPython/3.10.9

Release files / tiktoken-0.3.2-cp38-cp38-musllinux_1_1_aarch64.whl

Download URL tiktoken-0.3.2-cp38-cp38-musllinux_1_1_aarch64.whl
Size 1.6 MB
Tags CPython 3.8 Linux musl 1.1+ ARM64
SHA-256 checksum
How to use checksums
a7ab927d64451b76084fe5276a0a5ea939eed8f2424b92cca0a1888578200f00
BLAKE2b-256 checksum
How to use checksums
a90cceae445bff3621b84cf8a7ba495724bb37445531ca90c16542429fdbeaf8
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/4.0.2 CPython/3.10.9

Release files / tiktoken-0.3.2-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl

Download URL tiktoken-0.3.2-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Size 1.7 MB
Tags CPython 3.8 Linux glibc 2.17+ x86-64
SHA-256 checksum
How to use checksums
63a636f7cfd3eefa9fb3367782a5c9bde1a00b4717931ad31725894ab4b5c74e
BLAKE2b-256 checksum
How to use checksums
8a6d3002988aeb6ae156dc74cee0804eb7c2aa526fdbebadb3f726cdec82184b
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/4.0.2 CPython/3.10.9

Release files / tiktoken-0.3.2-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl

Download URL tiktoken-0.3.2-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Size 1.6 MB
Tags CPython 3.8 Linux glibc 2.17+ ARM64
SHA-256 checksum
How to use checksums
d7839e211a438b74e84ae572b609aad4a149cc7a3092988664fd537519c29036
BLAKE2b-256 checksum
How to use checksums
20145739c8f51f96d4fcf7cc20ba69d339056b3d0082e1a40f3419e1a7a245e2
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/4.0.2 CPython/3.10.9

Release files / tiktoken-0.3.2-cp38-cp38-macosx_11_0_arm64.whl

Download URL tiktoken-0.3.2-cp38-cp38-macosx_11_0_arm64.whl
Size 702.6 kB
Tags CPython 3.8 macOS 11.0+ ARM64
SHA-256 checksum
How to use checksums
5f65b7646bbeeee42f6d65928d6ac72ac275525f4b72ecaf8b660eae96fc90a5
BLAKE2b-256 checksum
How to use checksums
65251b816d3fda693009f7e120be6f586c41e230bb5eed211113d01f99719270
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/4.0.2 CPython/3.10.9

Release files / tiktoken-0.3.2-cp38-cp38-macosx_10_9_x86_64.whl

Download URL tiktoken-0.3.2-cp38-cp38-macosx_10_9_x86_64.whl
Size 735.3 kB
Tags CPython 3.8 macOS 10.9+ x86-64
SHA-256 checksum
How to use checksums
9714e292eab5892094ab4448b904898c6a1098041922fe103e5f3692658d6a46
BLAKE2b-256 checksum
How to use checksums
cc1c86d031a4c2d52752165d9e785bca4ce8f16d2aa501053f2c121583871d11
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/4.0.2 CPython/3.10.9

Security for tiktoken 0.3.2

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page