⏳ tiktoken

tiktoken is a fast BPE tokeniser for use with OpenAI's models.

import tiktoken
enc = tiktoken.get_encoding("o200k_base")
assert enc.decode(enc.encode("hello world")) == "hello world"

# To get the tokeniser corresponding to a specific model in the OpenAI API:
enc = tiktoken.encoding_for_model("gpt-4o")

The open source version of tiktoken can be installed from PyPI:

pip install tiktoken

The tokeniser API is documented in tiktoken/core.py.

Example code using tiktoken can be found in the OpenAI Cookbook.

Performance

tiktoken is between 3-6x faster than a comparable open source tokeniser:

Performance measured on 1GB of text using the GPT-2 tokeniser, using GPT2TokenizerFast from tokenizers==0.13.2, transformers==4.24.0 and tiktoken==0.2.0.

Getting help

Please post questions in the issue tracker.

If you work at OpenAI, make sure to check the internal documentation or feel free to contact @shantanu.

What is BPE anyway?

Language models don't see text like you and I, instead they see a sequence of numbers (known as tokens). Byte pair encoding (BPE) is a way of converting text into tokens. It has a couple desirable properties:

It's reversible and lossless, so you can convert tokens back into the original text
It works on arbitrary text, even text that is not in the tokeniser's training data
It compresses the text: the token sequence is shorter than the bytes corresponding to the original text. On average, in practice, each token corresponds to about 4 bytes.
It attempts to let the model see common subwords. For instance, "ing" is a common subword in English, so BPE encodings will often split "encoding" into tokens like "encod" and "ing" (instead of e.g. "enc" and "oding"). Because the model will then see the "ing" token again and again in different contexts, it helps models generalise and better understand grammar.

tiktoken contains an educational submodule that is friendlier if you want to learn more about the details of BPE, including code that helps visualise the BPE procedure:

from tiktoken._educational import *

# Train a BPE tokeniser on a small amount of text
enc = train_simple_encoding()

# Visualise how the GPT-4 encoder encodes text
enc = SimpleBytePairEncoding.from_tiktoken("cl100k_base")
enc.encode("hello world aaaaaaaaaaaa")

Extending tiktoken

You may wish to extend tiktoken to support new encodings. There are two ways to do this.

Create your Encoding object exactly the way you want and simply pass it around.

cl100k_base = tiktoken.get_encoding("cl100k_base")

# In production, load the arguments directly instead of accessing private attributes
# See openai_public.py for examples of arguments for specific encodings
enc = tiktoken.Encoding(
    # If you're changing the set of special tokens, make sure to use a different name
    # It should be clear from the name what behaviour to expect.
    name="cl100k_im",
    pat_str=cl100k_base._pat_str,
    mergeable_ranks=cl100k_base._mergeable_ranks,
    special_tokens={
        **cl100k_base._special_tokens,
        "<|im_start|>": 100264,
        "<|im_end|>": 100265,
    }
)

Use the tiktoken_ext plugin mechanism to register your Encoding objects with tiktoken.

This is only useful if you need tiktoken.get_encoding to find your encoding, otherwise prefer option 1.

To do this, you'll need to create a namespace package under tiktoken_ext.

Layout your project like this, making sure to omit the tiktoken_ext/__init__.py file:

my_tiktoken_extension
├── tiktoken_ext
│   └── my_encodings.py
└── setup.py

my_encodings.py should be a module that contains a variable named ENCODING_CONSTRUCTORS. This is a dictionary from an encoding name to a function that takes no arguments and returns arguments that can be passed to tiktoken.Encoding to construct that encoding. For an example, see tiktoken_ext/openai_public.py. For precise details, see tiktoken/registry.py.

Your setup.py should look something like this:

from setuptools import setup, find_namespace_packages

setup(
    name="my_tiktoken_extension",
    packages=find_namespace_packages(include=['tiktoken_ext*']),
    install_requires=["tiktoken"],
    ...
)

Then simply pip install ./my_tiktoken_extension and you should be able to use your custom encodings! Make sure not to use an editable install.

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

tiktoken-0.14.0.tar.gz (38.9 kB view details)

Uploaded Aug 17, 2026 Source

Built Distributions

If you're not sure about the file name format, learn more about wheel file names.

The dropdown lists show the available interpreters, ABIs, and platforms. Enable javascript to be able to filter the list of wheel files.

tiktoken-0.14.0-cp315-cp315t-win_amd64.whl (987.9 kB view details)

Uploaded Aug 17, 2026 CPython 3.15tWindows x86-64

tiktoken-0.14.0-cp315-cp315t-musllinux_1_2_x86_64.whl (1.3 MB view details)

Uploaded Aug 17, 2026 CPython 3.15tmusllinux: musl 1.2+ x86-64

tiktoken-0.14.0-cp315-cp315t-musllinux_1_2_aarch64.whl (1.3 MB view details)

Uploaded Aug 17, 2026 CPython 3.15tmusllinux: musl 1.2+ ARM64

tiktoken-0.14.0-cp315-cp315t-manylinux_2_28_x86_64.whl (1.2 MB view details)

Uploaded Aug 17, 2026 CPython 3.15tmanylinux: glibc 2.28+ x86-64

tiktoken-0.14.0-cp315-cp315t-manylinux_2_28_aarch64.whl (1.2 MB view details)

Uploaded Aug 17, 2026 CPython 3.15tmanylinux: glibc 2.28+ ARM64

tiktoken-0.14.0-cp315-cp315t-macosx_11_0_arm64.whl (1.0 MB view details)

Uploaded Aug 17, 2026 CPython 3.15tmacOS 11.0+ ARM64

tiktoken-0.14.0-cp315-cp315t-macosx_10_15_x86_64.whl (1.1 MB view details)

Uploaded Aug 17, 2026 CPython 3.15tmacOS 10.15+ x86-64

tiktoken-0.14.0-cp315-cp315-win_amd64.whl (988.2 kB view details)

Uploaded Aug 17, 2026 CPython 3.15Windows x86-64

tiktoken-0.14.0-cp315-cp315-musllinux_1_2_x86_64.whl (1.3 MB view details)

Uploaded Aug 17, 2026 CPython 3.15musllinux: musl 1.2+ x86-64

tiktoken-0.14.0-cp315-cp315-musllinux_1_2_aarch64.whl (1.3 MB view details)

Uploaded Aug 17, 2026 CPython 3.15musllinux: musl 1.2+ ARM64

tiktoken-0.14.0-cp315-cp315-manylinux_2_28_x86_64.whl (1.2 MB view details)

Uploaded Aug 17, 2026 CPython 3.15manylinux: glibc 2.28+ x86-64

tiktoken-0.14.0-cp315-cp315-manylinux_2_28_aarch64.whl (1.2 MB view details)

Uploaded Aug 17, 2026 CPython 3.15manylinux: glibc 2.28+ ARM64

tiktoken-0.14.0-cp315-cp315-macosx_11_0_arm64.whl (1.0 MB view details)

Uploaded Aug 17, 2026 CPython 3.15macOS 11.0+ ARM64

tiktoken-0.14.0-cp315-cp315-macosx_10_15_x86_64.whl (1.1 MB view details)

Uploaded Aug 17, 2026 CPython 3.15macOS 10.15+ x86-64

tiktoken-0.14.0-cp314-cp314t-win_amd64.whl (987.7 kB view details)

Uploaded Aug 17, 2026 CPython 3.14tWindows x86-64

tiktoken-0.14.0-cp314-cp314t-musllinux_1_2_x86_64.whl (1.3 MB view details)

Uploaded Aug 17, 2026 CPython 3.14tmusllinux: musl 1.2+ x86-64

tiktoken-0.14.0-cp314-cp314t-musllinux_1_2_aarch64.whl (1.3 MB view details)

Uploaded Aug 17, 2026 CPython 3.14tmusllinux: musl 1.2+ ARM64

tiktoken-0.14.0-cp314-cp314t-manylinux_2_28_x86_64.whl (1.2 MB view details)

Uploaded Aug 17, 2026 CPython 3.14tmanylinux: glibc 2.28+ x86-64

tiktoken-0.14.0-cp314-cp314t-manylinux_2_28_aarch64.whl (1.2 MB view details)

Uploaded Aug 17, 2026 CPython 3.14tmanylinux: glibc 2.28+ ARM64

tiktoken-0.14.0-cp314-cp314t-macosx_11_0_arm64.whl (1.0 MB view details)

Uploaded Aug 17, 2026 CPython 3.14tmacOS 11.0+ ARM64

tiktoken-0.14.0-cp314-cp314t-macosx_10_15_x86_64.whl (1.1 MB view details)

Uploaded Aug 17, 2026 CPython 3.14tmacOS 10.15+ x86-64

tiktoken-0.14.0-cp314-cp314-win_amd64.whl (988.0 kB view details)

Uploaded Aug 17, 2026 CPython 3.14Windows x86-64

tiktoken-0.14.0-cp314-cp314-musllinux_1_2_x86_64.whl (1.3 MB view details)

Uploaded Aug 17, 2026 CPython 3.14musllinux: musl 1.2+ x86-64

tiktoken-0.14.0-cp314-cp314-musllinux_1_2_aarch64.whl (1.3 MB view details)

Uploaded Aug 17, 2026 CPython 3.14musllinux: musl 1.2+ ARM64

tiktoken-0.14.0-cp314-cp314-manylinux_2_28_x86_64.whl (1.2 MB view details)

Uploaded Aug 17, 2026 CPython 3.14manylinux: glibc 2.28+ x86-64

tiktoken-0.14.0-cp314-cp314-manylinux_2_28_aarch64.whl (1.2 MB view details)

Uploaded Aug 17, 2026 CPython 3.14manylinux: glibc 2.28+ ARM64

tiktoken-0.14.0-cp314-cp314-macosx_11_0_arm64.whl (1.0 MB view details)

Uploaded Aug 17, 2026 CPython 3.14macOS 11.0+ ARM64

tiktoken-0.14.0-cp314-cp314-macosx_10_15_x86_64.whl (1.1 MB view details)

Uploaded Aug 17, 2026 CPython 3.14macOS 10.15+ x86-64

tiktoken-0.14.0-cp313-cp313-win_amd64.whl (940.9 kB view details)

Uploaded Aug 17, 2026 CPython 3.13Windows x86-64

tiktoken-0.14.0-cp313-cp313-musllinux_1_2_x86_64.whl (1.3 MB view details)

Uploaded Aug 17, 2026 CPython 3.13musllinux: musl 1.2+ x86-64

tiktoken-0.14.0-cp313-cp313-musllinux_1_2_aarch64.whl (1.3 MB view details)

Uploaded Aug 17, 2026 CPython 3.13musllinux: musl 1.2+ ARM64

tiktoken-0.14.0-cp313-cp313-manylinux_2_28_x86_64.whl (1.2 MB view details)

Uploaded Aug 17, 2026 CPython 3.13manylinux: glibc 2.28+ x86-64

tiktoken-0.14.0-cp313-cp313-manylinux_2_28_aarch64.whl (1.2 MB view details)

Uploaded Aug 17, 2026 CPython 3.13manylinux: glibc 2.28+ ARM64

tiktoken-0.14.0-cp313-cp313-macosx_11_0_arm64.whl (1.0 MB view details)

Uploaded Aug 17, 2026 CPython 3.13macOS 11.0+ ARM64

tiktoken-0.14.0-cp313-cp313-macosx_10_13_x86_64.whl (1.1 MB view details)

Uploaded Aug 17, 2026 CPython 3.13macOS 10.13+ x86-64

tiktoken-0.14.0-cp312-cp312-win_amd64.whl (941.2 kB view details)

Uploaded Aug 17, 2026 CPython 3.12Windows x86-64

tiktoken-0.14.0-cp312-cp312-musllinux_1_2_x86_64.whl (1.3 MB view details)

Uploaded Aug 17, 2026 CPython 3.12musllinux: musl 1.2+ x86-64

tiktoken-0.14.0-cp312-cp312-musllinux_1_2_aarch64.whl (1.3 MB view details)

Uploaded Aug 17, 2026 CPython 3.12musllinux: musl 1.2+ ARM64

tiktoken-0.14.0-cp312-cp312-manylinux_2_28_x86_64.whl (1.2 MB view details)

Uploaded Aug 17, 2026 CPython 3.12manylinux: glibc 2.28+ x86-64

tiktoken-0.14.0-cp312-cp312-manylinux_2_28_aarch64.whl (1.2 MB view details)

Uploaded Aug 17, 2026 CPython 3.12manylinux: glibc 2.28+ ARM64

tiktoken-0.14.0-cp312-cp312-macosx_11_0_arm64.whl (1.0 MB view details)

Uploaded Aug 17, 2026 CPython 3.12macOS 11.0+ ARM64

tiktoken-0.14.0-cp312-cp312-macosx_10_13_x86_64.whl (1.1 MB view details)

Uploaded Aug 17, 2026 CPython 3.12macOS 10.13+ x86-64

tiktoken-0.14.0-cp311-cp311-win_amd64.whl (944.4 kB view details)

Uploaded Aug 17, 2026 CPython 3.11Windows x86-64

tiktoken-0.14.0-cp311-cp311-musllinux_1_2_x86_64.whl (1.3 MB view details)

Uploaded Aug 17, 2026 CPython 3.11musllinux: musl 1.2+ x86-64

tiktoken-0.14.0-cp311-cp311-musllinux_1_2_aarch64.whl (1.3 MB view details)

Uploaded Aug 17, 2026 CPython 3.11musllinux: musl 1.2+ ARM64

tiktoken-0.14.0-cp311-cp311-manylinux_2_28_x86_64.whl (1.2 MB view details)

Uploaded Aug 17, 2026 CPython 3.11manylinux: glibc 2.28+ x86-64

tiktoken-0.14.0-cp311-cp311-manylinux_2_28_aarch64.whl (1.2 MB view details)

Uploaded Aug 17, 2026 CPython 3.11manylinux: glibc 2.28+ ARM64

tiktoken-0.14.0-cp311-cp311-macosx_11_0_arm64.whl (1.0 MB view details)

Uploaded Aug 17, 2026 CPython 3.11macOS 11.0+ ARM64

tiktoken-0.14.0-cp311-cp311-macosx_10_12_x86_64.whl (1.1 MB view details)

Uploaded Aug 17, 2026 CPython 3.11macOS 10.12+ x86-64

tiktoken-0.14.0-cp310-cp310-win_amd64.whl (944.1 kB view details)

Uploaded Aug 17, 2026 CPython 3.10Windows x86-64

tiktoken-0.14.0-cp310-cp310-musllinux_1_2_x86_64.whl (1.3 MB view details)

Uploaded Aug 17, 2026 CPython 3.10musllinux: musl 1.2+ x86-64

tiktoken-0.14.0-cp310-cp310-musllinux_1_2_aarch64.whl (1.3 MB view details)

Uploaded Aug 17, 2026 CPython 3.10musllinux: musl 1.2+ ARM64

tiktoken-0.14.0-cp310-cp310-manylinux_2_28_x86_64.whl (1.2 MB view details)

Uploaded Aug 17, 2026 CPython 3.10manylinux: glibc 2.28+ x86-64

tiktoken-0.14.0-cp310-cp310-manylinux_2_28_aarch64.whl (1.2 MB view details)

Uploaded Aug 17, 2026 CPython 3.10manylinux: glibc 2.28+ ARM64

tiktoken-0.14.0-cp310-cp310-macosx_11_0_arm64.whl (1.0 MB view details)

Uploaded Aug 17, 2026 CPython 3.10macOS 11.0+ ARM64

tiktoken-0.14.0-cp310-cp310-macosx_10_12_x86_64.whl (1.1 MB view details)

Uploaded Aug 17, 2026 CPython 3.10macOS 10.12+ x86-64

tiktoken-0.14.0-cp39-cp39-win_amd64.whl (946.5 kB view details)

Uploaded Aug 17, 2026 CPython 3.9Windows x86-64

tiktoken-0.14.0-cp39-cp39-musllinux_1_2_x86_64.whl (1.3 MB view details)

Uploaded Aug 17, 2026 CPython 3.9musllinux: musl 1.2+ x86-64

tiktoken-0.14.0-cp39-cp39-musllinux_1_2_aarch64.whl (1.3 MB view details)

Uploaded Aug 17, 2026 CPython 3.9musllinux: musl 1.2+ ARM64

tiktoken-0.14.0-cp39-cp39-manylinux_2_28_x86_64.whl (1.2 MB view details)

Uploaded Aug 17, 2026 CPython 3.9manylinux: glibc 2.28+ x86-64

tiktoken-0.14.0-cp39-cp39-manylinux_2_28_aarch64.whl (1.2 MB view details)

Uploaded Aug 17, 2026 CPython 3.9manylinux: glibc 2.28+ ARM64

tiktoken-0.14.0-cp39-cp39-macosx_11_0_arm64.whl (1.0 MB view details)

Uploaded Aug 17, 2026 CPython 3.9macOS 11.0+ ARM64

tiktoken-0.14.0-cp39-cp39-macosx_10_12_x86_64.whl (1.1 MB view details)

Uploaded Aug 17, 2026 CPython 3.9macOS 10.12+ x86-64

File details

Details for the file tiktoken-0.14.0.tar.gz.

File metadata

Download URL: tiktoken-0.14.0.tar.gz
Upload date: Aug 17, 2026
Size: 38.9 kB
Tags: Source
Uploaded using Trusted Publishing? No
Uploaded via: twine/7.0.0 CPython/3.14.2

File hashes

Hashes for tiktoken-0.14.0.tar.gz
Algorithm	Hash digest
SHA256	`231dec90efcdccf1b565a1416107736f1e09b1a08fe736ef9d6363e626d03874`
MD5	`9aedbd9239fa1afaa27d571e49a958cb`
BLAKE2b-256	`6662167a842aa0429d45f5e797354fd4343a96f6043d67d0513c675c7b8d36e6`

Algorithm	Hash digest
SHA256	`561e7580f84a79859af1ef6f676968e9030fcc3fe195700b15235bca64f009c9`
MD5	`faa1d0cea2380c199486c621bf9d9499`
BLAKE2b-256	`355e9b01afd037bfa22a0033963fa091e0f75b6fb15cd85bffb42ff86e697323`

Algorithm	Hash digest
SHA256	`149d97453c4c98c04b081d64a85e635921269b532710d6faf81e9e82b790e7d3`
MD5	`2e2bb06a5b4fd655499c5a089129c547`
BLAKE2b-256	`c58450966fb6918a0fb9b32721277e5342bf729a2d74350074d662fbedf9772e`

Algorithm	Hash digest
SHA256	`86f66c85e796f5d05d5c4a60ec1d40cbfebc47a32464053528c797163fa9ab89`
MD5	`a55577c4dcf64176daa65d47688292bd`
BLAKE2b-256	`4ef680760e98a08e6649d2d68afb6035af713121dfb615acce8c4f73810ec438`

Algorithm	Hash digest
SHA256	`78571efc311c30b73f31eb949a921d6dac39a5d9dc42d1cfa8f8db157b3447b1`
MD5	`049b5b56d882345a5c4d3195306c335a`
BLAKE2b-256	`3e882f3f85a968cdc514152129af0a060ebcccb067005a2f29b0d5ef3c838514`

Algorithm	Hash digest
SHA256	`2ea70afba6b9eddbf22c165142e5f0a2ad7aa36a452873c48b57bb2aeb8492ae`
MD5	`46d47916c88d46a8f97d47f070194708`
BLAKE2b-256	`fbbaef427fc638f1439181c5e12dd26b70e881861f89c007aa7e5b36300f8342`

Algorithm	Hash digest
SHA256	`d0781223705199b289faa59601bb9c2441712d4c600dd13c43d8fd6a33d22cd5`
MD5	`b13f7eb78021287a5ed24af5a76ff5ae`
BLAKE2b-256	`6597a5a7bfccf25b1bb65e82bae8edff11ac3c9c041c374b7b4a823d60c38133`

Algorithm	Hash digest
SHA256	`11d8211b290855d2721334ff17dd9b3a17bfb26872be01f25d73612ef7ece890`
MD5	`2c38e0049e07f8942381baa286c4190e`
BLAKE2b-256	`2a005162e90c851a28da18ed382d34898b79a8022548e5619a64e14c03ce7c3d`

Algorithm	Hash digest
SHA256	`1b6e4adcfd285c44502aed51df98aaaca4f0fea028165dbf8a9e857b9f98d8ea`
MD5	`6ce25d789f63ce81ff9ceccee16f847b`
BLAKE2b-256	`02eeddf46ca78e371f5890e96b6e7d089a85b3536432be219851eb0481786ca8`

Algorithm	Hash digest
SHA256	`3c5349c9f916283bba32bec8af69b763e4faa304dc004d0eaaea66a3cf004c1f`
MD5	`4fa9e73cadc38416b453a71e7348e742`
BLAKE2b-256	`0c83467be424746c039c5493c0f4102feab16b9b48eb6f5c089b2a2438e3cde2`

Algorithm	Hash digest
SHA256	`728303a072163130c5b477b1f20d6211895569c1d5302c24ffc93a3009160871`
MD5	`ada11bf50e2a36db9165a1b14b9e53fb`
BLAKE2b-256	`4bd298a38579db25c4a8a84e31dd95d9072ec5f21f7e70de591da0412e29b25b`

Algorithm	Hash digest
SHA256	`60c47ca69ddda0dea8256fffd12e1b86f4b59734a20e4a70c61f63cc5f021df4`
MD5	`5dee00b573811c43e83301604615e4b8`
BLAKE2b-256	`7c9df9c56d7a943a4468abf9ef37661bb9b8e0cd3aa8aa87368c7146cc3f3222`

Algorithm	Hash digest
SHA256	`e4ddf863b59347deaa92302dcd90e5eb003cdc9be06ec2b692c38d1bdd9efd49`
MD5	`8fd2ab85add384d8eac3e6a3a455a9e6`
BLAKE2b-256	`cd7b8537be0836f3df99b2a636b44399bfa43cd757f2b8b4097dacb794cf24a7`

Algorithm	Hash digest
SHA256	`19d643d701fdaa70e5b9c7f8f96abcaffe77ca5e482a3a1a7dde46feb4284695`
MD5	`7a0ca5be8880eaa1dd844c4d3eba126f`
BLAKE2b-256	`c26b09999a9bf1d559670d1680e8f8e419ac0e2c5f6aac82e9bfdf70f260b30a`

Algorithm	Hash digest
SHA256	`14b47e3674f2624803a8acc8fb367b7e24fc53055f9df3296482fe9a3a34a232`
MD5	`82e9b9df6a1939fad90a2b7a6930b0c8`
BLAKE2b-256	`1d10b73b7e319179e0f60b32475f783b044f9cece872c53b6662664e9084b0d0`

Algorithm	Hash digest
SHA256	`7aab286a020660a039097912a088236b985d18a3090d73f136c4413d29d37ca0`
MD5	`78bdb33616376e0ffe27aa891c99ce24`
BLAKE2b-256	`188bba48a73729c9270989b36f37ab2ed5525e52690d715097c9fa791aaa5d05`

Algorithm	Hash digest
SHA256	`ca4db6ff5c5bf600f9b7761a0070ed44dfe5797a76bd432fb978bc480ef40c58`
MD5	`c3d23b3cbfc31813ee676e271c85ed0b`
BLAKE2b-256	`1d4b323631116fc986d9cc5bbeb2b8223c7c85e61a8bb94ea5ab4951023b149b`

Algorithm	Hash digest
SHA256	`2fc834fbe3f6a0736905c36ab709537e6840dbd63b982dc9e0216ae7d305ba1a`
MD5	`3a1050e56815f727492b64e7a9cc5e15`
BLAKE2b-256	`29ceaf8964c38bc8226dd8950305b7a255fa33345d5572f78af7275a313d28e0`

Algorithm	Hash digest
SHA256	`26cc4b4840fa0e9f4b72ed489883e12f57e00d1021ca794720e3c29a12f0edef`
MD5	`261ab513a7a56fc0b364adb843ff07cc`
BLAKE2b-256	`0be846de4400d5bf859f640feee85bd7e32235f68ddf25db53c63be78e581e3a`

Algorithm	Hash digest
SHA256	`e2eca764c53490f8930dbce329e0769f11108d87d908282a80c5c130e26e7037`
MD5	`5d0fdc86ef29e5588f45c9a8220fe822`
BLAKE2b-256	`5a7c3184d17b868456f17b60b1a75f5ec0405618a43aa753336df341d8f11781`

Algorithm	Hash digest
SHA256	`86951a971c53979ec857bd8c4a32dc227ab0fd33f6c12a3bd62d3fbf5f0bfcaa`
MD5	`a8de2a3f2275f9b02b2023e695815540`
BLAKE2b-256	`792946ad8061f57bd9f8b2ea0aa82bf574e0f2aa040b0857a1582adba9957899`

tiktoken 0.14.0

⏳ tiktoken

Performance

Getting help

What is BPE anyway?

Extending tiktoken

Project links

Key dates

1 maintainer

Credits

License

Requires

Provides Extra

Download files

Source Distribution

Built Distributions

File details

File metadata

File hashes

File details

File metadata

File hashes

File details

File metadata

File hashes

File details

File metadata

File hashes

File details

File metadata

File hashes

File details

File metadata

File hashes

File details

File metadata

File hashes

File details

File metadata

File hashes

File details

File metadata

File hashes

File details

File metadata

File hashes

File details

File metadata

File hashes

File details

File metadata

File hashes

File details

File metadata

File hashes

File details

File metadata

File hashes

File details

File metadata

File hashes

File details

File metadata

File hashes

File details

File metadata

File hashes

File details

File metadata

File hashes

File details

File metadata

File hashes

File details

File metadata

File hashes

File details

File metadata

File hashes

File details

Algorithm	Hash digest
SHA256	`6eb94895c45f26bb8f5546e5fd8a069efcf6e3f108ea9d5cbe3bf6f7f3983438`
MD5	`6aad546736024f479cab9fb963faf46d`
BLAKE2b-256	`68c97756717408d3d0dfea3f046c9466144b28afde39ff69d5808f2475dcd7f5`

Algorithm	Hash digest
SHA256	`eceeff0c62419bc78d4b6e70a4762a4d25df3ae8f2d5946e3853ce93e7a57098`
MD5	`75fc7966fef5f77e98d016ed0871e7dd`
BLAKE2b-256	`fcecf5fa35ec13f07279fdcaf3cc9c04bbb154ea591d23978651f2b672593e8a`

Algorithm	Hash digest
SHA256	`2cc19ac87b41c9493c9778ff5847f0c8bbcf5bd0ec6b87ce06c1c802adc8a771`
MD5	`91072459ca764d6271278a98c6655645`
BLAKE2b-256	`ab6b81f158d0f90adb826cd704069c2129a046cb784a2a09861009519fc41cf4`

Algorithm	Hash digest
SHA256	`979c1524f753b662b0f3cd261b135afe6659cce33caaa7a5ea00dd1756b3055c`
MD5	`0812c23fef72366869168e22ff2725a8`
BLAKE2b-256	`e61fbe7cb06ab2108f612f3e92e7b76cf391e192db0db37a984616f0cc32aafc`

Algorithm	Hash digest
SHA256	`e3442bbb2f0c588cec876061e37ae67b455b9df9978b003c8fe30e45f2ef5b42`
MD5	`6b8978f3fbfd04979511cc35c63c53ef`
BLAKE2b-256	`5b7d144af98dc5ad68108451a82e2f5a17f80e2663f5115058b8dfd215c1ad02`

Algorithm	Hash digest
SHA256	`f702e0aeeb6506e57687e881c59e844ebe8f0a6a097ddafe20e3ab25f387be4e`
MD5	`fdb717dc507fcac2444719cdffee4803`
BLAKE2b-256	`892992a1120a12e4bcf2d5464350d1a91b68a433d63ce656bb7f806c27aec09c`

Algorithm	Hash digest
SHA256	`f2af4a336ea56d6c14f27741a0e1d8294a35dd0b038bcf990d232ebb54eb994b`
MD5	`6f26b9d9deae7d7709c49347943d89e8`
BLAKE2b-256	`62852ae74575e321148484147e10b53c3b1717c59ebaa9edb4fe18b1f5c055f8`

Algorithm	Hash digest
SHA256	`e067f4cbcc5d036e8aff7fe7a6b530a8f4de2e4616ad9005a24a1879e24e6450`
MD5	`24f450ef92c9eafb89e7898304cca5f8`
BLAKE2b-256	`59b01cf129f4af8fc513931f931023def596b7c4bfc77026513cd9d851da9e88`

Algorithm	Hash digest
SHA256	`90a762670c7f968184723769a06ed51f5cf5ce5dcd1e30164f25c72d85c2d1f1`
MD5	`91166e694c889b5712f1cbf8cdeeb5ba`
BLAKE2b-256	`c9e0c65603f0c44811def666d3fbf611bf2af3b5e1ef613e06c19411419830b3`

Algorithm	Hash digest
SHA256	`3fd7c14b1cb45b486c39fc9b3443bb341f3e2fc7e6f31247f3435a5836651632`
MD5	`7581f26bd0f335a37e90bbbe5229756c`
BLAKE2b-256	`cb564c12f07b812f84206f38d723eb1ebfdd34bad9309b5dbc0bee6bbcff4cbf`