toktok

A fast, exact BPE tokenizer for OpenAI encodings — Rust core, Python bindings. Token ids are byte-identical to tiktoken.

encoder		throughput	CPU per MB	p99 latency
toktok	Rust	99.6 MB/s	0.0117 s	11.7 µs
bpe-openai	Rust	30.2	—	42.6 µs
tiktoken-rs	Rust	7.4	—	164.3 µs
tiktoken	Python	16.2	0.0598 s	156.3 µs

_{The Pile, 25 MB, cl100k_base, single thread on a GitHub runner. Every
encoder's ids are verified identical before timing. Full tables, memory figures
and method: docs/BENCHMARKS.md.}

Encodings: cl100k_base (GPT-3.5/GPT-4), o200k_base (GPT-4o), o200k_harmony (GPT-OSS). They are compiled into the binary — no data files, no downloads, no runtime dependencies.

Python

uv add toktok-rs        # or: pip install toktok-rs

One function. Give it your texts and an encoding, get a token count per text:

import toktok

toktok.batch_count(["hello world", "how many tokens is this?"])
# [2, 6]

toktok.batch_count(docs, "gpt-4o")                    # model names work too
toktok.batch_count(docs, "cl100k_base", threads=4)    # 0 = every core (default)
toktok.batch_count(["a<|endoftext|>b"], with_special=True)   # [3]

`batch_count(texts, encoding="cl100k_base", threads=0, with_special=False)`

Returns list[int] — one count per text, in the order given.

parameter	default	what it does
`texts`	—	any iterable of `str`. An empty iterable returns `[]`; an empty string counts as `0`.
`encoding`	`"cl100k_base"`	which tokenizer to count with. Takes an encoding name — `cl100k_base`, `o200k_base`, `o200k_harmony` — or a model name, which resolves to that model's encoding: `gpt-4o`, `gpt-4`, `gpt-3.5-turbo`, `o1`, `o3`, `text-embedding-3-small`, `openai/gpt-oss-20b`. An org prefix is stripped, so HF-style ids work. Anything unrecognized raises `KeyError`.
`threads`	`0`	worker threads. `0` uses every core; `1` counts on the calling thread; any other number caps the pool. Counting releases the GIL, so this scales — including on free-threaded builds. Threads are only worth it for large batches; for a handful of short strings the pool costs more than it saves.
`with_special`	`False`	how to treat special-token strings. `False` counts `<\|endoftext\|>` as the ordinary text it looks like (7 tokens); `True` counts it as the single special token it is (1). Use `True` when your text already contains rendered chat/control markup and you want the count the model will see.

Ids are never built: one scratch buffer per thread is reused across texts, so a batch of any size allocates O(threads), not O(tokens).

Wheels cover CPython 3.11–3.14 plus free-threaded 3.14t, on Linux (x86_64, aarch64), macOS and Windows.

Need ids, decoding or offsets? toktok._encoding(name) returns the full tokenizer — see docs/PYTHON.md.

Rust

cargo add toktok-rs     # the crate is toktok-rs, the library is toktok

let tok = toktok::Tokenizer::builtin("cl100k_base")?;   // nothing to download

let ids = tok.encode(b"hello world");
let text = tok.decode(&ids);                 // lossless, even on invalid UTF-8
let n = tok.count(b"how many tokens is this?");

// batch APIs are parallel; 0 threads means every core
let docs: Vec<&[u8]> = vec![b"first", b"second"];
let counts = tok.count_batch(&docs, 0, false);   // no ids materialized
let batches = tok.encode_batch(&docs, 0, false);

let (ids, spans) = tok.encode_with_offsets(b"per-token byte spans");

Every batch method takes the same two knobs as the Python API:

argument	meaning
`threads: usize`	`0` uses every core; `1` runs on the calling thread; anything else caps the pool.
`with_special: bool`	`false` counts/encodes special strings as ordinary text; `true` maps them to their single special id.

Tokenizer is Send + Sync — load it once and share it. Full API on docs.rs/toktok-rs.

Cargo features

feature	default	what it does
`embedded-data`	on	compiles the vocabularies into the binary (~4 MB of `.rodata`), so `Tokenizer::builtin()` needs no files at runtime. Turn it off (`default-features = false`) to drop them and load from a directory with `Tokenizer::load_dir(dir, name)` instead.

docs/BENCHMARKS.md — full results, memory and latency profile, how to reproduce
docs/PYTHON.md — the tokenizer behind batch_count
docs/DEVELOPMENT.md — layout, tests, how it's fast
RELEASING.md — scripts/release.sh patch|minor|major|X.Y.Z, and the registry setup

License

MIT — see LICENSE. Ported from quicktok (MIT); vocabulary data derives from tiktoken (MIT). See NOTICE.

Metadata

Release files for toktok-rs 0.1.1

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for toktok-rs 0.1.1
File	Size	Uploaded
toktok_rs-0.1.1.tar.gz	2.2 MB	Sep 2, 2026	Details

Built distributions (wheels)

Table of built distributions (wheels) for toktok-rs 0.1.1
File	Interpreter	ABI	Platform	Reset
toktok_rs-0.1.1-cp314-cp314t-win_amd64.whl 2.4 MB Sep 2, 2026	CPython 3.14	CPython 3.14 free-threading	Windows x86-64	Details
toktok_rs-0.1.1-cp314-cp314t-manylinux_2_28_x86_64.whl 2.6 MB Sep 2, 2026	CPython 3.14	CPython 3.14 free-threading	Linux glibc 2.28+ x86-64	Details
toktok_rs-0.1.1-cp314-cp314t-manylinux_2_28_aarch64.whl 2.5 MB Sep 2, 2026	CPython 3.14	CPython 3.14 free-threading	Linux glibc 2.28+ ARM64	Details
toktok_rs-0.1.1-cp314-cp314t-macosx_10_12_x86_64.macosx_11_0_arm64.macosx_10_12_universal2.whl 5.1 MB Sep 2, 2026	CPython 3.14	CPython 3.14 free-threading	macOS 11.0+ ARM64, macOS 10.12+ universal2 (ARM64, x86-64), macOS 10.12+ x86-64	Details
toktok_rs-0.1.1-cp311-abi3-win_amd64.whl 2.4 MB Sep 2, 2026	CPython 3.11	abi3	Windows x86-64	Details
toktok_rs-0.1.1-cp311-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl 2.6 MB Sep 2, 2026	CPython 3.11	abi3	Linux glibc 2.17+ x86-64	Details
toktok_rs-0.1.1-cp311-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl 2.5 MB Sep 2, 2026	CPython 3.11	abi3	Linux glibc 2.17+ ARM64	Details
toktok_rs-0.1.1-cp311-abi3-macosx_10_12_x86_64.macosx_11_0_arm64.macosx_10_12_universal2.whl 5.1 MB Sep 2, 2026	CPython 3.11	abi3	macOS 10.12+ x86-64, macOS 10.12+ universal2 (ARM64, x86-64), macOS 11.0+ ARM64	Details

Total release size: 27.4 MB

Release files / toktok_rs-0.1.1.tar.gz

Download URL	toktok_rs-0.1.1.tar.gz
Size	2.2 MB
Tags	Source
SHA-256 checksum How to use checksums	`5e4852957bba5f2a8ed26a0e035348deb239c369a671c7a301b669e97d1b580d`
BLAKE2b-256 checksum How to use checksums	`238fff2572aeae0849513c3643809f199497da3b6d034467b619fb8cdfd4f0bd`
Upload date	Sep 2, 2026
Uploaded using Trusted Publishing? What is trusted publishing?	Yes
Uploaded via	`twine/7.0.0 CPython/3.13.14`

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Identity

Publishing platform: GitHub Actions
Publishing repository: github.com/vectorize-io/toktok
Publishing commit: github.com/vectorize-io/toktok/tree/54a013f27cda577dd506d7ccb6cb8d9ff993052d

Workflow

Publishing configuration: github.com/vectorize-io/toktok/blob/54a013f27cda577dd506d7ccb6cb8d9ff993052d/.github/workflows/release.yml
Publishing logs: github.com/vectorize-io/toktok/actions/runs/33600816129/attempts/1

Artifact

Subject: toktok_rs-0.1.1.tar.gz
SHA-256 checksum: 5e4852957bba5f2a8ed26a0e035348deb239c369a671c7a301b669e97d1b580d Verifying attestations

Release files / toktok_rs-0.1.1-cp314-cp314t-win_amd64.whl

Download URL	toktok_rs-0.1.1-cp314-cp314t-win_amd64.whl
Size	2.4 MB
Tags	CPython 3.14 CPython 3.14 free-threading Windows x86-64
SHA-256 checksum How to use checksums	`99da238f2186408438e8654c87c33f7e36ba7845f62b9908f7483f47595fcbe8`
BLAKE2b-256 checksum How to use checksums	`7adcad9557db2fdf2dc66d85275b75492e4bee55db61bcd106348a5c43bf0613`
Upload date	Sep 2, 2026
Uploaded using Trusted Publishing? What is trusted publishing?	Yes
Uploaded via	`twine/7.0.0 CPython/3.13.14`

Provenance

Identity

Publishing platform: GitHub Actions
Publishing repository: github.com/vectorize-io/toktok
Publishing commit: github.com/vectorize-io/toktok/tree/54a013f27cda577dd506d7ccb6cb8d9ff993052d

Workflow

Publishing configuration: github.com/vectorize-io/toktok/blob/54a013f27cda577dd506d7ccb6cb8d9ff993052d/.github/workflows/release.yml
Publishing logs: github.com/vectorize-io/toktok/actions/runs/33600816129/attempts/1

Artifact

Subject: toktok_rs-0.1.1-cp314-cp314t-win_amd64.whl
SHA-256 checksum: 99da238f2186408438e8654c87c33f7e36ba7845f62b9908f7483f47595fcbe8 Verifying attestations

Release files / toktok_rs-0.1.1-cp314-cp314t-manylinux_2_28_x86_64.whl

Download URL	toktok_rs-0.1.1-cp314-cp314t-manylinux_2_28_x86_64.whl
Size	2.6 MB
Tags	CPython 3.14 CPython 3.14 free-threading Linux glibc 2.28+ x86-64
SHA-256 checksum How to use checksums	`97277904ea7ac90aefb7905f9eb27351d2d0fcec6c73cedac9111962c078e702`
BLAKE2b-256 checksum How to use checksums	`0c2c95616cd5dbb65f1d2da18a2900913f83f925a395856dc4cdaf95e1d1502d`
Upload date	Sep 2, 2026
Uploaded using Trusted Publishing? What is trusted publishing?	Yes
Uploaded via	`twine/7.0.0 CPython/3.13.14`

Provenance

Identity

Publishing platform: GitHub Actions
Publishing repository: github.com/vectorize-io/toktok
Publishing commit: github.com/vectorize-io/toktok/tree/54a013f27cda577dd506d7ccb6cb8d9ff993052d

Workflow

Publishing configuration: github.com/vectorize-io/toktok/blob/54a013f27cda577dd506d7ccb6cb8d9ff993052d/.github/workflows/release.yml
Publishing logs: github.com/vectorize-io/toktok/actions/runs/33600816129/attempts/1

Artifact

Subject: toktok_rs-0.1.1-cp314-cp314t-manylinux_2_28_x86_64.whl
SHA-256 checksum: 97277904ea7ac90aefb7905f9eb27351d2d0fcec6c73cedac9111962c078e702 Verifying attestations

Release files / toktok_rs-0.1.1-cp314-cp314t-manylinux_2_28_aarch64.whl

Download URL	toktok_rs-0.1.1-cp314-cp314t-manylinux_2_28_aarch64.whl
Size	2.5 MB
Tags	CPython 3.14 CPython 3.14 free-threading Linux glibc 2.28+ ARM64
SHA-256 checksum How to use checksums	`bc7bcd802f7f38dc18e003b0ae8e9a6052b9d60a0212caf679d90996d22321a8`
BLAKE2b-256 checksum How to use checksums	`1daca51166786c66dcccf8fff4d4ca9a0fe6df4bf19a6c554a161e062b3cccc5`
Upload date	Sep 2, 2026
Uploaded using Trusted Publishing? What is trusted publishing?	Yes
Uploaded via	`twine/7.0.0 CPython/3.13.14`

Provenance

Identity

Publishing platform: GitHub Actions
Publishing repository: github.com/vectorize-io/toktok
Publishing commit: github.com/vectorize-io/toktok/tree/54a013f27cda577dd506d7ccb6cb8d9ff993052d

Workflow

Publishing configuration: github.com/vectorize-io/toktok/blob/54a013f27cda577dd506d7ccb6cb8d9ff993052d/.github/workflows/release.yml
Publishing logs: github.com/vectorize-io/toktok/actions/runs/33600816129/attempts/1

Artifact

Subject: toktok_rs-0.1.1-cp314-cp314t-manylinux_2_28_aarch64.whl
SHA-256 checksum: bc7bcd802f7f38dc18e003b0ae8e9a6052b9d60a0212caf679d90996d22321a8 Verifying attestations

Release files / toktok_rs-0.1.1-cp314-cp314t-macosx_10_12_x86_64.macosx_11_0_arm64.macosx_10_12_universal2.whl

Download URL	toktok_rs-0.1.1-cp314-cp314t-macosx_10_12_x86_64.macosx_11_0_arm64.macosx_10_12_universal2.whl
Size	5.1 MB
Tags	CPython 3.14 CPython 3.14 free-threading macOS 10.12+ universal2 (ARM64, x86-64) macOS 10.12+ x86-64 macOS 11.0+ ARM64
SHA-256 checksum How to use checksums	`efb11b3f6107cec14edb38ee210177e13df52cfe3e9e610bb500d72f91474618`
BLAKE2b-256 checksum How to use checksums	`7c350c25398d2672c6ade1fe6acfbdac657e5462e0bfb6a54d9f5deec774d053`
Upload date	Sep 2, 2026
Uploaded using Trusted Publishing? What is trusted publishing?	Yes
Uploaded via	`twine/7.0.0 CPython/3.13.14`

Provenance

Identity

Publishing platform: GitHub Actions
Publishing repository: github.com/vectorize-io/toktok
Publishing commit: github.com/vectorize-io/toktok/tree/54a013f27cda577dd506d7ccb6cb8d9ff993052d

Workflow

Publishing configuration: github.com/vectorize-io/toktok/blob/54a013f27cda577dd506d7ccb6cb8d9ff993052d/.github/workflows/release.yml
Publishing logs: github.com/vectorize-io/toktok/actions/runs/33600816129/attempts/1

Artifact

Subject: toktok_rs-0.1.1-cp314-cp314t-macosx_10_12_x86_64.macosx_11_0_arm64.macosx_10_12_universal2.whl
SHA-256 checksum: efb11b3f6107cec14edb38ee210177e13df52cfe3e9e610bb500d72f91474618 Verifying attestations

Release files / toktok_rs-0.1.1-cp311-abi3-win_amd64.whl

Download URL	toktok_rs-0.1.1-cp311-abi3-win_amd64.whl
Size	2.4 MB
Tags	CPython 3.11 Windows x86-64 abi3
SHA-256 checksum How to use checksums	`78a8a242749b144c391ce8493eb17a0b1ca8cce3f798170ca101eb9b06db00cc`
BLAKE2b-256 checksum How to use checksums	`d47fcac148ea4e9d1fa0e5e5e3ff51b9e68b259a6734182cea9cf8696569ec87`
Upload date	Sep 2, 2026
Uploaded using Trusted Publishing? What is trusted publishing?	Yes
Uploaded via	`twine/7.0.0 CPython/3.13.14`

Provenance

Identity

Publishing platform: GitHub Actions
Publishing repository: github.com/vectorize-io/toktok
Publishing commit: github.com/vectorize-io/toktok/tree/54a013f27cda577dd506d7ccb6cb8d9ff993052d

Workflow

Publishing configuration: github.com/vectorize-io/toktok/blob/54a013f27cda577dd506d7ccb6cb8d9ff993052d/.github/workflows/release.yml
Publishing logs: github.com/vectorize-io/toktok/actions/runs/33600816129/attempts/1

Artifact

Subject: toktok_rs-0.1.1-cp311-abi3-win_amd64.whl
SHA-256 checksum: 78a8a242749b144c391ce8493eb17a0b1ca8cce3f798170ca101eb9b06db00cc Verifying attestations

Release files / toktok_rs-0.1.1-cp311-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl

Download URL	toktok_rs-0.1.1-cp311-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Size	2.6 MB
Tags	CPython 3.11 Linux glibc 2.17+ x86-64 abi3
SHA-256 checksum How to use checksums	`74a96f866d77e75cad7c042e19efe001b8bebfd396ccb7c23b4bfce9390efb08`
BLAKE2b-256 checksum How to use checksums	`aaa994f7e00c71a2a1676c7ee79148f9c30b34d3fcbbb5eaabe1a7b2f93511fb`
Upload date	Sep 2, 2026
Uploaded using Trusted Publishing? What is trusted publishing?	Yes
Uploaded via	`twine/7.0.0 CPython/3.13.14`

Provenance

Identity

Publishing platform: GitHub Actions
Publishing repository: github.com/vectorize-io/toktok
Publishing commit: github.com/vectorize-io/toktok/tree/54a013f27cda577dd506d7ccb6cb8d9ff993052d

Workflow

Publishing configuration: github.com/vectorize-io/toktok/blob/54a013f27cda577dd506d7ccb6cb8d9ff993052d/.github/workflows/release.yml
Publishing logs: github.com/vectorize-io/toktok/actions/runs/33600816129/attempts/1

Artifact

Subject: toktok_rs-0.1.1-cp311-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
SHA-256 checksum: 74a96f866d77e75cad7c042e19efe001b8bebfd396ccb7c23b4bfce9390efb08 Verifying attestations

Release files / toktok_rs-0.1.1-cp311-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl

Download URL	toktok_rs-0.1.1-cp311-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Size	2.5 MB
Tags	CPython 3.11 Linux glibc 2.17+ ARM64 abi3
SHA-256 checksum How to use checksums	`f4831127c8060d51b365f3dad1aba9ebe14ea5d1108c3b70ac1c8451ff4260fb`
BLAKE2b-256 checksum How to use checksums	`255e76009e831d49db4dc990c09a2cb2fc15d362904ab7a13eab1c72a1db2583`
Upload date	Sep 2, 2026
Uploaded using Trusted Publishing? What is trusted publishing?	Yes
Uploaded via	`twine/7.0.0 CPython/3.13.14`

Provenance

Identity

Publishing platform: GitHub Actions
Publishing repository: github.com/vectorize-io/toktok
Publishing commit: github.com/vectorize-io/toktok/tree/54a013f27cda577dd506d7ccb6cb8d9ff993052d

Workflow

Publishing configuration: github.com/vectorize-io/toktok/blob/54a013f27cda577dd506d7ccb6cb8d9ff993052d/.github/workflows/release.yml
Publishing logs: github.com/vectorize-io/toktok/actions/runs/33600816129/attempts/1

Artifact

Subject: toktok_rs-0.1.1-cp311-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
SHA-256 checksum: f4831127c8060d51b365f3dad1aba9ebe14ea5d1108c3b70ac1c8451ff4260fb Verifying attestations

Release files / toktok_rs-0.1.1-cp311-abi3-macosx_10_12_x86_64.macosx_11_0_arm64.macosx_10_12_universal2.whl

Download URL	toktok_rs-0.1.1-cp311-abi3-macosx_10_12_x86_64.macosx_11_0_arm64.macosx_10_12_universal2.whl
Size	5.1 MB
Tags	CPython 3.11 abi3 macOS 10.12+ universal2 (ARM64, x86-64) macOS 10.12+ x86-64 macOS 11.0+ ARM64
SHA-256 checksum How to use checksums	`ae698deef758f659fd85d9a0f621eab067b42f909b7d56187a98fc3079dc2c60`
BLAKE2b-256 checksum How to use checksums	`d99485c8731b347765c1daa5fe04a2a0e62da9ced1693e5183c65ed45f5fe104`
Upload date	Sep 2, 2026
Uploaded using Trusted Publishing? What is trusted publishing?	Yes
Uploaded via	`twine/7.0.0 CPython/3.13.14`

Provenance

Identity

Publishing platform: GitHub Actions
Publishing repository: github.com/vectorize-io/toktok
Publishing commit: github.com/vectorize-io/toktok/tree/54a013f27cda577dd506d7ccb6cb8d9ff993052d

Workflow

Publishing configuration: github.com/vectorize-io/toktok/blob/54a013f27cda577dd506d7ccb6cb8d9ff993052d/.github/workflows/release.yml
Publishing logs: github.com/vectorize-io/toktok/actions/runs/33600816129/attempts/1

Artifact

Subject: toktok_rs-0.1.1-cp311-abi3-macosx_10_12_x86_64.macosx_11_0_arm64.macosx_10_12_universal2.whl
SHA-256 checksum: ae698deef758f659fd85d9a0f621eab067b42f909b7d56187a98fc3079dc2c60 Verifying attestations

Release history Release notifications | RSS feed

0.1.4

Sep 2, 2026 9 release files

0.1.3

Sep 2, 2026 9 release files

0.1.2

Sep 2, 2026 9 release files

This release

0.1.1 This release

Sep 2, 2026 9 release files

toktok-rs 0.1.1

toktok

Python

batch_count(texts, encoding="cl100k_base", threads=0, with_special=False)

Rust

More

License

Project links

Key dates

1 maintainer

Credits

GitHub Statistics

License expression

Requires

Provides Extra

Tags

Classifiers

Metadata

Project links

Key dates

1 maintainer

Credits

GitHub Statistics

License expression

Requires

Provides Extra

Tags

Classifiers

Release files for toktok-rs 0.1.1

Source distribution (sdist)

Built distributions (wheels)

Release files / toktok_rs-0.1.1.tar.gz

Provenance

PyPI Publish Attestation

Identity

Workflow

Artifact

Release files / toktok_rs-0.1.1-cp314-cp314t-win_amd64.whl

Provenance

PyPI Publish Attestation

Identity

Workflow

Artifact

Release files / toktok_rs-0.1.1-cp314-cp314t-manylinux_2_28_x86_64.whl

Provenance

PyPI Publish Attestation

Identity

Workflow

Artifact

Release files / toktok_rs-0.1.1-cp314-cp314t-manylinux_2_28_aarch64.whl

Provenance

PyPI Publish Attestation

Identity

Workflow

Artifact

Release files / toktok_rs-0.1.1-cp314-cp314t-macosx_10_12_x86_64.macosx_11_0_arm64.macosx_10_12_universal2.whl

Provenance

PyPI Publish Attestation

Identity

Workflow

Artifact

Release files / toktok_rs-0.1.1-cp311-abi3-win_amd64.whl

Provenance

PyPI Publish Attestation

Identity

Workflow

Artifact

Release files / toktok_rs-0.1.1-cp311-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl

Provenance

PyPI Publish Attestation

Identity

Workflow

Artifact

Release files / toktok_rs-0.1.1-cp311-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl

Provenance

PyPI Publish Attestation

Identity

Workflow

Artifact

Release files / toktok_rs-0.1.1-cp311-abi3-macosx_10_12_x86_64.macosx_11_0_arm64.macosx_10_12_universal2.whl

`batch_count(texts, encoding="cl100k_base", threads=0, with_special=False)`