toktok

A fast, exact BPE tokenizer for OpenAI encodings — Rust core, Python bindings. Token ids are byte-identical to tiktoken.

encoder		throughput	CPU per MB	p99 latency
toktok	Rust	99.6 MB/s	0.0117 s	11.7 µs
bpe-openai	Rust	30.2	—	42.6 µs
tiktoken-rs	Rust	7.4	—	164.3 µs
tiktoken	Python	16.2	0.0598 s	156.3 µs

_{The Pile, 25 MB, cl100k_base, single thread on a GitHub runner. Every
encoder's ids are verified identical before timing. Full tables, memory figures
and method: docs/BENCHMARKS.md.}

Encodings: cl100k_base (GPT-3.5/GPT-4), o200k_base (GPT-4o), o200k_harmony (GPT-OSS). They are compiled into the binary — no data files, no downloads, no runtime dependencies.

Python

uv add toktok-rs        # or: pip install toktok-rs

One function. Give it your texts and an encoding, get a token count per text:

import toktok

toktok.batch_count(["hello world", "how many tokens is this?"])
# [2, 6]

toktok.batch_count(docs, "gpt-4o")                    # model names work too
toktok.batch_count(docs, "cl100k_base", threads=4)    # 0 = every core (default)
toktok.batch_count(["a<|endoftext|>b"], with_special=True)   # [3]

`batch_count(texts, encoding="cl100k_base", threads=0, with_special=False)`

Returns list[int] — one count per text, in the order given.

parameter	default	what it does
`texts`	—	any iterable of `str`. An empty iterable returns `[]`; an empty string counts as `0`.
`encoding`	`"cl100k_base"`	which tokenizer to count with. Takes an encoding name — `cl100k_base`, `o200k_base`, `o200k_harmony` — or a model name, which resolves to that model's encoding: `gpt-4o`, `gpt-4`, `gpt-3.5-turbo`, `o1`, `o3`, `text-embedding-3-small`, `openai/gpt-oss-20b`. An org prefix is stripped, so HF-style ids work. Anything unrecognized raises `KeyError`.
`threads`	`0`	worker threads. `0` uses every core; `1` counts on the calling thread; any other number caps the pool. Counting releases the GIL, so this scales — including on free-threaded builds. Threads are only worth it for large batches; for a handful of short strings the pool costs more than it saves.
`with_special`	`False`	how to treat special-token strings. `False` counts `<\|endoftext\|>` as the ordinary text it looks like (7 tokens); `True` counts it as the single special token it is (1). Use `True` when your text already contains rendered chat/control markup and you want the count the model will see.

Ids are never built: one scratch buffer per thread is reused across texts, so a batch of any size allocates O(threads), not O(tokens).

Wheels cover CPython 3.11–3.14 plus free-threaded 3.14t, on Linux (x86_64, aarch64), macOS and Windows.

Need ids, decoding or offsets? toktok._encoding(name) returns the full tokenizer — see docs/PYTHON.md.

Rust

cargo add toktok-rs     # the crate is toktok-rs, the library is toktok

let tok = toktok::Tokenizer::builtin("cl100k_base")?;   // nothing to download

let ids = tok.encode(b"hello world");
let text = tok.decode(&ids);                 // lossless, even on invalid UTF-8
let n = tok.count(b"how many tokens is this?");

// batch APIs are parallel; 0 threads means every core
let docs: Vec<&[u8]> = vec![b"first", b"second"];
let counts = tok.count_batch(&docs, 0, false);   // no ids materialized
let batches = tok.encode_batch(&docs, 0, false);

let (ids, spans) = tok.encode_with_offsets(b"per-token byte spans");

Every batch method takes the same two knobs as the Python API:

argument	meaning
`threads: usize`	`0` uses every core; `1` runs on the calling thread; anything else caps the pool.
`with_special: bool`	`false` counts/encodes special strings as ordinary text; `true` maps them to their single special id.

Tokenizer is Send + Sync — load it once and share it. Full API on docs.rs/toktok-rs.

Cargo features

feature	default	what it does
`embedded-data`	on	compiles the vocabularies into the binary (~4 MB of `.rodata`), so `Tokenizer::builtin()` needs no files at runtime. Turn it off (`default-features = false`) to drop them and load from a directory with `Tokenizer::load_dir(dir, name)` instead.

docs/BENCHMARKS.md — full results, memory and latency profile, how to reproduce
docs/PYTHON.md — the tokenizer behind batch_count
docs/DEVELOPMENT.md — layout, tests, how it's fast
RELEASING.md — scripts/release.sh patch|minor|major|X.Y.Z, and the registry setup

License

MIT — see LICENSE. Ported from quicktok (MIT); vocabulary data derives from tiktoken (MIT). See NOTICE.

Metadata

Release files for toktok-rs 0.1.2

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for toktok-rs 0.1.2
File	Size	Uploaded
toktok_rs-0.1.2.tar.gz	2.2 MB	Sep 2, 2026	Details

Built distributions (wheels)

Table of built distributions (wheels) for toktok-rs 0.1.2
File	Interpreter	ABI	Platform	Reset
toktok_rs-0.1.2-cp314-cp314t-win_amd64.whl 2.4 MB Sep 2, 2026	CPython 3.14	CPython 3.14 free-threading	Windows x86-64	Details
toktok_rs-0.1.2-cp314-cp314t-manylinux_2_28_x86_64.whl 2.6 MB Sep 2, 2026	CPython 3.14	CPython 3.14 free-threading	Linux glibc 2.28+ x86-64	Details
toktok_rs-0.1.2-cp314-cp314t-manylinux_2_28_aarch64.whl 2.5 MB Sep 2, 2026	CPython 3.14	CPython 3.14 free-threading	Linux glibc 2.28+ ARM64	Details
toktok_rs-0.1.2-cp314-cp314t-macosx_10_12_x86_64.macosx_11_0_arm64.macosx_10_12_universal2.whl 5.1 MB Sep 2, 2026	CPython 3.14	CPython 3.14 free-threading	macOS 10.12+ x86-64, macOS 10.12+ universal2 (ARM64, x86-64), macOS 11.0+ ARM64	Details
toktok_rs-0.1.2-cp311-abi3-win_amd64.whl 2.4 MB Sep 2, 2026	CPython 3.11	abi3	Windows x86-64	Details
toktok_rs-0.1.2-cp311-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl 2.6 MB Sep 2, 2026	CPython 3.11	abi3	Linux glibc 2.17+ x86-64	Details
toktok_rs-0.1.2-cp311-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl 2.5 MB Sep 2, 2026	CPython 3.11	abi3	Linux glibc 2.17+ ARM64	Details
toktok_rs-0.1.2-cp311-abi3-macosx_10_12_x86_64.macosx_11_0_arm64.macosx_10_12_universal2.whl 5.1 MB Sep 2, 2026	CPython 3.11	abi3	macOS 10.12+ x86-64, macOS 10.12+ universal2 (ARM64, x86-64), macOS 11.0+ ARM64	Details

Total release size: 27.4 MB

Release files / toktok_rs-0.1.2.tar.gz

Download URL	toktok_rs-0.1.2.tar.gz
Size	2.2 MB
Tags	Source
SHA-256 checksum How to use checksums	`5eac708a247eb850a95798bf2911a8911026b94f938247bcd42b7c3dd55058d5`
BLAKE2b-256 checksum How to use checksums	`af93149d6ce86ee2a037d2c8c51c3bd18d7ea82b2de7036744433d50a0be9cd9`
Upload date	Sep 2, 2026
Uploaded using Trusted Publishing? What is trusted publishing?	Yes
Uploaded via	`twine/7.0.0 CPython/3.13.14`

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Identity

Publishing platform: GitHub Actions
Publishing repository: github.com/vectorize-io/toktok
Publishing commit: github.com/vectorize-io/toktok/tree/f41199c93b40d1aef13cb4ae2d2f9ea8582c1c2c

Workflow

Publishing configuration: github.com/vectorize-io/toktok/blob/f41199c93b40d1aef13cb4ae2d2f9ea8582c1c2c/.github/workflows/release.yml
Publishing logs: github.com/vectorize-io/toktok/actions/runs/33601719497/attempts/1

Artifact

Subject: toktok_rs-0.1.2.tar.gz
SHA-256 checksum: 5eac708a247eb850a95798bf2911a8911026b94f938247bcd42b7c3dd55058d5 Verifying attestations

Release files / toktok_rs-0.1.2-cp314-cp314t-win_amd64.whl

Download URL	toktok_rs-0.1.2-cp314-cp314t-win_amd64.whl
Size	2.4 MB
Tags	CPython 3.14 CPython 3.14 free-threading Windows x86-64
SHA-256 checksum How to use checksums	`de32024a79f3495b04988846d00ac6771fd0e1bacfca5fbfce47017ed858bffe`
BLAKE2b-256 checksum How to use checksums	`6d03abb8ef07727fb5b755ee83a701ee3531328cabc0d9cd1be2e98ea06e4479`
Upload date	Sep 2, 2026
Uploaded using Trusted Publishing? What is trusted publishing?	Yes
Uploaded via	`twine/7.0.0 CPython/3.13.14`

Provenance

Identity

Publishing platform: GitHub Actions
Publishing repository: github.com/vectorize-io/toktok
Publishing commit: github.com/vectorize-io/toktok/tree/f41199c93b40d1aef13cb4ae2d2f9ea8582c1c2c

Workflow

Publishing configuration: github.com/vectorize-io/toktok/blob/f41199c93b40d1aef13cb4ae2d2f9ea8582c1c2c/.github/workflows/release.yml
Publishing logs: github.com/vectorize-io/toktok/actions/runs/33601719497/attempts/1

Artifact

Subject: toktok_rs-0.1.2-cp314-cp314t-win_amd64.whl
SHA-256 checksum: de32024a79f3495b04988846d00ac6771fd0e1bacfca5fbfce47017ed858bffe Verifying attestations

Release files / toktok_rs-0.1.2-cp314-cp314t-manylinux_2_28_x86_64.whl

Download URL	toktok_rs-0.1.2-cp314-cp314t-manylinux_2_28_x86_64.whl
Size	2.6 MB
Tags	CPython 3.14 CPython 3.14 free-threading Linux glibc 2.28+ x86-64
SHA-256 checksum How to use checksums	`40fdeec36c07e066ee306306ad56ec2f401a1f53a0d3d274701c68cee2353fee`
BLAKE2b-256 checksum How to use checksums	`6a1f000aa056bf5cdcc12dcc3cdc71960d4cb8182e189e14408103219e2496c5`
Upload date	Sep 2, 2026
Uploaded using Trusted Publishing? What is trusted publishing?	Yes
Uploaded via	`twine/7.0.0 CPython/3.13.14`

Provenance

Identity

Publishing platform: GitHub Actions
Publishing repository: github.com/vectorize-io/toktok
Publishing commit: github.com/vectorize-io/toktok/tree/f41199c93b40d1aef13cb4ae2d2f9ea8582c1c2c

Workflow

Publishing configuration: github.com/vectorize-io/toktok/blob/f41199c93b40d1aef13cb4ae2d2f9ea8582c1c2c/.github/workflows/release.yml
Publishing logs: github.com/vectorize-io/toktok/actions/runs/33601719497/attempts/1

Artifact

Subject: toktok_rs-0.1.2-cp314-cp314t-manylinux_2_28_x86_64.whl
SHA-256 checksum: 40fdeec36c07e066ee306306ad56ec2f401a1f53a0d3d274701c68cee2353fee Verifying attestations

Release files / toktok_rs-0.1.2-cp314-cp314t-manylinux_2_28_aarch64.whl

Download URL	toktok_rs-0.1.2-cp314-cp314t-manylinux_2_28_aarch64.whl
Size	2.5 MB
Tags	CPython 3.14 CPython 3.14 free-threading Linux glibc 2.28+ ARM64
SHA-256 checksum How to use checksums	`1426279e7d8394ec7580e02851f1dfe521ecbfca1324975b9879f01a83795d8d`
BLAKE2b-256 checksum How to use checksums	`d42072d8e5c3f18fd13326dcf16d77d7afe5cc343395ccd201432db8d8d8fff5`
Upload date	Sep 2, 2026
Uploaded using Trusted Publishing? What is trusted publishing?	Yes
Uploaded via	`twine/7.0.0 CPython/3.13.14`

Provenance

Identity

Publishing platform: GitHub Actions
Publishing repository: github.com/vectorize-io/toktok
Publishing commit: github.com/vectorize-io/toktok/tree/f41199c93b40d1aef13cb4ae2d2f9ea8582c1c2c

Workflow

Publishing configuration: github.com/vectorize-io/toktok/blob/f41199c93b40d1aef13cb4ae2d2f9ea8582c1c2c/.github/workflows/release.yml
Publishing logs: github.com/vectorize-io/toktok/actions/runs/33601719497/attempts/1

Artifact

Subject: toktok_rs-0.1.2-cp314-cp314t-manylinux_2_28_aarch64.whl
SHA-256 checksum: 1426279e7d8394ec7580e02851f1dfe521ecbfca1324975b9879f01a83795d8d Verifying attestations

Release files / toktok_rs-0.1.2-cp314-cp314t-macosx_10_12_x86_64.macosx_11_0_arm64.macosx_10_12_universal2.whl

Download URL	toktok_rs-0.1.2-cp314-cp314t-macosx_10_12_x86_64.macosx_11_0_arm64.macosx_10_12_universal2.whl
Size	5.1 MB
Tags	CPython 3.14 CPython 3.14 free-threading macOS 10.12+ universal2 (ARM64, x86-64) macOS 10.12+ x86-64 macOS 11.0+ ARM64
SHA-256 checksum How to use checksums	`fa806931e05746078768ca7a049392635341777bc92a2d20ce247fc69d8ed9a5`
BLAKE2b-256 checksum How to use checksums	`76562978a51d0b8d692a9b8cd7ee3061bd350232831323b4c0ba471c68aaf2c3`
Upload date	Sep 2, 2026
Uploaded using Trusted Publishing? What is trusted publishing?	Yes
Uploaded via	`twine/7.0.0 CPython/3.13.14`

Provenance

Identity

Publishing platform: GitHub Actions
Publishing repository: github.com/vectorize-io/toktok
Publishing commit: github.com/vectorize-io/toktok/tree/f41199c93b40d1aef13cb4ae2d2f9ea8582c1c2c

Workflow

Publishing configuration: github.com/vectorize-io/toktok/blob/f41199c93b40d1aef13cb4ae2d2f9ea8582c1c2c/.github/workflows/release.yml
Publishing logs: github.com/vectorize-io/toktok/actions/runs/33601719497/attempts/1

Artifact

Subject: toktok_rs-0.1.2-cp314-cp314t-macosx_10_12_x86_64.macosx_11_0_arm64.macosx_10_12_universal2.whl
SHA-256 checksum: fa806931e05746078768ca7a049392635341777bc92a2d20ce247fc69d8ed9a5 Verifying attestations

Release files / toktok_rs-0.1.2-cp311-abi3-win_amd64.whl

Download URL	toktok_rs-0.1.2-cp311-abi3-win_amd64.whl
Size	2.4 MB
Tags	CPython 3.11 Windows x86-64 abi3
SHA-256 checksum How to use checksums	`d803c0073f6c53b73bd99575ef84c680078fc0f38bd6c4813a1a839a63226989`
BLAKE2b-256 checksum How to use checksums	`17052c745ce1dc32c691c99caa0764d65aafb8db88f9a19cd1b780d80c43915f`
Upload date	Sep 2, 2026
Uploaded using Trusted Publishing? What is trusted publishing?	Yes
Uploaded via	`twine/7.0.0 CPython/3.13.14`

Provenance

Identity

Publishing platform: GitHub Actions
Publishing repository: github.com/vectorize-io/toktok
Publishing commit: github.com/vectorize-io/toktok/tree/f41199c93b40d1aef13cb4ae2d2f9ea8582c1c2c

Workflow

Publishing configuration: github.com/vectorize-io/toktok/blob/f41199c93b40d1aef13cb4ae2d2f9ea8582c1c2c/.github/workflows/release.yml
Publishing logs: github.com/vectorize-io/toktok/actions/runs/33601719497/attempts/1

Artifact

Subject: toktok_rs-0.1.2-cp311-abi3-win_amd64.whl
SHA-256 checksum: d803c0073f6c53b73bd99575ef84c680078fc0f38bd6c4813a1a839a63226989 Verifying attestations

Release files / toktok_rs-0.1.2-cp311-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl

Download URL	toktok_rs-0.1.2-cp311-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Size	2.6 MB
Tags	CPython 3.11 Linux glibc 2.17+ x86-64 abi3
SHA-256 checksum How to use checksums	`c5c86d83c878afb0ab3c480e1b4f26663ae4172fb6ad8c65b4a18c77d0b4e243`
BLAKE2b-256 checksum How to use checksums	`56109cd1b711490b896155b887ab60918d1deef6f2921d3b7b81c7396eb027cb`
Upload date	Sep 2, 2026
Uploaded using Trusted Publishing? What is trusted publishing?	Yes
Uploaded via	`twine/7.0.0 CPython/3.13.14`

Provenance

Identity

Publishing platform: GitHub Actions
Publishing repository: github.com/vectorize-io/toktok
Publishing commit: github.com/vectorize-io/toktok/tree/f41199c93b40d1aef13cb4ae2d2f9ea8582c1c2c

Workflow

Publishing configuration: github.com/vectorize-io/toktok/blob/f41199c93b40d1aef13cb4ae2d2f9ea8582c1c2c/.github/workflows/release.yml
Publishing logs: github.com/vectorize-io/toktok/actions/runs/33601719497/attempts/1

Artifact

Subject: toktok_rs-0.1.2-cp311-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
SHA-256 checksum: c5c86d83c878afb0ab3c480e1b4f26663ae4172fb6ad8c65b4a18c77d0b4e243 Verifying attestations

Release files / toktok_rs-0.1.2-cp311-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl

Download URL	toktok_rs-0.1.2-cp311-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Size	2.5 MB
Tags	CPython 3.11 Linux glibc 2.17+ ARM64 abi3
SHA-256 checksum How to use checksums	`54a6464e3983777a6087f87989f71b9694ef7811ed67b1cbb68acb29aa957ddd`
BLAKE2b-256 checksum How to use checksums	`c3ad24336b1c7c9a58698e9529136ac3fb44e9ce8e19f3e0b5b23219fa286489`
Upload date	Sep 2, 2026
Uploaded using Trusted Publishing? What is trusted publishing?	Yes
Uploaded via	`twine/7.0.0 CPython/3.13.14`

Provenance

Identity

Publishing platform: GitHub Actions
Publishing repository: github.com/vectorize-io/toktok
Publishing commit: github.com/vectorize-io/toktok/tree/f41199c93b40d1aef13cb4ae2d2f9ea8582c1c2c

Workflow

Publishing configuration: github.com/vectorize-io/toktok/blob/f41199c93b40d1aef13cb4ae2d2f9ea8582c1c2c/.github/workflows/release.yml
Publishing logs: github.com/vectorize-io/toktok/actions/runs/33601719497/attempts/1

Artifact

Subject: toktok_rs-0.1.2-cp311-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
SHA-256 checksum: 54a6464e3983777a6087f87989f71b9694ef7811ed67b1cbb68acb29aa957ddd Verifying attestations

Release files / toktok_rs-0.1.2-cp311-abi3-macosx_10_12_x86_64.macosx_11_0_arm64.macosx_10_12_universal2.whl

Download URL	toktok_rs-0.1.2-cp311-abi3-macosx_10_12_x86_64.macosx_11_0_arm64.macosx_10_12_universal2.whl
Size	5.1 MB
Tags	CPython 3.11 abi3 macOS 10.12+ universal2 (ARM64, x86-64) macOS 10.12+ x86-64 macOS 11.0+ ARM64
SHA-256 checksum How to use checksums	`52a1f44b88b69a757b0feda97b6fa2a2f7cd4fd24f83c25141f51c9fad326a8a`
BLAKE2b-256 checksum How to use checksums	`2cb1edfaf0f7ea1601e82df8b2b7b4aecda502768936401f89a9378c614ae76b`
Upload date	Sep 2, 2026
Uploaded using Trusted Publishing? What is trusted publishing?	Yes
Uploaded via	`twine/7.0.0 CPython/3.13.14`

Provenance

Identity

Publishing platform: GitHub Actions
Publishing repository: github.com/vectorize-io/toktok
Publishing commit: github.com/vectorize-io/toktok/tree/f41199c93b40d1aef13cb4ae2d2f9ea8582c1c2c

Workflow

Publishing configuration: github.com/vectorize-io/toktok/blob/f41199c93b40d1aef13cb4ae2d2f9ea8582c1c2c/.github/workflows/release.yml
Publishing logs: github.com/vectorize-io/toktok/actions/runs/33601719497/attempts/1

Artifact

Subject: toktok_rs-0.1.2-cp311-abi3-macosx_10_12_x86_64.macosx_11_0_arm64.macosx_10_12_universal2.whl
SHA-256 checksum: 52a1f44b88b69a757b0feda97b6fa2a2f7cd4fd24f83c25141f51c9fad326a8a Verifying attestations

Release history Release notifications | RSS feed

0.1.4

Sep 2, 2026 9 release files

0.1.3

Sep 2, 2026 9 release files

This release

0.1.2 This release

Sep 2, 2026 9 release files

0.1.1

Sep 2, 2026 9 release files

toktok-rs 0.1.2

toktok

Python

batch_count(texts, encoding="cl100k_base", threads=0, with_special=False)

Rust

More

License

Project links

Key dates

1 maintainer

Credits

GitHub Statistics

License expression

Requires

Provides Extra

Tags

Classifiers

Metadata

Project links

Key dates

1 maintainer

Credits

GitHub Statistics

License expression

Requires

Provides Extra

Tags

Classifiers

Release files for toktok-rs 0.1.2

Source distribution (sdist)

Built distributions (wheels)

Release files / toktok_rs-0.1.2.tar.gz

Provenance

PyPI Publish Attestation

Identity

Workflow

Artifact

Release files / toktok_rs-0.1.2-cp314-cp314t-win_amd64.whl

Provenance

PyPI Publish Attestation

Identity

Workflow

Artifact

Release files / toktok_rs-0.1.2-cp314-cp314t-manylinux_2_28_x86_64.whl

Provenance

PyPI Publish Attestation

Identity

Workflow

Artifact

Release files / toktok_rs-0.1.2-cp314-cp314t-manylinux_2_28_aarch64.whl

Provenance

PyPI Publish Attestation

Identity

Workflow

Artifact

Release files / toktok_rs-0.1.2-cp314-cp314t-macosx_10_12_x86_64.macosx_11_0_arm64.macosx_10_12_universal2.whl

Provenance

PyPI Publish Attestation

Identity

Workflow

Artifact

Release files / toktok_rs-0.1.2-cp311-abi3-win_amd64.whl

Provenance

PyPI Publish Attestation

Identity

Workflow

Artifact

Release files / toktok_rs-0.1.2-cp311-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl

Provenance

PyPI Publish Attestation

Identity

Workflow

Artifact

Release files / toktok_rs-0.1.2-cp311-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl

Provenance

PyPI Publish Attestation

Identity

Workflow

Artifact

Release files / toktok_rs-0.1.2-cp311-abi3-macosx_10_12_x86_64.macosx_11_0_arm64.macosx_10_12_universal2.whl

`batch_count(texts, encoding="cl100k_base", threads=0, with_special=False)`