ctok

ctok reconstructs Claude token counts offline, with no API call, network access, or runtime dependencies. It is unofficial and is not affiliated with Anthropic.

The reconstruction targets counts. Claude does not expose token boundaries, so tokenize() returns one valid minimum-cost tiling, not a claim about Anthropic's exact segmentation. The research behind the model is described in On the biology of Claude's tokenizer.

Install

pip install ctok

Quick start

from ctok import token_count, tokenize

token_count("hello, world")           # 10, using the v3 family
token_count("hello, world", "4.7")    # 15
token_count("hello, world", "5.0")    # 10

tokens = tokenize("NASA likes tokenizers")
assert len(tokens) == token_count("NASA likes tokenizers")

The command-line interface prints the marked stream and its tiling:

ctok "hello, world"

Supported families

version is a string such as "4.7". Components are compared as integers, so "4.10" sorts after "4.9". Python reads the float literal 4.10 as 4.1, so passing a non-string version raises TypeError.

v5 uses the v4.7 vocabulary with a different message frame.

How it works

For one user message, ctok:

1. normalizes the text, including NFC and family-specific quote folding;
2. rewrites it into a stream with word, case, and byte markers;
3. finds a minimum-cost tiling over the measured vocabulary and UTF-8 byte fallback;
4. adds the measured message frame.

token_count(text) is len(tokenize(text)). The output notation makes internal structure visible:

Measured accuracy

These results compare ctok with recorded count_tokens responses:

v5 is omitted from the table because its deviation from recorded counts matches v4.7's on every gated document. Separate API tests cover its message-frame rules.

The stored measurement sets contain no under-counts: 0 of 1,664,940 v3 texts and 0 of 1,722,961 v4.7 texts. This does not guarantee the result for arbitrary input. Goldfish, the main Rosetta sample, and UDHR informed piece selection. MultiPL-E and the separate Rosetta sample did not.

Run the public gates with:

uv run pytest
uv run python tests/gates.py

Vocabulary evidence

The two vocabulary files contain 48,645 v3 pieces and 15,240 v4.7 pieces. Every entry has a fixed membership witness or is one of the structural marker atoms checked by the test suite.

from ctok import pieces, witness

len(pieces("4.7"))                       # 15240
witness("⟨bow⟩the⟨eow⟩", "4.7")
# {'probe': 'the', 'raw': 12, 'kind': 'raw'}

A witness says that one marked piece costs one token in a calibrated probe. It does not prove the encoder rewrite or resolve ties between equal-cost tilings. tests/test_witness.py checks every published witness and requires complete witnessed-or-special coverage.

License

MIT