ctok

ctok reconstructs Claude token counts offline, with no API call or network access. It is unofficial and is not affiliated with Anthropic.

The reconstruction targets counts. Claude does not expose token boundaries, so tokenize() returns one valid minimum-cost tiling, not a claim about Anthropic's exact segmentation. The research behind the model is described in On the biology of Claude's tokenizer.

Count a string

ctok is on PyPI, so uvx fetches and runs it with no install. With no --version it reports both generations, since the same string does not cost the same on each.

uvx ctok "hello, world"

  v3.0
  stream: '⟨bow⟩hello⟨eow⟩,⟨eow⟩⟨bow⟩world⟨eow⟩'
  tokens: ['⟨bow⟩hello⟨eow⟩', ',⟨eow⟩', '⟨bow⟩world⟨eow⟩']

  content 3 + frame 7 = 10

  v4.8
  stream: '⟨bow⟩hello⟨eow⟩,⟨eow⟩⟨bow⟩world⟨eow⟩'
  tokens: ['⟨bow⟩h', 'ello⟨eow⟩', ',⟨eow⟩', '⟨bow⟩world⟨eow⟩']

  content 4 + frame 6 = 10

--version 4.7 picks one family instead.

As a library

from ctok import token_count, tokenize

token_count("hello, world")           # 10, using the v3 family
token_count("hello, world", "4.7")    # 15
token_count("hello, world", "4.8")    # 10

tokens = tokenize("NASA likes tokenizers")
assert len(tokens) == token_count("NASA likes tokenizers")

Supported families

version is a string such as "4.7". Components are compared as integers, so "4.10" sorts after "4.9". Python reads the float literal 4.10 as 4.1, so passing a non-string version raises TypeError.

v4.8+ uses the v4.7 vocabulary with a six-token frame. Opus 5 alone makes trailing ASCII whitespace free, which ctok intentionally does not model.

How it works

For one user message, ctok:

1. normalizes the text, including NFC and family-specific quote folding;
2. rewrites it into a stream with word, case, and byte markers. For BMP combining marks, Unicode Alphabetic decides the split: Alphabetic marks stay with the word and non-Alphabetic marks, other than variation selectors, stand outside it;
3. finds a minimum-cost tiling over the measured vocabulary and UTF-8 byte fallback;
4. adds the measured message frame.

token_count(text) is len(tokenize(text)). The output notation makes internal structure visible:

Measured accuracy

These results compare ctok with recorded count_tokens responses:

v4.8+ is omitted because the table gates physical vocabularies. It reuses v4.7's vocabulary.

The stored measurement sets contain no under-counts: 0 of 2,276,929 v3 texts and 0 of 2,328,425 v4.7 texts. This does not guarantee the result for arbitrary input. Goldfish, the main Rosetta sample, and UDHR informed piece selection. MultiPL-E and the separate Rosetta sample did not.

Run the public gates with:

uv run pytest
uv run python tests/gates.py

Vocabulary evidence

The two vocabulary files contain 48,792 v3 pieces and 15,283 v4.7 pieces. Every entry has a fixed membership witness or is one of the structural marker atoms checked by the test suite.

from ctok import pieces, witness

len(pieces("4.7"))                       # 15283
witness("⟨bow⟩the⟨eow⟩", "4.7")
# {'probe': 'the', 'raw': 12, 'kind': 'raw'}

A witness says that one marked piece costs one token in a calibrated probe. It does not prove the encoder rewrite or resolve ties between equal-cost tilings. tests/test_witness.py checks every published witness and requires complete witnessed-or-special coverage.

License

MIT