assay

Unit standardization for token price indices.

Inference is priced in dollars per million tokens, but a token is not a standard unit — each provider's tokenizer segments identical text into a different number of them. Two providers with the same headline $/1M tokens can bill materially different amounts for the same work, and any index aggregating across providers is summing incommensurable quantities.

assay measures how each tokenizer segments a fixed, versioned reference corpus and emits normalization factors — with confidence intervals — that convert provider-specific counts to a common basis: the Standard Token Equivalent (STE). That makes $/1M tokens comparable.

It measures the denominator of token pricing only, and needs no paid inference: tokenizers run locally or via free counting endpoints. It does not evaluate model quality — conflating the two produces bad measurements of both.

Install

uvx --from token-assay assay factors   # zero-install: the published table, now
pip install token-assay                # or: uv add token-assay

The PyPI distribution is token-assay; the import and the command are assay — hence the --from.

Findings

docs/DRIFT-LEDGER.md records every token-unit redefinition found across seven model families, paired with the list price either side of it, measured on the reference corpus with a 95% CI on every figure.

Three redefinitions, in three of the largest families, changing the cost of identical work by up to 55% (OpenAI), 34% (Anthropic — 41% on conversational English), and 40% (Meta), depending on content type.

In two of them the price per token did not move at all, so a dollars-per-token index reports perfect stability across the entire change. That blindness is algebraic, not statistical: the redefinition scales numerator and denominator together, so no amount of transaction data recovers it.

Four families were stable across every generation measured, and the two redefining families that span more than one boundary held their unit constant across the others — which rules out the "tokenizers just drift" reading. Redefinition is a discrete, datable event.

Use

# The published table, from data shipped in the package — no keys, no network.
assay factors                        # STE vector: per-class + per-segment, with CIs
assay factors --weights mine.toml    # substitute your own workload mix
assay compare --price anthropic/claude-opus-5=5/25 --price openai/gpt-5.4=2.50/15

# Recompute every published figure offline from the committed artifact.
assay report --counts results/counts-corpus-v0.1.jsonl.gz
assay report --counts results/counts-corpus-v0.1.jsonl.gz --export results/

# Reproduce the corpus and raw counts (free credentials; see .env.example).
assay corpus fetch && assay measure --out counts.jsonl.gz

# Diagnostics. These re-measure live, so they need network — and, for Anthropic and
# Google, a key. Without one those rows report `unmeasured` rather than guessing;
# the offline path to the same figures is `assay report --counts` above.
assay ledger                # redefinitions and their cost effect
assay sweep --provider anthropic
assay canary --check        # hosted-tokenizer drift; exit 3 = drift

Provenance is a first-class output

Every count records where it came from, and figures are not interchangeable across those sources:

Mirrors are never substituted silently — a gated repository is a recorded skip, and counts from one are refused by the publication gate rather than by convention.

Local vocabularies are pinned: the exact bytes are re-fetchable and verified by content digest. Hosted endpoints can only be stamped with a model id and a timestamp, because the vendor does not publish the vocabulary and can revise it under a stable name. For those, assay canary is the only integrity control there is, and it runs weekly in CI against a committed baseline. Its probe set is four short texts covering three scripts, so it is a lower bound on detection, not a guarantee — METHODOLOGY.md §7.2 states what it would miss and why widening it is a versioned migration rather than an edit.

Scale

24 tokenizers · 7 families · corpus-v0.1 = 1,740 documents, 16 languages — 1,500 of them across the five published workload segments, 240 in a diagnostic segment that carries no published figure · 39,120 counts, zero degradations.

The count is not a full matrix and does not claim to be: the 13 local tokenizers counted all 1,740 documents, the 11 hosted ones only the 1,500 that figures are drawn from.

The corpus ships as manifest-plus-fetcher — pins, hashes and a retrieval script rather than third-party text. Assembled locally it is CC BY-SA 4.0 (FLORES+ requires share-alike); the measurement outputs are CC0; the code is MIT.

Development

uv sync
cp .env.example .env        # every endpoint used here is free
uv run pytest               # offline; needs no credentials
uv run pytest -m live       # hits real endpoints; needs keys
uv run ruff check src tests && uv run mypy

ANTHROPIC_API_KEY is required for Claude counts — Anthropic publishes no tokenizer, so count_tokens is the only admissible source. GEMINI_API_KEY and HF_TOKEN widen coverage. OpenAI needs no key: tiktoken runs locally and is exact.

License

MIT