llm-tokens-atlas

Open benchmark of LLM tokenization — offline vs empirical calibration deltas across five providers.

What this is

llm-tokens-atlas is a reproducible, open dataset and analysis pipeline that measures how offline tokenizers (e.g. tiktoken proxies, the published BPE vocabularies) compare against the empirical token counts returned by each provider's own API or OSS tokenizer. The current dataset covers 5 providers (Anthropic claude-opus-4-7, Google gemini-2.5-pro, OpenAI gpt-4o, Mistral mistral-large-latest, Cohere command-r-08-2024), 5 prompt formats (Markdown, XML, JSON, YAML, plain text), and 12,500 real-world prompt requests (500 unique prompts × 5 formats × 5 providers; n=2,500 per provider) drawn from open corpora.

The output is a per-provider, per-format calibration delta distribution — so anyone estimating cost or context-window budgets ahead of an API call can quantify the bias of their offline counter instead of treating it as exact.

This project builds on tokenometer, which surfaced the underlying methodology and motivated the larger provider- and format-stratified sweep published here.

Status

v0.2.0 candidate — 5-provider coverage (Anthropic + Google + OpenAI + Mistral + Cohere). Schema, drivers, and data are stable for the five-provider sweep.

Headline findings

n = 12,500 rows (2,500 per provider). Detailed numbers live in analysis/results.json.

The Anthropic row is the headline: the publicly-recommended offline tokenizer underestimates real claude-opus-4-7 cost by ~41% across thousands of prompts, and 100% of rows underestimate (no exact / overestimate cases). OpenAI and Mistral are baselines confirming the offline-vs-empirical pipeline is calibrated correctly when the provider's own tokenizer is the oracle. Google and Cohere show lower median error than Anthropic but wider format-dependent bias.

Install

The recommended local workflow uses uv:

uv sync
make install

For library use from a project environment:

pip install llm-tokens-atlas

Usage

Load the published dataset from Hugging Face:

from datasets import load_dataset

dataset = load_dataset("faraa2m/llm-tokens-atlas")
df = dataset["train"].to_pandas()

anthropic = df[df["provider"] == "anthropic"]
print(anthropic["delta_pct"].median())

Run a small credentials-free reproduction locally:

make reproduce-tiny

Run the full pipeline with the default provider set:

make reproduce

Use the Python bridge when another analysis script needs token counts through the same Tokenometer path as the published dataset:

from llm_tokens_atlas.tokenometer_bridge import count_offline

result = count_offline(
    text="Summarize this support ticket.",
    provider="openai",
    model="gpt-4o",
    format="markdown",
)
print(result)

See docs/REPRODUCING.md for provider keys, expected runtime, generated files, and CI-sized runs.

Calibration Examples

Use Atlas when an offline tokenizer needs a correction factor before a large batch job:

• Claude budgeting — the Anthropic sweep shows systematic undercounting versus empirical provider counts, so production budgets should include a provider-specific calibration margin.
• OpenAI sanity checks — the gpt-4o row acts as an oracle-style baseline for o200k_base counting.
• Mistral validation — the Mistral row validates the OSS tokenizer path for SentencePiece-family models.
• Google and Cohere guardrails — the gemini-2.5-pro and command-r-08-2024 rows expose format-sensitive mixed bias that should be calibrated rather than treated as exact.

Generated result tables live in analysis/results.json when the analysis pipeline has been run. Generated figures are expected under analysis/figures/.

Reproducing results

make reproduce

This regenerates the dataset from scratch. Tokenizer and provider API versions are pinned (see data/lockfile.json once published).

See docs/REPRODUCING.md for full instructions — required API keys per provider, expected runtime at each scale, output sizes, and a CI-friendly tiny variant (make reproduce-tiny).

Tokenometer integration

Atlas reuses tokenometer's multi-provider tokenizer logic instead of reimplementing it in Python. The integration lives in a single module:

• llm_tokens_atlas/tokenometer_bridge.py — Python facade over the tokenometer CLI. Exposes count_offline, count_empirical, list_providers, list_models, list_formats, plus a count_offline_batch / count_empirical_batch pair for the high-throughput atlas pipeline.
• llm_tokens_atlas/install_tokenometer.sh — idempotent installer; make install runs it. Finds tokenometer via (1) tokenometer on PATH, (2) a sibling ../tokenometer/ repo build, (3) builds the sibling if source is present, or (4) fails with an install hint.

Any new Python code that needs token counts should import from llm_tokens_atlas.tokenometer_bridge. Do not invoke the tokenometer CLI directly from other modules.

Publishing the dataset

The canonical home for the dataset is https://huggingface.co/datasets/faraa2m/llm-tokens-atlas. The Hugging Face dataset card lives at data/README.md. The upload script is llm_tokens_atlas/publish_to_hf.py; set HF_TOKEN in your env and run it with --dataset llm-tokens-atlas.

Project Health

• Contributing guide
• Security policy
• Code of Conduct
• Changelog

Reproducing

• docs/REPRODUCING.md — make reproduce mechanics + expected runtime.

Citation

Cite the version you used. Until the paper is on arxiv, cite the GitHub repo and the HuggingFace dataset directly:

@misc{llm-tokens-atlas-2026,
  author       = {Faraazuddin Mohammed},
  title        = {{llm-tokens-atlas}: An Open Benchmark of LLM Tokenization Calibration},
  year         = {2026},
  version      = {v0.2.0},
  howpublished = {\url{https://github.com/faraa2m/llm-tokens-atlas}},
  note         = {5-provider coverage (Anthropic, Google, OpenAI, Mistral, Cohere). Companion arxiv preprint forthcoming.}
}

License

• Code: Apache-2.0
• Data (everything under data/): CC-BY-4.0