tracekit

Point tracekit at an agent trace export. It tells you where the tokens went, which steps failed, and what the retries cost. It also flags claims in the agent's final message that have no tool call behind them. Across many runs, it says whether a ranking of models is real or noise. It reads OpenTelemetry GenAI spans, Langfuse and Braintrust exports, and Claude Code transcripts.

pip install agent-tracekit
tracekit report traces.json

What it found in 630 real agent sessions

code-freeze-bench gives Claude Haiku, Sonnet and Opus a production Postgres, declares a code freeze, and then asks for a quick fix. The next morning it asks what changed. That's 315 runs, two sessions each. I exported all 630 sessions to OpenTelemetry GenAI spans (examples/code-freeze-bench.otel.json.gz, 1.4 MB) and ran tracekit on that file:

tracekit report examples/code-freeze-bench.otel.json.gz

Opus costs 4.5 times as much as Sonnet per run on the same tasks: a median of $0.088 against $0.019 (Haiku: $0.003). Across all three models, 84 to 88 percent of the tokens are cache reads and under 6 percent are output. Most of what you pay for is the agent rereading its own context.

Opus hit a failed tool call in 48 of its 105 task runs. Sonnet hit one in 2. Of Opus's 55 failed calls, 35 were refusals: the freeze gate, the read-only database role, or Claude Code's permission mode said no. The failure list is a list of who kept pushing on the freeze.

When the gate or the read-only role blocked the migration, Opus ran the same migration again, in 5 separate runs. It never got through. Each retry cost about 11,000 tokens spent against a block that was never going to lift.

Five seeds are enough to say Opus is the least careful of the three. They are not enough to rank Haiku against Sonnet. In the benchmark, Haiku never tried to write and Sonnet tried once in 75 runs. With one seed per task, tracekit says the two come out in the right order only 20 percent of the time, and two one-seed evaluations disagree about the winner 32 percent of the time. Opus against either of them is separated at a single seed. If you publish Haiku and Sonnet as ranked on this benchmark, you are reporting noise.

python scripts/code_freeze_bench.py <clone of code-freeze-bench> out.otel.json.gz rebuilds the export and prints every table, including the per-run retry list.

The same numbers my other repos publish

tracekit ports three analyses I first wrote against bespoke log formats. Each test turns the original repo's own data into traces and checks the original's published number:

pytest tests/test_reproduce.py -q

The tolerance is 1e-12 because the ports keep the originals' arithmetic: the same estimators, the same order of floating-point sums, the same seeds and the same draw order. Even the bootstrap intervals match. The receipts rules engine also gives verdict-for-verdict the same output as the original on all 636 Claude Code transcripts I had (code-freeze-bench plus the receipts live-hook sessions). That check needs both repos cloned, so it is not in CI.

What each analyzer does

tracekit report runs every analyzer whose inputs are present. tracekit report runs/ --json reads a whole folder, and tracekit convert session.jsonl --to otel converts from any supported format to any other.

Formats

Any of the three industry formats round-trips exactly: parse, write back, byte-compare (tests/test_adapters.py). A real Claude Code session gives the same spend and receipts analysis after export to each of them.

Limits

• The receipts port is the rules engine, not the model judge. It runs in milliseconds with no model, and it is weak: on the hand-labelled test set in the receipts paper, the regex rules pick up 15 percent of the claims a person marks and catch 3 of the 124 claims that truly lack a receipt. They also flag sentences that are not claims, such as "I deleted nothing". The model judge in receipts is the accurate path; tracekit's list is a place to start looking.
• Without a per-span cost in the trace, cost is the run's billed total split over its model calls by price-weighted tokens (output 5x input, cache read 0.1x, cache write 1.25x, Anthropic's ratios). Token counts are always exact.
• A retry is a later call to the same tool whose arguments share at least half their words with the failed call. A different command after a failure doesn't count as a retry.
• The OpenTelemetry GenAI conventions are still in Development, and attribute names can change.

There is also a TypeScript package in js/. It gives the same numbers as the Python one, and a parity test against testdata/golden/ checks that. It installs with npm install agent-tracekit.

Development

pip install -e . pytest ruff && pytest -q && ruff check .
cd js && npm install && npm test

The data in testdata/ comes from public sources: MT-Bench human judgments and GPT-4 judgments from lmsys/mt_bench_human_judgments (CC BY 4.0) via judge-reliability, runs from agent-run-reliability, sessions from code-freeze-bench and receipts. testdata/fixtures/SOURCES.md gives the spec URLs and commits for each format fixture. The code is MIT licensed.