cachecheck

Five agent frameworks silently undercounted Anthropic's prompt cache. This finds it in your repo in one command.

uvx cachecheck .

No dependencies. Exit code 1 when it finds something, so it drops straight into CI.

The defect

Anthropic reports input_tokens net of the prompt cache, and bills cache_read_input_tokens and cache_creation_input_tokens separately on top. It does not send a total.

So this line, which appears in a lot of shipped code, is wrong:

total = usage.input_tokens + usage.output_tokens

Every token served from cache falls out of the number. In a voice agent that re-reads its history each turn, a turn billed for 2,100 tokens with 2,000 cached reports 150. Nothing raises, the number looks plausible, and anything costing off it under-reports.

Where this came from

Each rule is a defect found by hand in a shipped library, every one with a pull request and a test that fails without the fix:

Framework	Pull request	State
Pipecat (Daily)	#5163	merged
LiveKit Agents	#6663	merged, by LiveKit's co-founder
Haystack (deepset)	#3717	merged
mcp-use	#2127	open
LlamaIndex	#22548	closed, subsystem being retired

The full write-up of each, with the wrong number and the right one, is at wrong-numbers.

How many of those it actually catches

Four of the five. That is measured, not asserted: each framework's source was pulled at the parent of its fix commit and scanned, and the fixed version was scanned too.

	pre-fix	post-fix
Pipecat	caught	silent
LiveKit Agents	caught	silent
Haystack	caught	silent
mcp-use	caught	silent
LlamaIndex	missed	n/a

LlamaIndex's defect is in provider-agnostic core code that names no provider, and the gate requires one. That is the price of not reporting every OpenAI file in a repository, and it is a real miss rather than a rounding error.

The first published version found two of the five. It missed Pipecat, the defect this is named after, and it reported LiveKit's merged fix as a defect. Both are fixed. If you are reading this because a badge told you the tool was thorough, the numbers above are the ones to trust.

On the three repositories at HEAD, where every one of these fixes has landed and the answer should be silence, it reports 2 findings across 2,380 files. Before these changes it reported 14, of which all 14 were wrong.

What it checks

total-excludes-cache · a total of input plus output that leaves the cache out, in a file that reads the cache counters elsewhere. The breakdown is right and only the total is wrong, which is far harder to spot than ignoring the cache outright. This is Pipecat's shape, and it is the most common one.

net-input-as-gross · assigns input_tokens to prompt_tokens. Those names do not mean the same thing: prompt_tokens includes the cached tokens and input_tokens is net of them, so the cache is lost in the rename with no arithmetic anywhere to give it away. This is Haystack's shape.

cache-blind-total · a total of input plus output, in a file that never reads cache_read_input_tokens. Those tokens are billed and are not inside input_tokens.

cache-read-never-read · computes with input_tokens and never reads the cache counter at all, so every cached token is invisible to the accounting.

cache-creation-ignored · reads cache reads but not cache writes. Cache writes are billed above the base rate, so they cost more per token than the ones they save. Bedrock calls this field cacheWriteInputTokens.

What it deliberately does not flag

OpenAI reports cached_tokens inside prompt_tokens. Adding it there would double count, so a file handling OpenAI-shaped usage is correct as written and is left alone.

Comments, docstrings and type declarations name these fields without computing anything. An earlier version flagged inputTokens?: number; and a docstring showing sample output. Rule 2 now requires evidence of arithmetic, not a mention.

Test files build fixtures rather than billing anyone, and are skipped unless you pass --tests.

A checker that cries wolf on correct code gets switched off, and then it catches nothing at all.

Usage

cachecheck .                 # human readable
cachecheck . --json          # machine readable
cachecheck . --quiet         # exit code only, for CI
cachecheck . --tests         # include test files
cachecheck path/to/file.py   # a single file

Scans .py, .ts, .tsx, .js, .mjs, .jsx, .go, .rb, .java, .kt.

GitHub Action

- uses: arthi-arumugam-git/cachecheck@v1
  with:
    path: .

Fails the job when anything is found. Set fail-on-findings: false to report without failing.

Install

uvx cachecheck .        # no install
pip install cachecheck  # or install it

Python 3.9 or newer. No dependencies, on purpose: this has to run in someone else's CI without an argument about their lockfile.

Honest limits

It is a static checker built on patterns, not a type checker. It reads code, not runtime behaviour.

It only reads files that name Anthropic, Claude or Bedrock. Every question here has an answer on those providers and nowhere else, because only they report the input count net of the cache. The cost is that provider-agnostic code which aggregates usage without naming anyone is skipped, and LlamaIndex's core token counter is exactly that shape. That miss is the direct consequence of this line, not an accident.

A file naming one of those can still be a false positive. Bedrock's Nova Sonic reports speech and text tokens that no prompt cache serves, and it is still a Bedrock file. Read the evidence line before acting on a finding.

Rule 5 reads a window around the assignment, so a file that folds the cache in more than eight lines away from the rename will be reported. That is a heuristic and it is the weakest thing here.

Silence is a result, not proof. It means nothing matched these five shapes in the files it scanned.

Licence

MIT.