promptcachelint

Explain why your LLM prompt cache missed.

Prompt caching on Anthropic and OpenAI is a prefix match: one changed byte anywhere before the cache marker and everything after it is billed again at full price. The providers tell you that it happened (cache_read_input_tokens: 0) but not where. promptcachelint records your requests, diffs each one against the previous request of the same conversation, and names the block, the offset and the characters that broke the prefix, plus the usual suspects it can see without history: timestamps in the system prompt, a missing marker, a prefix below the model's minimum.

promptcachelint: 3 requests in 1 session(s); prompt tokens 4880 (read 0, write 4806, uncached 74); hit ratio 0.0%; 2 prefix break(s), ~3112 tokens lost (estimate)
  (sessions grouped automatically by first message; use promptcachelint.session(id) for exact grouping)

session 600a924e2101 [anthropic] 3 req, hit ratio 0.0%, 2 break(s)
  #0  claude-opus-5  segments=2 cacheable=2  in=24 read=0 write=1601
      ! CL012 iso-datetime inside the cacheable prefix at system[0]
        → Timestamps change every request. Move them after the last breakpoint or drop them.
  #1  claude-opus-5  segments=4 cacheable=4  in=25 read=0 write=1602
      ✗ prefix broken at system[0] +6190 (~1553 tokens lost, estimate)
        was: '…ime: 2026-09-20T10:15'
        now: '…ime: 2026-09-20T10:16'
  #2  claude-opus-5  segments=6 cacheable=6  in=25 read=0 write=1603
      ✗ prefix broken at system[0] +6190 (~1559 tokens lost, estimate)
        was: '…ime: 2026-09-20T10:16'
        now: '…ime: 2026-09-20T10:17'

Zero dependencies. Python 3.11+. Anthropic Messages API, OpenAI Chat Completions and Responses (with the full history in the request; when previous_response_id keeps the conversation server-side, the request body holds only the new turn and there is no prefix to analyse).

Install

pip install promptcachelint            # core: explicit recording, analysis, CLI
pip install "promptcachelint[httpx2]"  # + transport that records SDK traffic automatically

Quick start

The SDKs accept an http_client. Keep their own client class (it carries the SDK's connection limits and timeouts) and wrap only the transport:

import anthropic, httpx2
import promptcachelint
from promptcachelint.transport import wrap_transport

recorder = promptcachelint.Recorder(
    promptcachelint.LogWatcher(),               # warn in the log the moment a prefix breaks
    promptcachelint.JsonlSink("trace.jsonl"),   # keep a trace for the offline report
)
claude = anthropic.Anthropic(
    http_client=anthropic.DefaultHttpxClient(
        transport=wrap_transport(httpx2.HTTPTransport(), recorder)
    )
)
# openai: openai.OpenAI(http_client=openai.DefaultHttpxClient(transport=wrap_transport(...)))
# async:  anthropic.DefaultAsyncHttpxClient(transport=wrap_transport(httpx2.AsyncHTTPTransport(), recorder))

# ... run your agent as usual ...

print(promptcachelint.analyze(recorder.records).to_text())

Then, or on a trace from production:

promptcachelint report trace.jsonl              # human-readable; broken lines are skipped and counted
promptcachelint report trace.jsonl --json       # machine-readable
promptcachelint report trace.jsonl --fail-on CL001,CL002   # non-zero exit for CI
promptcachelint lint request.json --provider anthropic     # static checks on one body
promptcachelint codes

No SDK, or a gateway of your own? Record explicitly:

recorder.record("anthropic", body, response=response_json)          # non-streaming
recorder.record("anthropic", body, sse_events=events, stream=True)  # streaming

What it reports

What is exact and what is estimated

The diff runs on a canonical rendering of the request: each tool, system block and message block becomes one segment; text blocks keep their text as is, structured blocks are serialized as JSON with sorted keys, and cache markers are stripped (the marker moving forward is not an invalidator). Offsets are character offsets into that block's own text, so they point at the character you wrote. Key order inside JSON blocks is not compared, because the provider parses the JSON before hashing.

Costs come from the provider's own usage fields and are exact. Only "tokens that could have been read" is an estimate (about 4 characters per token) and is labelled as such everywhere.

Sessions

Consecutive requests are diffed within a session. Two modes:

• Explicit: with promptcachelint.session("conv-42"): around the calls, or session_id= on recorder.record. Reliable; use it in production.
• Automatic (default): a request continues the session whose first message block it repeats byte for byte, when its message history is the previous history plus new turns (or an exact retry). Growth of the history is the signature of a continuation, so a tool reorder or a rewritten system prompt mid-conversation stays in the session and is reported. Interleaved conversations that share a system prompt stay apart, and so do two users who both open with "hi" from their second turn on. Two rewrites that would otherwise look like new conversations are recognised and reported as prefix breaks with their own hint: a sliding history window (oldest turns dropped; detected up to 16 blocks deep) and an edited history (old tool results truncated, a turn summarised; detected when most of the previous blocks are still there in order). The report says when grouping was automatic; explicit sessions remain the reliable choice in production.

Privacy and cost

A trace contains full prompts. Before it leaves the machine, pass a redactor to the sink:

from promptcachelint.redact import strip_media, hash_text
promptcachelint.JsonlSink("trace.jsonl", redact=strip_media)  # drop base64 images/PDFs, keep text
promptcachelint.JsonlSink("trace.jsonl", redact=hash_text)    # keep structure and sizes only

With hash_text every text becomes its SHA-256 digest repeated or cut to the same length in characters, so token estimates and size-based findings stay right, different texts stay different, and the report still says which block changed and by how much, but offsets and excerpts point into placeholders.

Memory. Recorder keeps the last 10 000 records by default (keep=; None for unlimited, 0 for a pure live setup that only feeds sinks). LogWatcher runs the analyzer without history: one request per session is remembered for the diff, and sessions idle for six hours are dropped, so a long-running process holds memory proportional to its active conversations, not to its lifetime traffic. The offline report (analyze, promptcachelint report) keeps everything for the trace it is given.

CPU. Recording runs on the caller's thread right after the response body is consumed: one pass over the request body plus an O(1) session lookup. Recorder is thread-safe. On a latency-critical async path with very large prompts, record from a worker instead of the transport.

In tests

from promptcachelint.testing import assert_cache_stable

def test_agent_keeps_its_cache(recorder):
    run_agent(http_client=...)
    assert_cache_stable(recorder.records)   # fails with the rendered report on CL001/002/010/013

Custom gateways

Register an adapter for a proxy that speaks a known dialect on another host:

from promptcachelint.providers import AnthropicProvider, register_provider

class Gateway(AnthropicProvider):
    name = "gateway"
    def matches(self, url: str) -> bool:
        return "llm.internal" in url

register_provider(Gateway())

Not in scope (yet)

Rewriting requests or placing markers for you, Gemini, semantic caches, a proxy server. The diagnosis has to be right before the fix can be automatic.

License

MIT