fbc-cache-lint 0.1.0

GPU-free, cross-engine inference-cache linter for vLLM /metrics and llama.cpp slot logs

fbc-cache-lint

Lint your LLM inference cache. Find the money it's leaking.

fbc-cache-lint is a GPU-free, cross-engine linter for inference-cache telemetry. Point it at a recorded vLLM /metrics dump or a llama.cpp llama-server slot log and it prints ranked, opinionated findings — each quantified in wasted tokens / KV bytes / dollars and paired with a concrete remediation naming the exact flag to change. It reads recorded dumps: no GPU, no live server, nothing to attach to production.

Metrics dashboards tell you the hit rate is 55%. cache-lint tells you why it fell from 80%, what it's costing you, and which flag to flip.

Install

pipx install fbc-cache-lint      # isolated CLI
# or
pip install fbc-cache-lint

Requires Python 3.11+. Zero third-party dependencies — the linter is pure standard library, so the install is small and fast. That's the whole thing; no clone needed.

Quickstart — run it now, on real data

The package ships two real llama-server captures (phi-3-mini Q4_K_M, build 8227, same tool/model/machine) so you can see it work with no dump of your own. Start with the healthy run:

fbc-cache-lint demo

cache-lint report — llama_server_slots.REAL.phi3-mini.log
source: llama.cpp slot log (251 events)
trace: 27 requests, 83.6% prefix reuse (4,172/4,989 prompt tokens), 2 cold starts

No cache findings above the configured thresholds. OK.

----------------------------------------------------------------------------
Costs are estimates: $0.05/1k prefill tokens, 131,072 B/token KV,
512 recompute tokens/preemption (override with --price-* flags).

That is a healthy run — 83.6% of prompt tokens served straight from the KV cache, sub-2 s decodes, no eviction churn — and the linter says so, with numbers. (A linter that only ever shouts is noise; confirming a cache is healthy is half the job.)

Now the same tool, same model, same machine — deliberately misconfigured (prefix caching disabled + an oversized generation cap on ~1.1k-token prompts) so there is a real problem to catch. It is a real run, not a hand-authored fixture; the report says so:

fbc-cache-lint demo misconfig --fail-on medium   # exit 1 so CI can gate on it

cache-lint report — llama_server_slots.REAL.INDUCED-MISCONFIG.phi3-mini.log
source: llama.cpp slot log (55 events)
NOTE: REAL run, DELIBERATELY MISCONFIGURED to demonstrate detections — not a found-in-the-wild trace.
trace: 6 requests, 0.0% prefix reuse (0/6,746 prompt tokens), 6 cold starts
1 finding  |  843.2 MiB KV

[1] MEDIUM long_decode_kv_pin  (llama.cpp)
    Stranded/pinned blocks from long-decode holds
    impact: 843.2 MiB KV
    6 slot(s) held a large KV prefix through a long decode: worst was slot 0
    — 1,126-token prefix pinned for 25s of decode. 843 MiB of KV pinned
    across these holds.
    fix: Long decodes hold their prompt KV non-evictable and can starve
         other requests. Cap generation length per request (`--n-predict` /
         `n_predict`), reduce `--parallel` so one long decode does not
         monopolise the cache, or offload idle slot KV with `--slot-save-
         path` + the `/slots/{id}/save` endpoint.

Two real runs, same linter: 84%-reuse healthy → OK; caching-off + uncapped generation → a flagged, quantified problem naming the exact knob (--n-predict).

What it finds

Finding	What it catches	Remediation names	Needs
prefix_cache_hit_rate_decay	Hit rate sliding over time (scaling event, prompt-template drift)	session-affinity / stable shared prefix	vLLM /metrics series
eviction_churn	KV cache oversubscribed, preempting + recomputing running requests	--gpu-memory-utilization, DYN_KVBM_CPU_CACHE_GB	vLLM /metrics series
route_utilisation_skew	Round-robin routing giving ~1/N hit rate across replicas/tenants	sticky routing, shared prefix tier (LMCache)	vLLM /metrics (per-route)
long_decode_kv_pin	Long decodes pinning a big KV prefix, starving other requests	--n-predict, --parallel, slot offload	llama.cpp slot log
prompt_order_breaks_prefix	Shared content that isn't a prefix, so it can't be reused	move static content to the front	llama.cpp slot log (LCS)

Findings are ranked by estimated cost (dollars, then wasted tokens, then KV bytes).

Honest scope: which engine surfaces which finding

A finding can only fire on the telemetry that carries its signal:

• A llama.cpp slot log (like the two shipped demos) surfaces the llama.cpp-sourced findings — the long-decode KV pin above, and (on builds that log LCS token counts) the prompt-order break. It does not carry the counters the dollar-quantified findings read, so those stay silent on a slot log — a coverage boundary, not a clean bill.
• A vLLM /metrics series surfaces the three time-series findings — hit-rate decay, eviction churn, and per-route skew, the ones quantified in dollars. Diffing consecutive scrapes is how the windowed rates are computed, so record a series, not a single snapshot.

This dev box has no vLLM cluster to record a real /metrics series from, so those three findings are demonstrated on a labelled synthetic dump (real metric names/types/labels, hand-authored values). Its report says so in the footer:

cache-lint report — vllm_metrics_scrapes.SYNTHETIC.prom
source: vLLM /metrics (5 scrapes)
2 findings  |  est. $11.77 wasted  |  235,324 tokens  |  28.7 GiB KV

[1] HIGH   eviction_churn  (vllm)     impact: $10.29 / 205,824 tokens / 25.1 GiB KV
[2] HIGH   prefix_cache_hit_rate_decay  (vllm)   impact: $1.48 / 29,500 tokens / 3.6 GiB KV
... footer: NOTE: SYNTHETIC demo data — hand-authored fixtures, not a live capture.

Point the tool at your own recorded dump for real numbers on your workload.

Use your own data

vLLM — record a /metrics time-series (the linter diffs consecutive scrapes to get windowed hit-rate and eviction trends):

while :; do echo "# scrape_unix_ms $(date +%s%3N)"; curl -s localhost:8000/metrics; sleep 15; done > scrapes.prom
fbc-cache-lint report scrapes.prom

A single snapshot works too, but time-dependent findings need a series. You can also pipe straight in:

curl -s localhost:8000/metrics | fbc-cache-lint report -

llama.cpp — capture llama-server stderr (slot logs need verbose: -lv 1):

llama-server -m model.gguf -lv 1 2> slots.log
fbc-cache-lint report slots.log

The format is auto-detected; force it with --source vllm or --source llama.cpp.

JSON + CI

--json emits the same findings as a machine-readable document (summary + ranked findings, each with currency, remediation, and raw evidence). The exit code makes it a CI gate:

Code	Meaning
0	ran cleanly; no finding at/above the --fail-on severity
1	ran cleanly; a finding met/exceeded --fail-on (default: high)
2	usage / IO / unrecognised-input error

fbc-cache-lint report scrapes.prom                 # fail only on HIGH (default)
fbc-cache-lint report scrapes.prom --fail-on none  # report, never fail
fbc-cache-lint report scrapes.prom --fail-on medium --json

The dollar and KV-byte figures are estimates from --price-per-1k-tokens, --kv-bytes-per-token, and --recompute-tokens-per-preemption; set them to your model and pricing for accurate numbers.

About the demo data

The two demo captures are real: byte-identical slices of actual llama-server runs (phi-3-mini, build 8227), so their reports carry no synthetic disclaimer. The misconfigured one is labelled DELIBERATELY MISCONFIGURED … not a found-in-the-wild trace — a real run staged to demonstrate a detection, never presented as one caught in the wild. The .SYNTHETIC. vLLM example is format-faithful but hand-valued, and every report over it says so in the footer.

The bigger picture

cache-lint is the shippable wedge of a larger project. The same repository also contains the FBC codec — a factored block codec for inference-cache metadata (a separate, gated technical component; the linter shares no code with it and never pulls its scientific-stack dependencies). The codec's one honest, reproducible compression result — and, just as prominently, the domains where it loses — is written up in the codec benchmark.

Source, issues, and the full roadmap: https://github.com/the10kclub/fbc. Licensed under Apache-2.0.