llmtrafficlens 0.3.2

Profile LLM gateway logs for prefix-cache reuse potential, and export anonymized replayable traces

llmtrafficlens

Point it at your LLM gateway's request log. It answers how much a prefix cache could save on your real traffic — before you build anything — and turns the log into an anonymized trace the standard benchmark tools can replay.

Two commands. Everything downstream of the trace is left to the tools that already do it well.

  gateway log CSV ──► llmtrafficlens profile ──► report.html / .json
   · request_json                                 decision numbers
   · timestamp, status
                  └──► llmtrafficlens export  ──► trace.jsonl
                                                  AIPerf · SGLang · AIBrix
                                                  · Mooncake simulator

Install

pip install llmtrafficlens

Zero runtime dependencies. Python ≥ 3.10.

profile — the decision numbers

llmtrafficlens profile gateway.csv -o report

Input: a CSV with a request_json column holding the OpenAI-format request body. Picked up automatically when present: response_json, model_name, status_code, and a timestamp column (timestamp, ts, created_at, ...; epoch or ISO-8601 — or name it with --ts-column). Public Mooncake and qwen-bailian traces also work as input (--format mooncake|bailian).

Output (report.html + report.json):

• Hit rate by cache size — one LRU simulation per capacity, ending with the unbounded case: your ceiling.
• Dual-metric prefix leaderboard — top prefixes by reuse count vs by reusable token volume. The heads differ, and only the second one matters: a 15-token greeting reused 870× is worthless to a cache; a 19K-token system prompt reused 95× is the whole prize.
• Session-identifier presence — whether reuse is reachable by session affinity at all.
• Input/output token distributions, streaming ratio, model mix, QPS.

Reading it: a 3% ceiling ends the discussion — decline the routing project with evidence. A 40% ceiling with 85% session-id presence means cheap session affinity captures most of it. A 40% ceiling with 0% presence means reuse lives in cross-request shared prefixes that only prefix-aware routing reaches, and the leaderboard names the prefixes it has to keep resident.

The curve answers cache sizing. On the public Mooncake conversation trace it runs 3.3% at 16K tokens, 4.2% at 64K, 5.6% at 1M, 18.5% at 4M, 34.2% at 16M, against a 37.4% ceiling: that workload only takes off past ~4M tokens, so a small cache buys almost nothing. Multiply tokens by per-token KV size for GB, and mind two caveats from Methodology — the simulation models one global cache, and the engine's KV pool also holds in-flight requests.

export — an anonymized, replayable trace

llmtrafficlens export gateway.csv --to mooncake -o trace.jsonl   # or --to bailian

Hand-written benchmarks with uniform prompt groups inflate hit rates, because every group stays warm. Replaying your own traffic does not — and you can share the result, because each request becomes one line of structure with no text:

{"timestamp": 1753340000123, "input_length": 1994, "output_length": 117,
 "hash_ids": [4251731047194047120, 8125214104179782736, ...]}

hash_ids is a salted chained block-hash fingerprint: two requests share the first N hashes exactly when they share their first N blocks. Replay tools regenerate synthetic prompts from the hashes — same hash, same synthetic block — so the prefix-sharing structure of production is reproduced while the content is fake.

Then hand it to the tools built for replay:

# faithful replay at the recorded pace
aiperf profile --custom-dataset-type mooncake_trace --input-file trace.jsonl --fixed-schedule ...
# same trace, 2× the pace (rate sweeps without leaving replay)
aiperf profile --custom-dataset-type mooncake_trace --input-file trace.jsonl --synthesis-speedup-ratio 2.0 ...
# or SGLang
python -m sglang.bench_serving --dataset-name mooncake --dataset-path trace.jsonl ...

--to bailian adds chat_id / parent_chat_id / turn for AIPerf's bailian_trace mode; use it when your log carries session identifiers. Tell the consumer which block size you exported with — AIPerf defaults to 512 for mooncake and 16 for bailian (--prompt-input-tokens-block-size).

Untested claim, stated plainly: we have not round-tripped an export through AIPerf or SGLang ourselves. The formats match their documented schemas; actual replay verification is on the roadmap.

Try it on a public trace

No data of your own required, and it doubles as a correctness check:

curl -LO https://raw.githubusercontent.com/kvcache-ai/Mooncake/main/FAST25-release/traces/conversation_trace.jsonl
llmtrafficlens profile conversation_trace.jsonl --format mooncake -o mooncake-report
# → ceiling 0.3738 over 12,031 requests

Feed the same trace to the official KV Cache Hit Rate Simulator and compare its infinite-capacity ceiling. Our 37.4% is consistent with third-party analysis of this trace (a prefix-caching study reports ≈40% for the conversation workload, arXiv:2505.21919); an automated side-by-side check against the official simulator has not been run yet.

A second, fully synthetic sample lives in the source repository (not in the pip package): 360 requests, three shared system prompts with deliberately skewed popularity, sparse session keys, real timestamps. llmtrafficlens profile examples/sample_gateway.csv reports a 66.4% ceiling and, more interestingly, a leaderboard whose two heads disagree — 16 tokens × 180 requests is 2.3% of the reusable volume, while 2400 tokens × 30 requests is 55.6% of it. Regenerate it with python examples/generate_sample.py.

Methodology

Every number follows a published construction; deviations are listed, not hidden.

• Hit rate — replay requests in timestamp order against an LRU block cache. Unbounded capacity is the "ideal hit rate" of KVCache in the Wild (ATC'25) and the Mooncake simulator's infinite-capacity ceiling; bounded capacities give the Mooncake simulator's capacity curve. LRU is the default eviction policy in vLLM and SGLang. The simulation models a single global cache (one worker, or perfect routing): with N workers behind hash routing each sees a partition, so treat it as fleet-level guidance. The engine's KV pool also holds in-flight requests — the reuse pool is what remains.
• Block hashing — chained per-block hashing (hash(parent, block), 16-token blocks) follows the vLLM prefix-cache block scheme and the qwen-bailian trace convention. Deviation: in approximate mode we hash fixed-size character chunks, not tokenizer output, so two prompts match only if byte-identical.
• Token counts — chars/4 unless the log carries usage (exact counts are then used and marked in the report). CJK-heavy text tokenizes closer to 1–2 chars/token: absolute counts skew low, ratios stay usable.
• Coverage — the ceiling covers cross-request sharing only. Intra-session reuse is invisible unless the log carries session ids. Without a timestamp column, row order stands in: arrival stats are omitted and exports are flagged unfit for fixed-schedule replay.

Privacy

All processing is local. Reports contain aggregates only. Exports contain lengths, timestamps, anonymized session ids, and keyed block hashes — a vendor can reproduce your cache behavior and tune against it without recovering a word of any prompt. The hash key is generated at .ltl-salt — keep it stable across runs and treat it as a secret.

Scope

Observability platforms (Langfuse, Helicone, Datadog LLM) count tokens and cost but never look inside prompt structure. Trace analyzers (Mooncake simulator, AIPerf analyze-trace) require pre-hashed traces. Load generators and capacity simulators (AIPerf, SGLang, Vidur) consume traces nobody produces from production logs (our survey: July 2026). llmtrafficlens fills exactly that gap and stops there: raw gateway logs → structure → decision numbers → a standard trace. Replay, rate control, and deployment search belong to the tools above.

License

Apache-2.0