llmtrafficlens 0.3.0

Profile LLM gateway logs for prefix-cache reuse potential, and export anonymized replayable traces

llmtrafficlens

Point it at your LLM gateway's request log. It answers how much a prefix cache could save on your real traffic — before you build anything — and turns the log into an anonymized trace the standard benchmark tools can replay.

Two commands. Everything downstream of the trace is left to the tools that already do it well.

  gateway log CSV ──► llmtrafficlens profile ──► report.html / .json
   · request_json                                 decision numbers
   · timestamp, status
                  └──► llmtrafficlens export  ──► trace.jsonl
                                                  AIPerf · SGLang · AIBrix
                                                  · Mooncake simulator

Install

pip install llmtrafficlens

Zero runtime dependencies. Python ≥ 3.10.

profile — the decision numbers

llmtrafficlens profile gateway.csv -o report

Input: a CSV with a request_json column holding the OpenAI-format request body. Picked up automatically when present: response_json, model_name, status_code, and a timestamp column (timestamp, ts, created_at, ...; epoch or ISO-8601 — or name it with --ts-column). Public Mooncake and qwen-bailian traces also work as input (--format mooncake|bailian).

Output (report.html + report.json):

• Hit rate by cache size — one LRU simulation per capacity, ending with the unbounded case: your ceiling. If the ceiling is 3%, stop here. If it's 37%, the curve tells you how much cache memory it takes to approach it.
• Dual-metric prefix leaderboard — top prefixes by reuse count vs by reusable token volume. The heads differ, and only the second one matters: a 15-token greeting reused 870× is worthless to a cache; a 19K-token system prompt reused 95× is the whole prize.
• Session-identifier presence — 0% means reuse is cross-request shaped, so session-affinity routing cannot capture it and prefix-aware routing is required.
• Input/output token distributions, streaming ratio, model mix, QPS.

export — an anonymized, replayable trace

llmtrafficlens export gateway.csv --to mooncake -o trace.jsonl   # or --to bailian

Each request becomes one line — no text, only structure:

{"timestamp": 1753340000123, "input_length": 1994, "output_length": 117,
 "hash_ids": [4251731047194047120, 8125214104179782736, ...]}

hash_ids is a salted chained block-hash fingerprint: two requests share the first N hashes exactly when they share their first N blocks. Replay tools regenerate synthetic prompts from the hashes — same hash, same synthetic block — so the prefix-sharing structure of production is reproduced while the content is fake.

Then hand it to the tools built for replay:

# faithful replay at the recorded pace
aiperf profile --custom-dataset-type mooncake_trace --input-file trace.jsonl --fixed-schedule ...
# same trace, 2× the pace (rate sweeps without leaving replay)
aiperf profile --custom-dataset-type mooncake_trace --input-file trace.jsonl --synthesis-speedup-ratio 2.0 ...
# or SGLang
python -m sglang.bench_serving --dataset-name mooncake --dataset-path trace.jsonl ...

--to bailian adds chat_id / parent_chat_id / turn for AIPerf's bailian_trace mode; use it when your log carries session identifiers. Tell the consumer which block size you exported with — AIPerf defaults to 512 for mooncake and 16 for bailian (--prompt-input-tokens-block-size).

Untested claim, stated plainly: we have not round-tripped an export through AIPerf or SGLang ourselves. The formats match their documented schemas; actual replay verification is on the roadmap.

Try it on the bundled sample

The repository (not the pip package) ships a fully synthetic sample log — 360 requests, three shared system prompts with deliberately skewed popularity, sparse session keys, real timestamps:

git clone https://github.com/GMISWE/llmtrafficlens && cd llmtrafficlens
llmtrafficlens profile examples/sample_gateway.csv -o sample-report

Metric	Value	Reading
Ceiling	66.4%	worth building prefix-aware routing for
Top prefix by count	16 tok × 180 reqs = 2.3% of reusable volume	popularity ≠ value: not worth pinning
Top prefix by volume	2400 tok × 30 reqs = 55.6% of reusable volume	this is what routing must keep resident
Session-id presence	4.2%	reuse is cross-request shaped; affinity alone can't catch it

Regenerate the sample with python examples/generate_sample.py.

Scenarios

"Should we build prefix-cache-aware routing?" Run profile on last week's log. Ceiling 4% → decline the proposal with evidence. Ceiling 40% with 85% session-id presence → cheap session affinity captures most of it. Ceiling 40% with 0% presence → reuse lives in cross-request shared prefixes and only prefix-aware routing reaches it; attach the report to the design doc as its quantitative basis.

"How much cache memory per worker?" Read the curve. On the public Mooncake conversation trace it runs 4.2% at 64K tokens, 5.6% at 1M, 18.5% at 4M, 27.1% at 8M, 34.2% at 16M, against a 37.4% ceiling — that workload only takes off past ~2M tokens, so a small cache buys almost nothing. Multiply tokens by per-token KV size for GB, and mind the caveats in Methodology: the curve models one global cache, and the engine's KV pool also holds in-flight requests.

"Our load test says the cache works great; production disagrees." Hand-written benchmarks with uniform prompt groups inflate hit rates — every group stays warm. Export your real trace and replay that instead; --synthesis-speedup-ratio covers rate sweeps.

"A vendor wants to see our workload; legal says no." The export carries lengths, timestamps, and keyed block hashes. They can reproduce your cache behavior and tune against it; they cannot recover a word of any prompt.

Verify the math

curl -LO https://raw.githubusercontent.com/kvcache-ai/Mooncake/main/FAST25-release/traces/conversation_trace.jsonl
llmtrafficlens profile conversation_trace.jsonl --format mooncake -o mooncake-report
# → ceiling 0.3738 on the Mooncake conversation trace

Feed the same trace to the official KV Cache Hit Rate Simulator and compare its infinite-capacity ceiling. Our 37.4% is consistent with third-party analysis of this trace (a prefix-caching study reports ≈40% for the conversation workload, arXiv:2505.21919); an automated side-by-side check against the official simulator has not been run yet.

Methodology

Every number follows a published construction; deviations are listed, not hidden.

• Hit rate — replay requests in timestamp order against an LRU block cache. Unbounded capacity is the "ideal hit rate" of KVCache in the Wild (ATC'25) and the Mooncake simulator's infinite-capacity ceiling; bounded capacities give the Mooncake simulator's capacity curve. LRU is the default eviction policy in vLLM and SGLang. The simulation models a single global cache (one worker, or perfect routing): with N workers behind hash routing each sees a partition, so treat it as fleet-level guidance. The engine's KV pool also holds in-flight requests — the reuse pool is what remains.
• Block hashing — chained per-block hashing (hash(parent, block), 16-token blocks) follows the vLLM prefix-cache block scheme and the qwen-bailian trace convention. Deviation: in approximate mode we hash fixed-size character chunks, not tokenizer output, so two prompts match only if byte-identical.
• Token counts — chars/4 unless the log carries usage (exact counts are then used and marked in the report). CJK-heavy text tokenizes closer to 1–2 chars/token: absolute counts skew low, ratios stay usable.
• Coverage — the ceiling covers cross-request sharing only. Intra-session reuse is invisible unless the log carries session ids. Without a timestamp column, row order stands in: arrival stats are omitted and exports are flagged unfit for fixed-schedule replay.

Privacy

All processing is local. Reports contain aggregates only. Exports contain lengths, timestamps, anonymized session ids, and keyed block hashes. The hash key is generated at .ltl-salt — keep it stable across runs and treat it as a secret.

Scope

Observability platforms (Langfuse, Helicone, Datadog LLM) count tokens and cost but never look inside prompt structure. Trace analyzers (Mooncake simulator, AIPerf analyze-trace) require pre-hashed traces. Load generators and capacity simulators (AIPerf, SGLang, Vidur) consume traces nobody produces from production logs (our survey: July 2026). llmtrafficlens fills exactly that gap and stops there: raw gateway logs → structure → decision numbers → a standard trace. Replay, rate control, and deployment search belong to the tools above.

License

Apache-2.0