llmtrafficlens 0.1.8

LLM gateway traffic profiler: prefix-reuse analysis, cache-hit-rate upper bounds, and anonymized replayable trace export

llmtrafficlens

Point it at your LLM gateway's request log. It tells you how much a prefix cache could save on your real traffic — before you build anything — and turns the log into an anonymized dataset you can load-test with.

                       ┌────────────────────────────────────────────┐
  gateway log (CSV) ──►│ llmtrafficlens                             │
  · request_json       │  tokenize → block-hash → prefix tree       │
  · response_json      │  (all local; prompts never leave the box)  │
  · timestamp, status  └──┬─────────────┬───────────────┬───────────┘
                          ▼             ▼               ▼
                     report.html   trace.jsonl     SGLang bench args
                     "worth it?"   "test with it"  "or synthesize it"

Input → Output

Command	Input	Output	The question it answers
profile	gateway log CSV	report.html + .json	Is my traffic worth prefix-aware routing? What's the ceiling?
export	gateway log CSV	anonymized trace.jsonl	How do I load-test / share this traffic without leaking a single prompt?
fit	gateway log CSV	SGLang bench CLI args	What generator settings reproduce my traffic shape?
validate	public trace	hit-rate numbers	Is this tool's math correct? (compare with the Mooncake simulator)

Input format: a CSV with a request_json column holding the OpenAI-format request body. Optional columns picked up automatically: response_json, model_name, status_code, and a timestamp column (timestamp/ts/created_at/... — epoch or ISO-8601; or set --ts-column). Public Mooncake / qwen-bailian traces are also accepted as input (--format mooncake|bailian).

What you get

1. profile — the decision numbers

pip install llmtrafficlens
llmtrafficlens profile gateway.csv -o report

report.html shows, on your traffic:

• ECHR upper bound — the ceiling on prefix-cache savings: the share of input tokens that would hit under an infinite cache with perfect routing. If this is 3%, stop here; if it's 37%, keep reading.
• LRU capacity curve — hit rate at each cache size, so you can judge how close to the ceiling a real deployment can get.
• Dual-metric prefix leaderboard — top prefixes by reuse count vs by reusable token volume. The heads differ: a 15-token greeting reused 870× is worthless to a cache; a 19K-token system prompt reused 95× is the whole prize. This tells you what your routing actually has to keep resident.
• Session-identifier presence — 0% means your reuse is cross-request shaped and session-affinity routing cannot capture it; prefix-aware routing is required.
• Input/output token distributions, streaming ratio, model mix, QPS.

2. export — an anonymized, replayable dataset

llmtrafficlens export gateway.csv --to bailian -o trace.jsonl

Each request becomes one line — no text, only structure:

{"chat_id": "4abc41ba209004a0", "timestamp": 1753340000123,
 "input_length": 1994, "output_length": 117, "type": "glm-4", "turn": 1,
 "hash_ids": [4251731047194047120, 8125214104179782736, ...],
 "stream": true, "status": 200, "ts_exact": true}

hash_ids is a salted chained block-hash fingerprint (qwen-bailian convention, 16-token blocks): two requests share the first N hashes exactly when they share their first N blocks of tokens. Replay tools regenerate synthetic prompts from the hashes — same hash, same synthetic block — so cache behavior under replay matches production, while the content is fake. Replay with NVIDIA AIPerf:

aiperf profile --custom-dataset-type bailian_trace --input-file trace.jsonl ...

3. fit — generator settings instead of hand-tuning

llmtrafficlens fit gateway.csv --target sglang-gsp

Emits ready-to-run SGLang bench_serving arguments (group count, zipf popularity, prompt lengths) fitted to your measured histogram, plus a fidelity report: it simulates the request stream the fitted generator would emit and scores it with the same hit-rate metric as your real traffic. A small gap means the generator is a faithful stand-in; a large gap means it flattens your traffic and you should replay the exported trace instead.

Try it on the bundled sample (no data of your own needed)

The repository (not the pip package) ships a fully synthetic sample log — 360 requests, three shared system prompts with deliberately skewed popularity, sparse session keys, real timestamps (examples/sample_gateway.csv, regenerate with python examples/generate_sample.py):

git clone https://github.com/GMISWE/llmtrafficlens && cd llmtrafficlens
llmtrafficlens profile examples/sample_gateway.csv -o sample-report

What it finds (and why each number matters):

Metric	Value	Reading
ECHR upper bound	66.4%	worth building prefix-aware routing for
Top prefix by count	16 tok × 180 reqs = 2.3% of reusable volume	popularity ≠ value: not worth pinning
Top prefix by volume	2400 tok × 30 reqs = 55.6% of reusable volume	this is what routing must keep resident
Session-id presence	4.2%	reuse is cross-request shaped; affinity alone can't catch it

And fit on the same sample demonstrates the fidelity report doing its job — the gap is large, which is the tool telling you to replay the trace instead of trusting a generator:

llmtrafficlens fit examples/sample_gateway.csv --target sglang-gsp
# → zipf_alpha 1.56, fidelity gap +0.30  ← generator can't represent
#   this mix of shared groups and singletons; use export + replay

Scenarios

"Should we build prefix-cache-aware routing?"

You run a multi-tenant gateway and someone proposes cache-aware routing. Instead of arguing from intuition:

llmtrafficlens profile last-week.csv -o report

Three numbers settle it. ECHR upper bound 4% → decline the proposal with evidence; the traffic has no shared structure worth routing for. Upper bound 40%, session-id presence 85% → cheap session affinity captures most of it; no prefix machinery needed. Upper bound 40%, session-id presence 0% → reuse lives in cross-request shared prefixes; only prefix-aware routing reaches it — fund the project, and attach the report to the design doc as its quantitative basis.

"How much cache memory does each worker need?"

The LRU capacity curve in the report shows hit rate at each cache size. A typical shape: 8% at 64K tokens, 24% at 1M, 31% at 4M, ceiling 37%. The knee (here ~1–4M tokens) is your sizing answer; provisioning past it buys almost nothing. Multiply tokens by per-token KV size for GB.

"Our load test says the cache works great. Production disagrees."

A hand-written benchmark with uniform prompt groups inflates hit rates — every group stays warm. Two honest alternatives:

# closest to production: replay the real structure at the real pace
llmtrafficlens export last-week.csv --to bailian -o trace.jsonl
aiperf profile --custom-dataset-type bailian_trace --input-file trace.jsonl --fixed-schedule

# open-loop rate sweeps: fit a generator, but obey the fidelity report
llmtrafficlens fit last-week.csv --target sglang-gsp
# fidelity gap +0.02 → generator is a faithful stand-in, sweep away
# fidelity gap +0.30 → generator flattens your traffic; use replay above

"The vendor asks what our workload looks like. Legal says no."

export produces lengths, timestamps, anonymized session ids, and keyed block-hash fingerprints — enough for the recipient to reproduce your cache behavior and tune against it, while prompts are unrecoverable (salted keyed hashing; the salt never leaves you). What they learn: your request sizes, arrival pattern, and prefix-sharing structure. What they cannot learn: a single word of any prompt.

Verify the math (60 seconds, public data)

curl -LO https://raw.githubusercontent.com/kvcache-ai/Mooncake/main/FAST25-release/traces/conversation_trace.jsonl
llmtrafficlens validate conversation_trace.jsonl --format mooncake
# → echr_upper_bound: 0.3738 on the Mooncake conversation trace
llmtrafficlens profile conversation_trace.jsonl --format mooncake -o mooncake-report

To cross-check, feed the same trace to the official KV Cache Hit Rate Simulator and compare its infinite-capacity ceiling with the number above. Our 37.4% is consistent with third-party analyses of this trace (a prefix-caching study on the Mooncake traces reports ≈40% for the conversation workload — arXiv:2505.21919); an automated side-by-side check against the official simulator is on the roadmap and has not been run yet.

Privacy model

All processing is local. Reports contain aggregates only. Exports contain lengths, timestamps, anonymized session ids, and keyed block hashes — prompts are unrecoverable. The hash key is generated at .ltl-salt (keep it stable across runs, treat it as a secret).

Methodology and provenance

Every metric follows a published convention; deviations are listed, not hidden:

• Block hashing — chained per-block hashing (hash(parent, block), 16-token blocks) follows the vLLM prefix-cache block scheme and the qwen-bailian trace convention. Deviation: in approximate mode we hash fixed-size character chunks, not tokenizer output; two prompts match only if byte-identical. Exact-tokenizer mode is planned.
• ECHR upper bound — the "ideal hit rate" construction used by KVCache in the Wild (ATC'25) and the Mooncake simulator's infinite-capacity ceiling: replay requests in timestamp order, a block hits if any earlier request produced it. It ignores in-flight timing (a block is reusable the instant it first appears), which keeps it a true upper bound.
• LRU capacity curve — same construction as the Mooncake KV Cache Hit Rate Simulator; LRU is the eviction policy vLLM and SGLang actually use. The curve models a single global cache (one worker, or perfect routing): with N workers behind hash routing each sees a partition, so treat it as fleet-level guidance, not per-worker sizing. Remember the engine's KV pool also holds active requests — the reuse pool is what remains.
• Zipf fit — least-squares slope on log(count) vs log(rank); the standard quick estimator, not the stricter MLE (Clauset et al.).
• Fidelity gap — no bespoke statistic: we simulate the request stream the fitted generator would emit (per SGLang gsp semantics, group sizes at their zipf expectations) and score both the real and the simulated stream with the same ideal-hit-rate metric above. The gap is the difference between the two scores.

Notes on precision

• Token counts use a chars/4 approximation unless the log carries usage (exact counts are then used and marked in the report). CJK-heavy text tokenizes closer to 1–2 chars/token: absolute counts skew low, ratios (ECHR, leaderboard shares) stay usable.
• Without a timestamp column the profile still works, but arrival stats are omitted and exports are marked unfit for timestamped replay.
• ECHR upper bound covers cross-request sharing under infinite cache, perfect routing, no eviction. Real deployments land below it; intra-session reuse is only visible if the log carries session ids.

Positioning

Observability platforms (Langfuse, Helicone, Datadog LLM) count tokens and cost but never look inside the prompt structure. Trace analyzers (Mooncake simulator, AIPerf analyze-trace) require pre-hashed traces. Capacity simulators (Vidur, SplitwiseSim) consume traces nobody produces from production logs. llmtrafficlens is the missing pipeline: raw gateway logs → structural analysis → decision numbers → standard replayable trace.

License

Apache-2.0