llmtrafficlens 0.1.4

LLM gateway traffic profiler: prefix-reuse analysis, cache-hit-rate upper bounds, and anonymized replayable trace export

llmtrafficlens

Point it at your LLM gateway's request log. It tells you how much a prefix cache could save on your real traffic — before you build anything — and turns the log into an anonymized dataset you can load-test with.

                       ┌────────────────────────────────────────────┐
  gateway log (CSV) ──►│ llmtrafficlens                             │
  · request_json       │  tokenize → block-hash → prefix tree       │
  · response_json      │  (all local; prompts never leave the box)  │
  · timestamp, status  └──┬─────────────┬───────────────┬───────────┘
                          ▼             ▼               ▼
                     report.html   trace.jsonl     SGLang bench args
                     "worth it?"   "test with it"  "or synthesize it"

Input → Output

Command	Input	Output	The question it answers
profile	gateway log CSV	report.html + .json	Is my traffic worth prefix-aware routing? What's the ceiling?
export	gateway log CSV	anonymized trace.jsonl	How do I load-test / share this traffic without leaking a single prompt?
fit	gateway log CSV	SGLang bench CLI args	What generator settings reproduce my traffic shape?
validate	public trace	hit-rate numbers	Is this tool's math correct? (checked against the Mooncake simulator)

Input format: a CSV with a request_json column holding the OpenAI-format request body. Optional columns picked up automatically: response_json, model_name, status_code, and a timestamp column (timestamp/ts/created_at/... — epoch or ISO-8601; or set --ts-column). Public Mooncake / qwen-bailian traces are also accepted as input (--format mooncake|bailian).

What you get

1. profile — the decision numbers

pip install llmtrafficlens
llmtrafficlens profile gateway.csv -o report

report.html shows, on your traffic:

• ECHR upper bound — the ceiling on prefix-cache savings: the share of input tokens that would hit under an infinite cache with perfect routing. If this is 3%, stop here; if it's 37%, keep reading.
• LRU capacity curve — hit rate at each cache size, so you can judge how close to the ceiling a real deployment can get.
• Dual-metric prefix leaderboard — top prefixes by reuse count vs by reusable token volume. The heads differ: a 15-token greeting reused 870× is worthless to a cache; a 19K-token system prompt reused 95× is the whole prize. This tells you what your routing actually has to keep resident.
• Session-identifier presence — 0% means your reuse is cross-request shaped and session-affinity routing cannot capture it; prefix-aware routing is required.
• Input/output token distributions, streaming ratio, model mix, QPS.

2. export — an anonymized, replayable dataset

llmtrafficlens export gateway.csv --to bailian -o trace.jsonl

Each request becomes one line — no text, only structure:

{"chat_id": "4abc41ba209004a0", "timestamp": 1753340000123,
 "input_length": 1994, "output_length": 117, "type": "glm-4", "turn": 1,
 "hash_ids": [4251731047194047120, 8125214104179782736, ...],
 "stream": true, "status": 200}

hash_ids is a salted chained block-hash fingerprint (qwen-bailian convention, 16-token blocks): two requests share the first N hashes exactly when they share their first N blocks of tokens. Replay tools regenerate synthetic prompts from the hashes — same hash, same synthetic block — so cache behavior under replay matches production, while the content is fake. Replay with NVIDIA AIPerf:

aiperf profile --custom-dataset-type bailian_trace --input-file trace.jsonl ...

3. fit — generator settings instead of hand-tuning

llmtrafficlens fit gateway.csv --target sglang-gsp

Emits ready-to-run SGLang bench_serving arguments (group count, zipf popularity, prompt lengths) fitted to your measured histogram, plus a fidelity report: the theoretical hit rate the synthetic config produces vs your measured ceiling, so you know when a generator is a faithful stand-in and when you should replay the trace instead.

Try it on the bundled sample (no data of your own needed)

The repository (not the pip package) ships a fully synthetic sample log — 360 requests, three shared system prompts with deliberately skewed popularity, sparse session keys, real timestamps (examples/sample_gateway.csv, regenerate with python examples/generate_sample.py):

git clone https://github.com/GMISWE/llmtrafficlens && cd llmtrafficlens
llmtrafficlens profile examples/sample_gateway.csv -o sample-report

What it finds (and why each number matters):

Metric	Value	Reading
ECHR upper bound	66.4%	worth building prefix-aware routing for
Top prefix by count	16 tok × 180 reqs = 2.3% of reusable volume	popularity ≠ value: not worth pinning
Top prefix by volume	2400 tok × 30 reqs = 55.6% of reusable volume	this is what routing must keep resident
Session-id presence	4.2%	reuse is cross-request shaped; affinity alone can't catch it

And fit on the same sample demonstrates the fidelity report doing its job — the gap is large, which is the tool telling you to replay the trace instead of trusting a generator:

llmtrafficlens fit examples/sample_gateway.csv --target sglang-gsp
# → zipf_alpha 1.56, fidelity gap +0.30  ← generator can't represent
#   this mix of shared groups and singletons; use export + replay

Scenarios

"Should we build prefix-cache-aware routing?"

You run a multi-tenant gateway and someone proposes cache-aware routing. Instead of arguing from intuition:

llmtrafficlens profile last-week.csv -o report

Three numbers settle it. ECHR upper bound 4% → decline the proposal with evidence; the traffic has no shared structure worth routing for. Upper bound 40%, session-id presence 85% → cheap session affinity captures most of it; no prefix machinery needed. Upper bound 40%, session-id presence 0% → reuse lives in cross-request shared prefixes; only prefix-aware routing reaches it — fund the project, and attach the report to the design doc as its quantitative basis.

"How much cache memory does each worker need?"

The LRU capacity curve in the report shows hit rate at each cache size. A typical shape: 8% at 64K tokens, 24% at 1M, 31% at 4M, ceiling 37%. The knee (here ~1–4M tokens) is your sizing answer; provisioning past it buys almost nothing. Multiply tokens by per-token KV size for GB.

"Our load test says the cache works great. Production disagrees."

A hand-written benchmark with uniform prompt groups inflates hit rates — every group stays warm. Two honest alternatives:

# closest to production: replay the real structure at the real pace
llmtrafficlens export last-week.csv --to bailian -o trace.jsonl
aiperf profile --custom-dataset-type bailian_trace --input-file trace.jsonl --fixed-schedule

# open-loop rate sweeps: fit a generator, but obey the fidelity report
llmtrafficlens fit last-week.csv --target sglang-gsp
# fidelity gap +0.02 → generator is a faithful stand-in, sweep away
# fidelity gap +0.30 → generator flattens your traffic; use replay above

"The vendor asks what our workload looks like. Legal says no."

export produces lengths, timestamps, anonymized session ids, and keyed block-hash fingerprints — enough for the recipient to reproduce your cache behavior and tune against it, while prompts are unrecoverable (salted keyed hashing; the salt never leaves you). What they learn: your request sizes, arrival pattern, and prefix-sharing structure. What they cannot learn: a single word of any prompt.

Verify the math (60 seconds, public data)

curl -LO https://raw.githubusercontent.com/kvcache-ai/Mooncake/main/FAST25-release/traces/conversation_trace.jsonl
llmtrafficlens validate conversation_trace.jsonl --format mooncake
# → echr_upper_bound: 0.3738  (Mooncake conversation trace; the official
#   KV Cache Hit Rate Simulator's infinite-capacity ceiling agrees)
llmtrafficlens profile conversation_trace.jsonl --format mooncake -o mooncake-report

Privacy model

All processing is local. Reports contain aggregates only. Exports contain lengths, timestamps, anonymized session ids, and keyed block hashes — prompts are unrecoverable. The hash key is generated at .ltl-salt (keep it stable across runs, treat it as a secret).

Notes on precision

• Token counts use a chars/4 approximation unless the log carries usage (exact counts are then used and marked in the report). CJK-heavy text tokenizes closer to 1–2 chars/token: absolute counts skew low, ratios (ECHR, leaderboard shares) stay usable.
• Without a timestamp column the profile still works, but arrival stats are omitted and exports are marked unfit for timestamped replay.
• ECHR upper bound covers cross-request sharing under infinite cache, perfect routing, no eviction. Real deployments land below it; intra-session reuse is only visible if the log carries session ids.

Positioning

Observability platforms (Langfuse, Helicone, Datadog LLM) count tokens and cost but never look inside the prompt structure. Trace analyzers (Mooncake simulator, AIPerf analyze-trace) require pre-hashed traces. Capacity simulators (Vidur, SplitwiseSim) consume traces nobody produces from production logs. llmtrafficlens is the missing pipeline: raw gateway logs → structural analysis → decision numbers → standard replayable trace.

License

Apache-2.0