llmtrafficlens 0.1.2

LLM gateway traffic profiler: prefix-reuse analysis, cache-hit-rate upper bounds, and anonymized replayable trace export

llmtrafficlens

Point it at your LLM gateway's request log. It tells you how much a prefix cache could save on your real traffic — before you build anything — and turns the log into an anonymized dataset you can load-test with.

                       ┌────────────────────────────────────────────┐
  gateway log (CSV) ──►│ llmtrafficlens                             │
  · request_json       │  tokenize → block-hash → prefix tree       │
  · response_json      │  (all local; prompts never leave the box)  │
  · timestamp, status  └──┬─────────────┬───────────────┬───────────┘
                          ▼             ▼               ▼
                     report.html   trace.jsonl     SGLang bench args
                     "worth it?"   "test with it"  "or synthesize it"

Input → Output

Command	Input	Output	The question it answers
profile	gateway log CSV	report.html + .json	Is my traffic worth prefix-aware routing? What's the ceiling?
export	gateway log CSV	anonymized trace.jsonl	How do I load-test / share this traffic without leaking a single prompt?
fit	gateway log CSV	SGLang bench CLI args	What generator settings reproduce my traffic shape?
validate	public trace	hit-rate numbers	Is this tool's math correct? (checked against the Mooncake simulator)

Input format: a CSV with a request_json column holding the OpenAI-format request body. Optional columns picked up automatically: response_json, model_name, status_code, and a timestamp column (timestamp/ts/created_at/... — epoch or ISO-8601; or set --ts-column). Public Mooncake / qwen-bailian traces are also accepted as input (--format mooncake|bailian).

What you get

1. profile — the decision numbers

pip install llmtrafficlens
llmtrafficlens profile gateway.csv -o report

report.html shows, on your traffic:

• ECHR upper bound — the ceiling on prefix-cache savings: the share of input tokens that would hit under an infinite cache with perfect routing. If this is 3%, stop here; if it's 37%, keep reading.
• LRU capacity curve — hit rate at each cache size, so you can judge how close to the ceiling a real deployment can get.
• Dual-metric prefix leaderboard — top prefixes by reuse count vs by reusable token volume. The heads differ: a 15-token greeting reused 870× is worthless to a cache; a 19K-token system prompt reused 95× is the whole prize. This tells you what your routing actually has to keep resident.
• Session-identifier presence — 0% means your reuse is cross-request shaped and session-affinity routing cannot capture it; prefix-aware routing is required.
• Input/output token distributions, streaming ratio, model mix, QPS.

2. export — an anonymized, replayable dataset

llmtrafficlens export gateway.csv --to bailian -o trace.jsonl

Each request becomes one line — no text, only structure:

{"chat_id": "4abc41ba209004a0", "timestamp": 1753340000123,
 "input_length": 1994, "output_length": 117, "type": "glm-4", "turn": 1,
 "hash_ids": [4251731047194047120, 8125214104179782736, ...],
 "stream": true, "status": 200}

hash_ids is a salted chained block-hash fingerprint (qwen-bailian convention, 16-token blocks): two requests share the first N hashes exactly when they share their first N blocks of tokens. Replay tools regenerate synthetic prompts from the hashes — same hash, same synthetic block — so cache behavior under replay matches production, while the content is fake. Replay with NVIDIA AIPerf:

aiperf profile --custom-dataset-type bailian_trace --input-file trace.jsonl ...

3. fit — generator settings instead of hand-tuning

llmtrafficlens fit gateway.csv --target sglang-gsp

Emits ready-to-run SGLang bench_serving arguments (group count, zipf popularity, prompt lengths) fitted to your measured histogram, plus a fidelity report: the theoretical hit rate the synthetic config produces vs your measured ceiling, so you know when a generator is a faithful stand-in and when you should replay the trace instead.

Verify the math (60 seconds, public data)

curl -LO https://raw.githubusercontent.com/kvcache-ai/Mooncake/main/FAST25-release/traces/conversation_trace.jsonl
llmtrafficlens validate conversation_trace.jsonl --format mooncake
# → echr_upper_bound: 0.3738  (Mooncake conversation trace; the official
#   KV Cache Hit Rate Simulator's infinite-capacity ceiling agrees)
llmtrafficlens profile conversation_trace.jsonl --format mooncake -o mooncake-report

Privacy model

All processing is local. Reports contain aggregates only. Exports contain lengths, timestamps, anonymized session ids, and keyed block hashes — prompts are unrecoverable. The hash key is generated at .ltl-salt (keep it stable across runs, treat it as a secret).

Notes on precision

• Token counts use a chars/4 approximation unless the log carries usage (exact counts are then used and marked in the report). CJK-heavy text tokenizes closer to 1–2 chars/token: absolute counts skew low, ratios (ECHR, leaderboard shares) stay usable.
• Without a timestamp column the profile still works, but arrival stats are omitted and exports are marked unfit for timestamped replay.
• ECHR upper bound covers cross-request sharing under infinite cache, perfect routing, no eviction. Real deployments land below it; intra-session reuse is only visible if the log carries session ids.

Positioning

Observability platforms (Langfuse, Helicone, Datadog LLM) count tokens and cost but never look inside the prompt structure. Trace analyzers (Mooncake simulator, AIPerf analyze-trace) require pre-hashed traces. Capacity simulators (Vidur, SplitwiseSim) consume traces nobody produces from production logs. llmtrafficlens is the missing pipeline: raw gateway logs → structural analysis → decision numbers → standard replayable trace.

License

Apache-2.0