llmtrafficlens 0.1.1

LLM gateway traffic profiler: prefix-reuse analysis, cache-hit-rate upper bounds, and anonymized replayable trace export

llmtrafficlens

LLM gateway traffic profiler — analyze prefix-reuse structure in your gateway logs, estimate the prefix-cache hit-rate ceiling before you build KV-cache-aware routing, and export anonymized replayable traces in community-standard formats.

Every KV-cache product tells you how much you saved after deployment. llmtrafficlens answers the prior question: is your traffic worth it, and what is the upper bound?

What it does

1. Traffic profile — input/output token distributions, streaming ratio, model mix, error rates.
2. Prefix structure — distinct prefixes, shared-prefix groups, and a dual-metric leaderboard: top prefixes by reuse count vs by reusable token volume. The two heads routinely differ: a 15-token prefix reused 870× is worthless to a cache, a 19K-token prefix reused 95× dominates it.
3. Session-shape detection — presence rate of session identifiers (prompt_cache_key, metadata.*): tells you whether reuse is cross-request shared-prefix shaped (needs prefix-aware routing) or intra-session shaped (session affinity suffices).
4. Reuse potential — ECHR (engine cache hit rate) upper bound under infinite cache + perfect routing, plus an LRU capacity/hit-rate curve.
5. Anonymized export — qwen-bailian-superset JSONL (salted chained block hashes; no text leaves your machine), replayable with NVIDIA AIPerf's bailian_trace mode. A Mooncake downgrade export is planned.
6. Generator fitting — emit SGLang generated-shared-prefix CLI args (zipf popularity fitted to your histogram) with a fidelity report.

Install

pip install llmtrafficlens

Zero runtime dependencies. Python ≥ 3.10.

Quickstart on a public trace

Validate the analyzer against the public Mooncake trace (FAST'25) — the result should match the official KV Cache Hit Rate Simulator's infinite-capacity ceiling:

curl -LO https://raw.githubusercontent.com/kvcache-ai/Mooncake/main/FAST25-release/traces/conversation_trace.jsonl
llmtrafficlens validate conversation_trace.jsonl --format mooncake
llmtrafficlens profile  conversation_trace.jsonl --format mooncake -o mooncake-report

Profile your own gateway logs

Input: a CSV with a request_json column holding the OpenAI-format request body (optional response_json, model_name, status_code columns are picked up when present).

llmtrafficlens profile gateway.csv -o report
# -> report.json + report.html

All processing is local. The report contains aggregates only — raw prompts never appear in any output.

Token counts use a chars/4 approximation unless the log carries usage (then exact counts are used and marked in the report). The chars/4 rule is tuned for English-dominant text; CJK-heavy traffic tokenizes closer to 1–2 chars per token, so absolute token counts skew low while ratios (ECHR, leaderboard shares) stay usable. A keyed hash salt is generated at .ltl-salt on first run; keep it stable to make runs and exports comparable, and treat it as a secret.

Timestamps are auto-detected from common column names (timestamp, ts, created_at, ...; epoch seconds/ms or ISO-8601) or set explicitly with --ts-column. Without one, row order stands in: the profile still works, but arrival stats are omitted and exports are marked unfit for timestamped replay.

Export a replayable anonymized trace

# bailian superset: 16-token blocks, session structure, stream/status
llmtrafficlens export gateway.csv --to bailian -o trace.jsonl

Replay with NVIDIA AIPerf (--fixed-schedule requires real timestamps in the source log; the export warns when timestamps are synthetic):

aiperf profile --custom-dataset-type bailian_trace --input-file trace.jsonl ...

Fit a load generator to your traffic

llmtrafficlens fit gateway.csv --target sglang-gsp

Emits --gsp-* arguments (group count, zipf alpha, prompt lengths) plus a fidelity report: the theoretical ECHR the synthetic config would produce vs your measured upper bound, so the flattening loss is explicit.

Reading the numbers

• ECHR upper bound counts cross-request shared prefixes only, under infinite cache / perfect routing / no eviction. Real deployments land below it; intra-session reuse is invisible unless your sample carries session identifiers.
• Session identifier presence 0% means session-affinity routing cannot capture any of the observed reuse — a prefix-aware policy is required.
• Block size defaults to 16 tokens (qwen-bailian convention; finer and engine-page aligned). Mooncake input implies 512.

Positioning

Observability platforms (Langfuse, Helicone, Datadog LLM) stop at token/cost counting. Trace analyzers (Mooncake simulator, AIPerf analyze-trace) require pre-hashed traces. Simulators (Vidur, SplitwiseSim) consume traces nobody produces from production logs. llmtrafficlens covers the missing pipeline: raw gateway logs → structural analysis → decision numbers → standard replayable trace.

License

Apache-2.0