llmtrafficlens 0.4.4

Profile LLM gateway logs for prefix-cache reuse potential, and export anonymized replayable traces

llmtrafficlens

Analyze an LLM gateway's request log for prefix-cache reuse, then export the log as an anonymized trace that standard benchmark tools can replay.

Two commands:

• profile — how much a prefix cache could save on this traffic, and how much cache memory that takes.
• export — the same log as a Mooncake- or bailian-format trace, with all text removed.

Background

An engine serves a request in two phases: prefill reads the whole prompt at once, then decode emits output tokens one by one. Prefill cost grows with prompt length, and it is pure recomputation whenever two requests begin with the same text — the same system prompt, the same few-shot examples, the same attached document.

A prefix cache avoids that: the intermediate state a prompt produces (its KV cache) stays in memory, and the next request starting with the same text reuses it instead of recomputing. Whether that pays off depends on three things, which are exactly what this tool reports:

• The traffic. If requests share no leading text, nothing can be reused. This is a property of your workload, not of your setup.
• Cache memory. KV state is bulky, so a cache holds a bounded number of tokens and evicts the rest. More memory, more hits — with diminishing returns you can measure instead of guess.
• Routing. Across several workers, a request only hits if it reaches the worker that already holds its prefix. Sending it there is what prefix-aware routing does; the alternative, session affinity, only works when requests carry a session identifier.

Reuse is tracked in fixed-size blocks (16 tokens by default) because engines cache at block granularity, not per character.

Install

pip install llmtrafficlens

No runtime dependencies. Python ≥ 3.10.

profile

llmtrafficlens profile gateway.csv -o report

Input. A CSV with a request_json column holding the OpenAI-format request body. These columns are used when present: response_json, model_name, status_code, and a timestamp column (timestamp, ts, created_at, ...; epoch or ISO-8601, or set --ts-column). Mooncake and qwen-bailian traces are accepted as input too (--format mooncake|bailian).

Output. report.html and report.json, containing:

• hit rate at each cache size, ending with the unbounded case — the ceiling;
• top prefixes by reuse count, and separately by reusable token volume;
• share of requests carrying a session identifier;
• input/output token distributions, streaming ratio, model mix, QPS.

How to read it.

Ceiling — the share of input tokens that could be served from cache if memory were unlimited and every request reached the right worker. It bounds everything else: at a few percent, no amount of engineering makes prefix caching worthwhile on this traffic.

Capacity curve — the hit rate at each cache size, so you can see what the ceiling costs. Where it flattens is the point past which buying memory stops helping. The next section converts token counts to gigabytes for a real model.

Session-identifier share — whether the cheap option is enough. When most requests carry an identifier, pinning each session to a worker captures the reuse. When none do (a common case for API traffic), the reuse sits in prefixes shared between unrelated requests, and only prefix-aware routing reaches it.

The two leaderboards — which prefixes to keep resident. They rank differently, and the token-volume one is what determines savings: a short prefix reused very often contributes almost no reusable volume, while a long prefix reused a few dozen times can account for most of it.

export

llmtrafficlens export gateway.csv --to mooncake -o trace.jsonl   # or --to bailian

A benchmark written by hand, with equally popular prompt groups, reports a higher hit rate than production reaches, because every group stays warm. Replaying the real log avoids that, and the export can be shared, because each request is reduced to one line of structure:

{"timestamp": 1753340000123, "input_length": 1994, "output_length": 117,
 "hash_ids": [4251731047194047120, 8125214104179782736, ...]}

hash_ids is a salted chained block hash: two requests share their first N hashes exactly when they share their first N blocks. Replay tools generate one synthetic block per hash, so the prefix-sharing structure is preserved while the content is not real.

# replay at the recorded pace
aiperf profile --custom-dataset-type mooncake_trace --input-file trace.jsonl --fixed-schedule ...
# replay at 2x the pace, for rate sweeps
aiperf profile --custom-dataset-type mooncake_trace --input-file trace.jsonl --synthesis-speedup-ratio 2.0 ...
# or SGLang
python -m sglang.bench_serving --dataset-name mooncake --dataset-path trace.jsonl ...

--to bailian adds chat_id, parent_chat_id and turn for AIPerf's bailian_trace mode; use it when the log carries session identifiers. Tell the consumer which block size the export used — AIPerf defaults to 512 for mooncake and 16 for bailian (--prompt-input-tokens-block-size).

We have not round-tripped an export through AIPerf or SGLang yet. The formats match their documented schemas, but replay is unverified.

Try it on a public trace

This needs no data of your own, and doubles as a correctness check:

curl -LO https://raw.githubusercontent.com/kvcache-ai/Mooncake/main/FAST25-release/traces/conversation_trace.jsonl
llmtrafficlens profile conversation_trace.jsonl --format mooncake -o mooncake-report

The curve over those 12,031 requests. Cache size is how many tokens of KV the engine can keep resident; the memory column prices that for GLM-5.2 at bf16, 93 KiB per token:

cache size	as GLM-5.2 KV	hit rate
16K tokens	1.5 GiB	3.3%
64K	5.8 GiB	4.2%
256K	23 GiB	4.3%
1M	93 GiB	5.6%
4M	372 GiB	18.5%
16M	1.5 TiB	34.2%
unbounded	—	37.4% (ceiling)

Read it as: this workload gains almost nothing until the cache passes 1M tokens, and most of its ceiling needs KV storage past a terabyte — more than device memory alone holds, which is the case tiered KV stores (host memory, SSD) are built for.

The unit is tokens because bytes depend on the model, and on more than the attention formula. GLM-5.2 caches two things per token:

component	derivation	per token
MLA latent	(512 + 64) × 2 bytes × 78 layers	87.75 KiB
DSA indexer	128 × 2 bytes × 21 full-indexer layers	5.25 KiB
		93 KiB

Only 21 of the 78 layers run a full indexer, the rest reuse the previous one's selection. This matches the kvcache-simulator catalog, which reports 95,232 bytes per token.

Do not carry 93 KiB to another model. Only some architectures have a DSA indexer to account for, and the attention term differs as well: Kimi K2.5 is 68.6 KiB per token, a 32-layer fp16 GQA model 128 KiB. Look yours up with kvcache-simulator list-models. That per-model bookkeeping is why this tool reports tokens and not bytes.

Cross-checked against NVIDIA AIPerf. aiperf analyze-trace (v0.11.0) computes prefix reuse on the same file. Its per-request block hit ratios reproduce ours to floating-point precision:

aiperf analyze-trace conversation_trace.jsonl --block-size 512
# cache_hit_rate: 0.38425808746366197

Our unbounded ceiling reads 37.4% rather than 38.4% because the two aggregate differently, not because they disagree: AIPerf averages each request's hit ratio, weighting a 900-token request the same as a 126,000-token one, while we divide total hit tokens by total input tokens. Computing AIPerf's definition from our own block matching yields 0.3842580874636671, a difference of 5e-15. Token weighting is the right choice for sizing a cache, since it is tokens that occupy it.

Cross-checked against the official simulator. The KVCache.AI hit-rate simulator ships as pip install kvcache-simulator. On the same trace with GLM-5.2 accounting it reports a 39.8% ceiling and an LRU curve of 4.5% at 30 blocks, 5.5% at 1,937, 17.9% at 7,729 and 34.6% at 30,918 — the same shape as ours at comparable block counts.

Its ceiling is 39.8% against our 37.4% for one reason, which its output states: it measures over the last 50% of requests, treating the first half as warm-up, while we measure the whole log. Applying its window to our replay gives 0.3983 against its 0.398. So the two agree; they answer slightly different questions. Ours is what the log as a whole would have achieved from a cold cache, which is the conservative reading; theirs is the steady state after the cache fills, which is what a long-running service sees. Expect our figure to sit a couple of points lower on traces with many first-time prefixes.

The repository also contains a synthetic sample log (examples/, not shipped in the pip package): 360 requests, three shared system prompts with skewed popularity, sparse session identifiers, real timestamps. It reports a 66.4% ceiling, and it shows the two leaderboards disagreeing — 16 tokens × 180 requests is 2.3% of the reusable volume, while 2400 tokens × 30 requests is 55.6% of it.

Methodology

Each metric follows a published construction. Deviations are listed.

• Hit rate — requests are replayed in timestamp order against an LRU block cache. Unbounded capacity gives the "ideal hit rate" of KVCache in the Wild (ATC'25), which is also the Mooncake simulator's infinite-capacity ceiling; bounded capacities give that simulator's capacity curve. LRU is the default eviction policy in both vLLM (FreeKVCacheBlockQueue) and SGLang (--radix-eviction-policy lru), though SGLang evicts radix-tree leaves rather than flat blocks.
  • The simulation models one global cache, i.e. a single worker or perfect routing. With N workers behind hash routing, each sees a partition, so read the curve as fleet-level guidance.
  • An engine's KV pool also holds in-flight requests. Only the remainder is available for reuse.
• Block hashing — chained per-block hashing (hash(parent, block), 16-token blocks; 16 is the smallest block vLLM's FlashAttention backend supports and the granularity it aligns to) follows the qwen-bailian trace convention. Deviation: in approximate mode we hash fixed-size character chunks rather than tokenizer output, so two prompts match only if they are byte-identical.
• Token counts — characters divided by four, unless the log carries usage, in which case exact counts are used and the report says so. CJK-heavy text tokenizes closer to 1–2 characters per token, so absolute counts run low while ratios remain usable.
• Coverage — the ceiling covers sharing between requests only. Intra-session reuse is not measurable without session identifiers in the log. Without a timestamp column, row order is used instead: arrival statistics are omitted and the export is flagged as unfit for fixed-schedule replay.

Privacy

Processing is local. Reports contain aggregates only. An export contains lengths, timestamps, anonymized session identifiers and salted block hashes; prompts cannot be reconstructed from it. The hash key is written to .ltl-salt on first run — keep it unchanged so runs stay comparable, and treat it as a secret.

Related tools and scope

We do not replay traffic, control request rate, or search deployment configurations. AIPerf, SGLang bench_serving and simulators such as Vidur do that, and all of them take a trace as input.

Observability platforms (Langfuse, Helicone, Datadog LLM Observability) record token counts and cost per request, not prefix structure. Trace analyzers such as the Mooncake simulator and aiperf analyze-trace require an already-hashed trace. As of July 2026 we found no tool that goes from a raw gateway log to a hashed trace, which is the step this fills.

License

Apache-2.0