KV Cache Simulator

kvcache-simulator runs the KVCache.AI hit-rate simulator locally on JSONL traces. It uses the same model accounting formulas as the KV Cache Size Calculator and the same prefix-aware hit-rate semantics as the KV Cache Hit Rate Simulator.

Installation

pip install kvcache-simulator

The default simulation backend compiles and runs a bundled C++ replay core on first use. You need a local C++ compiler such as c++ or clang++. If a compiler is not available, use --backend python.

Quick Start

kvcache-simulator run \
  --trace trace.jsonl.gz \
  --model glm-5.2 \
  --kv-precision fp8_int8 \
  --indexer-precision fp4_int4

The default output is a readable table. Use --format json when another script needs to consume the result.

kvcache-simulator run \
  --trace trace.jsonl.gz \
  --model deepseek-v4-pro \
  --kv-precision fp8_int8 \
  --indexer-precision fp4_int4 \
  --format json \
  --output result.json

List supported model ids:

kvcache-simulator list-models

python -m kvcache_sim ... also works, but the installed CLI command is preferred.

run is the main command. It evaluates the selected trace across a set of KV cache memory budgets. sweep is kept as an alias for users who prefer benchmark terminology.

Input Trace Format

Input is JSONL or JSONL.GZ, one request per line. The minimal accepted format is:

{"block_size":64,"hash_ids":[2001,2002],"input_length":128}

Required fields:

• hash_ids: cache block identities in request-prefix order.
• input_length: prefill input token count for this request.
• block_size: source-native block size. It can be omitted only when --block-size is provided.

Optional fields:

• timestamp: ignored by the simulator. Requests are replayed in file order, so sort production traces by timestamp before running the command.
• output_length: ignored by the hit-rate denominator. Generated output matters only if it appears later in another request's hash_ids.
• block_tokens: advanced field for exact per-block token weights. If present, it must be a positive integer list with the same length as hash_ids.

--block-size is only a fallback for records that omit block_size. If any record declares block_size, the trace-declared value is used and overrides the CLI fallback for the whole trace.

Options

Output Semantics

• Hit rate is measured over the last 50% of requests.
• Budget points that do not fill the cache before that measurement window are omitted, because they are not under memory pressure yet.
• Hit tokens count only the longest continuous cached prefix of each request. If a middle block misses, later blocks in that same request do not count as prefill hits even if their ids are already cached.
• speedup is an ideal prefill-only upper bound: 1 / (1 - hit_rate). 1.0x means no-cache prefill throughput where every prefill input token is computed. It does not include decode, KV lookup, network, batching, scheduling, or memory bandwidth overhead.

Performance Notes

The default C++ backend runs all cache-budget points in one batch after loading the trace and building the prefix trie once. This is usually faster than the Python backend, especially on large traces.

--jobs applies only to the Python backend. It parallelizes independent (policy, cache budget) simulation tasks. More jobs are not always faster: the default budget sweep has only a small number of budget points, tasks have uneven runtimes, and large traces can become memory-bandwidth limited.

Limits

The browser version caps uploads to protect the UI. This local package does not use that browser cap, but the C++ backend currently stores trace events and prefix node indexes in 32-bit arrays, so it supports at most 2^32 - 1 block events. In practice, memory and runtime are usually the real limits before that.

Development

The package bundles two files under src/kvcache_sim/resources/: models.yaml, a copy of data/kv_cache_calculator/models.yaml, and kv-cache-lab-native-sim.cc, the batch C++ core the cpp backend compiles on first use (it takes --policy batch --capacities ... --policies ...; the single-budget scripts/kv-cache-lab-native-sim.cc serves the site's precompute pipeline). After changing the catalog run scripts/sync-simulator-resources.sh; the test suite checks that the bundled copy matches.

pip install -e ./packages/kvcache-simulator
pytest packages/kvcache-simulator