kvplan

Plan vLLM deployments without the GPU.

Tell kvplan a model, your GPUs and your vLLM settings. It tells you whether vLLM will start, how much KV cache you will get, and how many full-length requests fit at once. It reads the model's real files from Hugging Face (only the headers, never the weights), so it works for new models the day they are uploaded.

$ kvplan check unsloth/Meta-Llama-3.1-8B-Instruct --gpu L4

unsloth/Meta-Llama-3.1-8B-Instruct on 1 x L4 (max_model_len 131072, max_num_seqs 256, gpu_memory_utilization 0.92)

  budget         20.69 GiB of 22.49 GiB per GPU
  weights        14.96 GiB per GPU (exact, from the weight files)
  runtime        0.83 GiB estimated: activation peak 0.22 GiB, torch.compile 0.11 GiB, CUDA graphs 0.25 GiB, other 0.25 GiB
  KV cache       4.91 GiB available (131,072 bytes per token per GPU, auto, block size 16)
  capacity       40,208 tokens; max concurrency at 131,072 tokens per request: 0.31x

  RESULT         vLLM will not start
  error          one request at max_model_len=131072 needs 16.00 GiB of KV cache, but only 4.91 GiB is available; vLLM's estimate of the longest length that fits is 40192
  try            --max-model-len 40192 (or lower)
  try            --kv-cache-dtype fp8 halves the KV cache
  try            --tensor-parallel-size 2 on 2 GPUs splits weights and KV further

Status: alpha. Weight and KV sizes are computed exactly from the model files. The runtime overheads (activation peak, torch.compile buffers, CUDA graphs) are fitted to vLLM 0.32 startup logs from text-model runs on RTX A6000 and A5000 GPUs, and the multimodal encoder memory is derived from vLLM's code and checked on 27 more runs; see Accuracy for what that does and does not cover.

Why

vLLM decides its KV cache size at startup, on the real GPU: it reserves gpu_memory_utilization of the total GPU memory, loads the weights, profiles a forward pass, and gives whatever is left to the KV cache. If one request at max_model_len doesn't fit, it refuses to start. You find this out after renting or buying the GPU.

Other tools approximate this with parameter count × bytes per parameter plus a fixed overhead. That misses quantisation scales, padded vocabularies, tied embeddings, sliding-window layers, latent (MLA) caches and how tensor parallelism splits or replicates KV heads. kvplan reproduces vLLM's own accounting instead. See docs/accounting.md for every rule and the vLLM source it comes from.

Install

pip install kvplan

or the latest development version: pip install git+https://github.com/ShawnNotFound/kvplan.

No dependencies beyond Python 3.10+. For gated models (Llama, Gemma), run kvplan login or set HF_TOKEN; see Gated models.

Use

kvplan check MODEL [--gpu NAME | --gpu-mem-gib GIB] [vLLM options] [--download]
kvplan inspect MODEL
kvplan recommend [--gpu NAME] [--gpus N|any] [--task chat|code|vision|audio|omni] [--priority quality|balanced|speed]
kvplan search WORDS...
kvplan download MODEL [--local-dir DIR]
kvplan login

MODEL is a Hugging Face repo id, a local directory or file, or just a model name. For a name without an owner, such as Qwen2.5-7B-Instruct, kvplan uses the most downloaded repo with exactly that name and says which. If nothing matches, or a repo id is misspelled, it lists the closest popular models instead (and in a terminal, lets you pick one). Without --gpu or --gpu-mem-gib, kvplan plans for the GPUs on this machine (--gpu auto). check understands the vLLM options that change memory: -tp/--tensor-parallel-size, --max-model-len, --max-num-seqs, --max-num-batched-tokens, --gpu-memory-utilization, --kv-cache-dtype, --block-size, --enforce-eager. Add --json for scripts and CI. The exit code is 0 when vLLM should start and 1 when it won't. It is 3 ("start not confirmed", "starts": null in the JSON) when a term kvplan cannot size could decide the outcome: an encoder whose startup profiling it cannot size, an architecture vLLM has no native implementation of, or attention geometry read from weight shapes. The output then names the unsized term, the memory left under kvplan's estimates, and what to do. Exit code 2 is a kvplan error.

In a terminal the result is coloured: green when vLLM should start, red when it won't, yellow when the start is not confirmed, with errors in red and warnings in yellow. --color never (or the NO_COLOR environment variable) turns colour off and --color always forces it; files, pipes and the JSON, CSV and Markdown formats are never coloured.

For the most precise result, give the GPU memory exactly as CUDA reports it (--gpu-mem-gib); the --gpu presets use the slightly larger nvidia-smi totals.

On the machine that will run vLLM, --gpu auto reads the GPUs from nvidia-smi, including memory other processes already hold. vLLM reserves a fraction of total memory, so a desktop session on a laptop GPU or another job on a shared server can stop it from starting even when the model fits:

$ kvplan check Qwen/Qwen2.5-1.5B-Instruct --gpu auto
...
  gpu 0          NVIDIA GeForce RTX 5090 Laptop GPU: 24,463 MiB total, 4,478 MiB in use (nvidia-smi)
  RESULT         vLLM will not start
  error          gpu_memory_utilization=0.92 asks for 21.98 GiB, but only about 19.02 GiB is free after the CUDA context and other processes. ...
  try            lower --gpu-memory-utilization to 0.79

With tensor parallelism it uses the first -tp GPUs, or the ones you name with --gpu-index 2,3.

$ kvplan inspect openai/gpt-oss-20b
openai/gpt-oss-20b
  architecture   gpt_oss (read from config)
  layers         24: 12 sliding, 12 full (window 128)
  attention      64 heads, 8 KV heads, head_dim 64
  experts        32 (all resident in GPU memory)
  context        131072
  KV per token   49,152 bytes per GPU at tp=1 (auto)
  weights        12.82 GiB per GPU at tp=1 (checkpoint 12.82 GiB in 3 file(s))

When vLLM should start, check ends with the command to download the model and the matching vllm serve line:

  RESULT         vLLM should start

  download       kvplan download Qwen/Qwen2.5-1.5B-Instruct   (or add --download to this check)
  serve          vllm serve Qwen/Qwen2.5-1.5B-Instruct --max-model-len 8192

kvplan download fetches the weight files kvplan sized plus the configs, tokenizer and chat template vLLM needs, and nothing else (no duplicate .bin, .pth or GGUF copies). With huggingface_hub installed (any machine with vLLM has it) the files go into the Hugging Face cache, so vllm serve REPO_ID finds them; otherwise, or with --local-dir, into a directory, resuming interrupted downloads.

Find a model

kvplan recommend lists popular Hugging Face models that vLLM should start on your GPUs, each one checked the same way as kvplan check. In a terminal it asks what you need: how many GPUs to use (one, all, or the fewest that work), the task, quality or speed, and the context length. Pass the answers as options, or --no-input for the defaults, to skip the questions.

$ kvplan recommend --gpu A5000 --task vision --priority speed --max-model-len 4096

vision models for 1 x A5000, speed (smaller models first: faster replies and more requests at once), 4,096-token context

   #  model                                                  size GPUs weights/GPU   KV tokens requests at once
   1  Qwen/Qwen2.5-VL-3B-Instruct-AWQ                        3.8B    1    3.29 GiB     441,140           107.7x
   2  Qwen/Qwen2.5-VL-3B-Instruct                            3.8B    1    7.12 GiB     348,207            85.0x
   3  Qwen/Qwen2.5-VL-7B-Instruct                            8.3B    1   15.57 GiB      65,440            16.0x

  Checked 28 of 32 candidates from Hugging Face's most downloaded image-text-to-text models. Ranked by size and downloads, not by benchmark scores.
  ...
Number for full details, d<number> to download, Enter to finish:

Pick a number to see the full kvplan check output, or d and a number to download it. Candidates are the most downloaded models for the task on Hugging Face. kvplan leaves out models vLLM has no native implementation of, GGUF and MLX conversions, base and draft models, and anything whose weights cannot fit, then orders the rest by size: larger first for quality, smaller first for speed, about a third of the memory for balanced. Size and popularity are only rough signs of quality, so treat the list as a shortlist to try, not a ranking. Image generation (Stable Diffusion, FLUX) is not served by vLLM, so kvplan does not recommend those.

kvplan search llama 3.1 8b instruct finds repos by a loose name, most downloaded first.

Gated models

Gated models (Llama, Gemma and others) need a Hugging Face token, and you need to accept the model's licence on its Hugging Face page. Either:

• run kvplan login and paste a read token from https://huggingface.co/settings/tokens. kvplan checks it and saves it where huggingface-cli login does ($HF_HOME/token, or HF_TOKEN_PATH), so vLLM and huggingface-cli use it too; or
• set HF_TOKEN in your environment (for example in your shell profile). It takes precedence over a saved token.

--token on any command overrides both. With a token, recommend also considers gated models.

Use it from scripts and apps

Output. Every command that prints results takes --format text|json|jsonl|csv|markdown (--json is short for --format json) and -o/--output PATH. The format follows the file name when you don't give one (-o results.csv writes CSV). A path ending in /, or an existing directory, gets one file per model. With --append, CSV and JSON Lines output is added to the file, so repeated runs build a log. JSON holds everything kvplan computed; the other formats hold one flat row per model (model, GPU, tp, context, verdict, weights, KV cache, KV tokens, concurrency, problems, vllm serve command).

Many models at once. kvplan check --models FILE checks every model in a file and prints one table, or writes one CSV, JSON or Markdown file. The file is a text file with one model per line, a CSV with a model column plus any option columns (gpu, tp, max_model_len, kv_cache_dtype, ...), or a JSON list of names or objects; - reads standard input. A model that cannot be read gets an error row instead of stopping the run. The exit code is the worst result: 0 all start, 3 some not confirmed, 1 some will not start, 2 some could not be checked.

$ kvplan check --models models.csv -o results/ --format csv
kvplan: wrote results/kvplan-check.csv

Config file. Put defaults in ~/.config/kvplan/config.toml (or config.json, or any file named by KVPLAN_CONFIG or --config). Top-level keys apply to every command; a table named after a command applies to that command only. Options on the command line still win.

gpu = "A6000"
format = "csv"
output = "results/"

[check]
max_model_len = 8192
tp = 2

[recommend]
task = "code"
priority = "speed"

TOML needs Python 3.11 or newer; JSON works everywhere.

Python. The command line is a thin layer over kvplan's Python API, which returns plain data and never prints or asks:

import kvplan

r = kvplan.check("Qwen/Qwen2.5-7B-Instruct", gpu="A6000", max_model_len=8192)
r.starts            # True, False, or None for "start not confirmed"
r.kv_tokens         # KV-cache capacity in tokens
r.to_dict()         # everything, as JSON-ready data (the same as --format json)
r.serve_command     # "vllm serve Qwen/Qwen2.5-7B-Instruct --max-model-len 8192"

kvplan.inspect("openai/gpt-oss-20b")["kv_bytes_per_token"]
kvplan.search("llama 3.1 8b instruct", limit=5)
kvplan.recommend(task="chat", priority="quality", gpu="A6000", gpus="any")["recommendations"]
kvplan.download("Qwen/Qwen2.5-7B-Instruct")      # returns where the files went

Without gpu or gpu_mem_gib, check plans for the GPUs on the machine it runs on. Errors are kvplan.KvplanError (a tuple to catch); an unknown model raises kvplan.ModelNotFound, whose suggestions lists the closest popular models.

What it covers

In a sample of the 300 most-downloaded text-generation models on Hugging Face, kvplan's sources describe 93% without a token and 98% with one (details in docs/coverage.md).

Not yet supported: models with Mamba or linear-attention layers (Qwen3-Next, Jamba, Nemotron-H, Falcon-H1), DeepSeek V3.2 sparse attention, Gemma 4 (full-attention layers with their own head size), encoder-decoder models, pipeline/expert/data parallelism, LoRA, speculative decoding, and llama.cpp. kvplan says so instead of guessing.

Accuracy

Checked against vLLM 0.32.0 on RTX A6000 and RTX A5000 GPUs.

• Text models (Qwen2.5 1.5B, Gemma 2 2B, Llama 3.2 3B, Llama 3.1 8B; CUDA graphs and eager; tensor parallelism 1 and 2): kvplan's KV-cache token count is within 1.9% of vLLM's in all 15 runs (mean 0.5%). These runs were used for calibration, so they are in-sample. On two RTX A6000s at tensor parallelism 2, which were not used, Llama 3.1 8B is within 0.22%.
• Multimodal models: kvplan reproduces the dummy images and videos vLLM profiles at startup for Qwen2-VL, Qwen2.5-VL and Qwen2.5-Omni, including the effect of --mm-processor-kwargs max_pixels and --limit-mm-per-prompt, and derives the vision encoder's memory from vLLM's code. On 16 fit runs (Qwen2.5-VL 7B, Qwen2.5-Omni 3B and 7B, Qwen2-Audio 7B on an A6000) it is within 0.72%; on 11 held-out runs (A5000, tensor parallelism 2) within 2.24%. Other encoders are not sized: for MiniCPM-o 2.6 kvplan was 19–23% high, and it now warns that the figure is uncertain for any encoder it cannot size.
• MoE and quantised models: on 12 runs of Granite 3.1 1B-A400M (MoE), Qwen2.5 1.5B AWQ and GPTQ, and Qwen3 1.7B FP8, none of them used for fitting, kvplan is within 1.01% (mean 0.33%). An fp8 KV cache did not start on the RTX A6000 or A5000 (compute capability 8.6) with vLLM 0.32, so kvplan says "start not confirmed" for it on GPUs older than 8.9.

Not measured yet: large MoE and MLA models, the size of an fp8 KV cache, other vLLM versions, Hopper or Blackwell GPUs. The full table, method and known gaps are in docs/validation.md.

Roadmap

• v0.2 (released): model search and recommendations, downloads, gated-model login, output formats and files, config files and a Python API
• v0.3: a measurement script anyone can run on their GPU, llama.cpp profile, published validation table
• v0.4: place: several models across several machines with mixed GPUs

Licence

Apache-2.0