kneebench

Point it at your vLLM server. Learn how many users it can really serve, where it breaks, why, and which flag to change next.
Benchmarks measured against physics, explained in plain words, with your own numbers.

Why this exists

Your team deployed a model on a GPU. Someone asks: "How many users can it take?" You run a benchmark and get a wall of numbers: TTFT P99, ITL, tokens per second at twelve concurrency levels. Which one is the answer? Is 6 ms per token good for this GPU? Did the server run out of memory, or just get slow?

Benchmark tools measure well. They don't explain. kneebench runs the measurements that matter, checks them against what the hardware can physically do, and writes the conclusion in plain words with your own numbers. A real verdict from a Llama 3.1 8B server on one H100 NVL:

For your chat assistant traffic (1,024 tokens in, 1,024 out), this server comfortably handles about 8 users at your target of 50 ms per token and 500 ms to the first token for 99 out of 100 requests. It breaks at about 11 because time to first token crosses your target, not because memory is full: the KV cache was only 2.9% used there.

Every finding says what was measured, what it means, why it happens (with a link to a short lesson) and what to try next. You change your server, rerun, and compare.

Screenshots

Verdict first. Capacity in your users and your targets, with a traffic-light scorecard.	The one plot that matters. Throughput against tail latency, and the best point under your target.
Three walls. Saturation, your latency target and memory; the lowest one is your capacity.	Checked against physics. How close one request gets to what the GPU allows.
A map, not one number. Every request size at the same load, checked against your targets.	What if? Move the targets and watch capacity move, from stored data with no rerun.
Detect, then ask. vLLM version, model, context length and KV cache are read from the endpoint.	Every run kept. Reports regenerate from stored data; export to PDF, HTML, Markdown or JSON.

Quickstart (macOS or Linux)

uv tool install git+https://github.com/S1LV3RJ1NX/kneebench   # kneebench and its bundled AIPerf
kneebench                                                     # opens the web UI at http://127.0.0.1:8765

The first start opens on a demo run: a real benchmark of Llama 3.1 8B on one H100, every phase, so you can explore a full report before benchmarking your own server. Delete it any time; kneebench demo brings it back (set KNEEBENCH_NO_DEMO=1 to skip it).

Need uv? curl -LsSf https://astral.sh/uv/install.sh | sh. Gated models (such as Llama) need HF_TOKEN so kneebench can download the tokenizer.

Prefer the terminal? Everything in the UI is also a command:

kneebench doctor --url http://your-vllm-host:8000      # check the setup, with fixes
kneebench run --url http://your-vllm-host:8000 --gpu h100-sxm   # see `kneebench gpus`
kneebench report 1 --format pdf                         # or md, json, html
kneebench run ... --shapes 2000/500                     # your typical request size
kneebench report 1 --slo-tpot-ms 30                     # what-if: other targets, no rerun
kneebench compare 1 2                                   # before and after a change

If your server needs a key, set KNEEBENCH_API_KEY (the CLI) or paste it into the form (the UI); it is never saved. Entered the wrong GPU? kneebench report <run-id> --gpu <id>, or Correct GPU in the report, recomputes everything from stored data with no rerun. PDF export downloads a headless Chromium once: kneebench doctor --install-pdf.

An early alpha is also on PyPI (uv tool install kneebench --prerelease allow); install from GitHub as above until 0.1.0. The install is large (about 800 MB) because AIPerf brings transformers, pandas and friends. To work on kneebench itself, see CONTRIBUTING.md.

Where to run it

kneebench only sends HTTP requests, so it runs anywhere: a laptop, a Linux box or a pod. Where it runs changes the numbers:

• Close to the server (same cluster or region) gives numbers you can act on.
• Across the internet adds network time to every first token, and a slow client can become the bottleneck. kneebench measures the distance on every run (TCP connect, TLS handshake, a plain /health round trip), judges first-token targets on the server's side of the network, and warns when the network dominates.

kneebench never starts, stops or reconfigures your server. It only benchmarks an existing OpenAI-compatible endpoint, one run at a time.

What it measures

Every run also checks that the test itself is trustworthy: steady state reached (throughput counts only the window at full load), client not the bottleneck, outputs really as long as requested, tails and not just medians, errors counted as failures, gateways not limiting load, versions recorded.

Did a change keep answers right?

Speed is half of a change like FP8 or a 4-bit checkpoint. The other half is quality (lesson 54), so every guided experiment can ask a fixed set of questions before you change anything and again afterwards:

• Before: the current model answers every question twice, at temperature 0 and 16 at a time. The second pass is the noise floor: how much answers move with nothing changed.
• After: the same questions again, compared question by question: what went right to wrong and back, the paired accuracy change with a 95% interval, a McNemar test, and every category.
• The rule comes first. You set the allowed drop (default 1 point at 95% confidence) before any answer exists. The verdict is pass, fail or inconclusive, and the changed answers are shown side by side to read.

Answers are checked by rules (exact, contains, number, multiple choice, regex), not by a model. kneebench bundles a 290-question starter set with answers a program can check; for the decision that matters, use your own prompts as JSONL (prompt or messages, answer, match, category). From the command line:

kneebench quality ask --repeat 2          # before the change: a pass plus the noise floor
kneebench quality ask --role after        # after the change
kneebench quality compare 1 3 --floor 2   # question by question, against the rule

How to read the report

Diagnosis first, evidence second, next steps last: a short verdict, a traffic-light scorecard, the findings that need attention, a ranked "try these next" list with copy-ready flags for your vLLM version, then the curve and the walls. Each finding says what was measured, what it means, why it happens (with a link to the lesson it comes from) and what to do. Problems open by default; background findings collapse to one line; healthy checks fold into a single list. The order follows your product type: a chat team sees time to first token first, a reasoning team sees memory first.

Exports are built for people and for LLMs: the PDF keeps real text, puts a data table next to every chart, ends with every raw number, and embeds the Markdown and JSON reports as attachments.

Limitations

• Endpoint only. No GPU telemetry; memory and queue findings come from vLLM's /metrics. If your gateway hides /metrics, those findings are marked as not visible.
• Synthetic prompts. Random prompts never share a prefix, so prefix caching can't help them. Real chat traffic often does better on time to first token.
• Client location matters. See Where to run it.
• No Windows. macOS and Linux only (WSL2 works).
• vLLM first. Other OpenAI-compatible servers work with reduced findings; SGLang support is on the roadmap.

Roadmap

• M0 Skeleton: packaging with bundled, pinned AIPerf; kneebench doctor; CI.
• M1 Detect, theory and baseline: auto-detection, physics floors, SQLite, single-request baseline, Markdown report.
• M2 MVP: concurrency sweep, the three walls, web UI, full report with charts and PDF export.
• M3 Rate sweep: Poisson and bursty arrivals.
• M4 Shape sweep: prompt and output length, your own request size, the shape map.
• M5 What-ifs and compare: move the SLO and watch capacity move; estimate other request sizes; before and after runs.
• M6 Release: v0.1.0 on PyPI.
• Later: MoE and MLA physics, GuideLLM backend, SGLang servers, optional friendlier wording layer.

Credits

• NVIDIA AIPerf (Apache-2.0) generates the load. kneebench bundles a pinned version.
• The analysis follows the Inference Engineering notes, lessons 24 to 27.
• GuideLLM (Apache-2.0) provides the mock server used in integration tests.
• Geist and Source Serif 4 (SIL Open Font License) set the UI and reports.

If kneebench saved you a day of squinting at benchmark tables, you can buy me a coffee.

License

MIT. See LICENSE. Screenshots are regenerated with scripts/make_screenshots.py.