inference-doctor 0.4.0

A pull request can pass every test and still make inference slower. Putting a benchmark in CI is the obvious fix, and it goes wrong in three ways:

• false alarms: GPU benchmarks are noisy, so a gate on two numbers fails changes that did nothing, and the team turns it off,
• missed regressions: with two or three runs, a real 12% slowdown can land inside the noise and pass,
• apples to oranges: the two sides ran on a different GPU, model or load, and the difference is the setup, not the change.

inference-doctor compares repeated baseline and candidate runs and returns PASS or FAIL only when the data can support it. Otherwise it returns INCONCLUSIVE and says why. It reads the files vllm bench serve, AIPerf, GuideLLM and SGLang already write.

_{A real H100 run (examples/inference-ci-reliability-001), rendered from the CLI by tools/render_readme_media.py. Throughput is 4.19% lower, but one noisy baseline block leaves an interval from -3.33% to +11.17%. Six runs cannot rule out a 10% regression, so there is no merge recommendation.}

How it decides

1. Is the comparison valid? Same GPU, model and benchmark parameters on every run, no failed requests, the metric present everywhere, and at least 3 runs per side. If not, the result is INCONCLUSIVE and the reason names the check that failed.
2. How big is the change, and how sure are we? A 90% confidence interval for the regression (positive is worse), from a Welch t interval on the log ratio of means.
3. Verdict. FAIL when the whole interval is at or past the threshold (default 10%), PASS when the whole interval is below it, INCONCLUSIVE when it contains the threshold. More runs narrow the interval.

Outliers are flagged but never removed: a bad run widens the interval instead of being hidden. Details and references: measurement quality.

Quick start

Python 3.11 or newer. No GPU: the example results are checked in, so the clone is only for them.

pip install inference-doctor
git clone https://github.com/dundysm/inference-doctor.git && cd inference-doctor

inference-doctor compare-runs \
  --baseline examples/quickstart/baseline \
  --candidate examples/quickstart/candidate \
  --metric texts_per_second

PASS: the candidate is 2% slower, and the interval, +1.08% to +2.92%, is well below 10%. Point --candidate at examples/quickstart/regressed-candidate to see a FAIL.

Use your benchmark's own output

No converter needed. Put each side's runs in a directory and point compare-runs at it:

inference-doctor compare-runs \
  --baseline results/main \
  --candidate results/my-branch \
  --metric output_token_throughput --metric ttft_p99_ms

Repeat --metric to gate on several at once: the result is FAIL if any metric fails, INCONCLUSIVE if none fails but one cannot be decided, and PASS only when all pass.

Metrics: output, total and request throughput, and TTFT, TPOT, ITL and end-to-end latency at mean, median, p95 and p99. Most tools do not record the GPU; pass --gpu to label it, or record it in the results (for vLLM, --metadata gpu=...). The input formats page lists every field read from each tool.

examples/vllm-bench-serve is a small example in vLLM's format where throughput passes and TTFT fails, so gating on both catches what a throughput-only gate would let through.

In GitHub Actions

Run your benchmark on your own GPU runner, then gate on the results:

- name: Gate inference performance
  uses: dundysm/inference-doctor@v1
  with:
    baseline: results/main
    candidate: results/my-branch
    metric: output_token_throughput, ttft_p99_ms
    min-repetitions: "5"

The Action writes a job summary with the verdict, the interval and the reason, and exposes them as outputs. FAIL, INCONCLUSIVE and errors fail the step, so noisy evidence cannot approve a change. inference-doctor does not provision GPUs or run the benchmark. See GitHub Actions.

How much to trust it

Simulated with known truth, at default settings, from gate calibration (4,000 trials per row):

The rest of each row is INCONCLUSIVE. Right at the 10% threshold, each wrong verdict happens about 5% of the time; further from it, at most 0.5%. Noise costs decisions, not correctness. Use 5 or more runs per side. These rates assume independent, roughly normal runs; interleave baseline and candidate runs to stay close to that.

Commands

compare-runs options: --metric (repeatable), --regression-percent (10), --confidence (0.90), --min-repetitions (3), --stability-cv-percent (an optional CV ceiling, off), --format (auto), --gpu, --model, --guidellm-benchmark and --json.

Exit codes, the same for every command: 0 PASS, 1 FAIL (for diagnose, a finding at or above --fail-on), 2 INCONCLUSIVE (or UNSTABLE for compare), 3 error with no decision (bad arguments or input, an unreachable Prometheus, or an internal error). inference-doctor --version prints the version.

What has been measured

• The decision rule: the calibration table above, regenerated by tools/gate_simulation.py and checked by tests/test_gate.py on every CI run.
• One real GPU run: examples/inference-ci-reliability-001, six runs per side of BAAI/bge-m3 on one H100 across two vLLM versions. The honest answer was INCONCLUSIVE, and it still is.
• The format readers: tested against records built from each tool's source code. They have not yet been run on a large set of real files from each tool, so please report any file they misread.
• Not yet: a gate running in another project's CI. See STATUS.md.

Help test it

The most useful thing right now is a real change: run compare-runs on results from your own benchmark and open a Benchmark feedback issue saying what it decided, whether that was right, and what was confusing. Redact prompts, credentials, private model names and customer data.

Contributing

See CONTRIBUTING.md. The changelog is CHANGELOG.md. The experiments that informed the design live under experiments/.

python -m pip install -e ".[dev]"
python -m pytest -q
ruff check . && ruff format --check . && mypy

License

Apache-2.0. See LICENSE.