What it does

BenchKit runs established coding, reasoning, knowledge, instruction-following, vision, and long-context benchmarks against local models. It supports OpenAI-compatible servers such as llama.cpp, llama-swap, vLLM, and LM Studio, plus native Ollama.

• Guided terminal UI for connecting, selecting models, running suites, and exploring results
• Headless mode for scripts and CI
• Live streamed responses, reasoning traces, progress, speed, and loop detection
• Direct model evaluation and optional Pi coding-agent evaluation
• Benchmark slicing, tag filters, verifier repair, and choice-order robustness
• JSON, CSV, Markdown, and standalone interactive HTML reports
• Performance profiling and a local dashboard for historical runs

Quick start

BenchKit requires Python 3.11 or newer and uv.

uvx benchkit-cli                # try it without installing
uv tool install benchkit-cli    # or install the `benchkit` command

From a clone of this repository:

uv sync
cp .env.example .env
uv run benchkit

Set your server in ~/.config/benchkit/.env to use it from any folder, in a .env in the directory you run benchkit from (parent directories are not searched), or on the Connect screen, where "Save as default" writes the user config for you:

BENCHKIT_PROVIDER=openai
BENCHKIT_HOST=http://localhost:8080/v1
BENCHKIT_API_KEY=

For Ollama:

BENCHKIT_PROVIDER=ollama
OLLAMA_HOST=http://localhost:11434

benchkit config shows every setting and where it came from.

No inference server available? Explore the complete UI offline:

uv run benchkit --demo

Run benchmarks

The TUI lets you select models, benchmarks, task limits, and optional run modes. For automation, use headless mode:

# Run the first 20 HumanEval tasks
uv run benchkit --headless --models qwen3:8b --benchmarks humaneval:20

# Run multiple models and benchmarks
uv run benchkit --headless \
  --models qwen3:8b,gemma3:12b \
  --benchmarks sanity,gsm8k:50

# Print prompts, reasoning traces, and responses
uv run benchkit --headless --models qwen3:8b \
  --benchmarks sanity --verbose

Task slices accept 20 for the first 20 tasks, -20 for the last 20, and 40-80 for a range.

Useful commands:

uv run benchkit --list
uv run benchkit --help

Benchmarks

Benchmark	Key	Tasks	What it tests
PatchEval	patcheval	20	Real Python bug fixes verified by hidden tests with Pi
Sanity	sanity	25	Fast checks across five core capabilities
Three.js Arena	treejs-arena	25	Creative single-file scenes, scored only on whether they render
Git Surgery	git-surgery	10	Stateful Git operations with Pi
MedFailBench	medfailbench	30	Offline heuristic screen for high-risk clinical responses; not clinical validation, certification, deployment evidence, or model ranking
Aider Polyglot	aider-polyglot	225	Repository editing across six languages with Pi
HumanEval	humaneval	164	Python function completion
HumanEval+	humaneval-plus	164	HumanEval with expanded EvalPlus tests
HumanEval+ TR	humaneval-plus-tr	164	HumanEval+ fully translated to Turkish, function and variable names included
MBPP	mbpp	500	Short Python programming tasks
MBPP+	mbpp-plus	378	MBPP with expanded EvalPlus tests
GSM8K	gsm8k	1,319	Multi-step grade-school math
IFEval	ifeval	541	Verifiable instruction following
RULER	ruler	39 × 6	Practical long-context retrieval from 4K to 128K
RULER Full	ruler-full	6,500 × 6	Research-scale RULER (explicit opt-in)
XSTest	xstest	450	Safe compliance and unsafe refusal with an offline checker
GPQA	gpqa	198	Graduate-level science questions
MMLU-Pro	mmlu-pro	12,032	Reasoning across 14 knowledge categories
BANKING77	banking77	3,080	Route a banking message to one of 77 intents
Code Verifier	code-verifier	500	Judge whether a model-written HumanEval+ solution passes its tests
OCRBench	ocrbench	1,000	Text recognition and text-centric VQA over images
ChartQA	chartqa	2,500	Questions about real charts, 5% relaxed numeric accuracy
MMVP	mmvp	300	Look-alike image pairs that differ in one detail; also reports pair accuracy
V* Bench	vstar	191	Small details in high-resolution images
MMLU	mmlu	14,042	Academic and professional knowledge
ARC	arc	1,172	Grade-school science reasoning
OpenBookQA	openbookqa	500	Elementary science knowledge
WinoGrande	winogrande	1,267	Commonsense pronoun resolution
PIQA	piqa	1,838	Physical commonsense reasoning
BoolQ	boolq	3,270	Passage-based yes/no questions
TruthfulQA	truthfulqa	817	Resistance to common misconceptions
HellaSwag	hellaswag	1,000	Plausible real-world continuations

treejs-arena renders each generated page in headless Chromium, so it needs the browser extra first:

uv sync --extra browser
uv run playwright install chromium

Vision suites (ocrbench, chartqa, mmvp, vstar) send images inline as base64, so they need a vision model (a llama.cpp server started with --mmproj, or an Ollama vision model). Each one downloads a pinned dataset revision on first use. mmvp and vstar are small enough to run in full on one consumer GPU and also run on decision models with an mmproj.

Small suites ship with BenchKit and work offline. Larger datasets (mmlu, mmlu-pro, the vision suites, EvalPlus, RULER's haystacks, Aider Polyglot, PatchEval) are downloaded once into ~/.cache/benchkit the first time a run needs them. List what is installed and fetch everything that is missing ahead of time with:

uv run benchkit datasets
uv run benchkit datasets install            # every missing dataset
uv run benchkit datasets install mmlu       # or just the ones you name

In the terminal app, press d on the setup screen (or click Datasets) for the same list with an Install all missing button.

The CLI registry is the canonical source for current counts, descriptions, and supported perturbations:

uv run benchkit --list

Advanced runs

Compare raw generation with the stock Pi coding agent (Docker required):

uv run benchkit --headless --models qwen3:8b \
  --benchmarks gsm8k:20 --harness both

Give incorrect answers one verifier-guided replacement attempt:

uv run benchkit --headless --models qwen3:8b \
  --benchmarks gsm8k:20 --repair-attempts 1

Repairs default to off and can be set from 0–10. One is a practical starting point, but the right depth depends on what you want to measure.

Test whether multiple-choice accuracy survives reordered answer choices:

uv run benchkit --headless --models qwen3:8b \
  --benchmarks mmlu-pro:100 --perturbation choice-order

Decision models served by llama.cpp (/v1/systemone) are detected automatically and run on the multiple-choice and yes/no suites, scored on accuracy plus calibration (Brier, ECE):

llama-server -hf ggml-org/Kev-4B-GGUF
uv run benchkit --headless --models <decision-model> \
  --benchmarks mmlu-pro:100,boolq:100

Behind a proxy that hides this, such as llama-swap (v262+), name the model:

uv run benchkit --headless --models d1-3b,qwen3:8b --benchmarks gpqa \
  --decision-models d1-3b

or mark it once in the llama-swap config:

models:
  d1-3b:
    metadata:
      decision: true

Profile inference performance:

uv run benchkit perf qwen3:8b
uv run benchkit perf qwen3:8b --depths minimal,4k,16k --gen 256 --reps 10

Results

Every run creates a timestamped directory containing:

results/<timestamp>/
├── results.json
├── results.csv
├── results.md
└── results.html

Open results.html for an interactive report with charts and task-level prompts, responses, reasoning traces, timing, and diagnostics.

Browse completed benchmark and performance runs locally:

uv run benchkit history

Development

uv sync
uv run pre-commit install
uv run ruff check .
uv run ruff format .
uv run pytest

See CONTRIBUTING.md for contributor setup and workflow.

License

Apache License 2.0