delphik 0.1.0

Pull the open benchmark-defect set into your terminal and run a fair, pinned eval.

delphik

Pull the open benchmark-defect set into your terminal, hold out the broken tasks, pin the version, and run your own fair eval.

delphik reads the open defect store (github.com/delphik-ai/open-defect) and tells you which tasks of a benchmark are currently broken or have a fix in flight — so you can exclude them and compare models on the same clean set.

Install

pip install delphik        # (or: pipx install delphik)

No configuration needed — it reads the public open-defect repo directly.

Use

# what's broken or in-flight in a benchmark right now?
$ delphik defects swebench-verified
swebench-verified  @ main
● 36 found   ● 22 fixing   → 49 tasks held out
  + 13 benchmark-level caveats (whole-benchmark; can't hold out per task)

# list the benchmarks in the store
$ delphik benchmarks

Run a clean, pinned eval with Harbor

--exclude-args prints Harbor -x <task> flags, ready for command substitution:

harbor run -d swebench-verified@2.0 \
  $(delphik defects swebench-verified --exclude-args) \
  -a claude-code -m <model>

Harbor excludes the held-out tasks (-x / --exclude-task-name) and pins the benchmark version (-d name@version), so any two runs are compared on the same clean, versioned set.

Reproducibility

The held-out set is a snapshot of an open-defect commit. Pin it with --ref:

delphik defects swebench-verified --ref <commit>     # reproducible set
delphik defects swebench-verified --refresh          # re-pull latest

What counts as "held out"

• held-out tasks — task-specific defects that are found (open) or fixing (fix in flight). These are excluded from a fair eval.
• benchmark-level caveats — defects in the harness/grading/dataset that affect the whole benchmark. They can't be held out per task, so they're reported separately, never folded into the count.
• fixed tasks are healthy again and are not held out.

Options

flag	meaning
--unresolved	held-out set (found + fixing) — the default view
--exclude-args	print -x <task> args for $(...) into Harbor
--json	emit the full DefectSet as JSON
--ref <ref>	read the store at a branch/tag/commit (default: main)
--refresh	re-download the store snapshot
--repo <path>	read a local delphik-open-defects checkout (dev)

Data is open under CC BY 4.0. Apache-2.0 code.