goalie 0.1.0

Verification harness for AI-generated code

goalie

Verification harness for AI-generated code. Checks that a diff actually does what it should, using oracles independent of whoever (or whatever) wrote it — and returns machine-readable verdicts an agent loop can act on.

Commands

goalie diff-test     # old vs new implementation on generated inputs
goalie properties    # generate + run property-based tests  [planned]
goalie mutate        # property tests by injecting bugs     [planned]
goalie bench         # performance regression check         [planned]

diff-test

For refactors, the old implementation is a free oracle. goalie extracts changed functions from the git diff, generates inputs with Hypothesis, and runs both versions:

{
  "target": "core/rolling.py::rolling_max",
  "verdict": "FAIL",
  "examples": 2000,
  "seed": 42,
  "counterexample": { "kwargs": { "xs": "[1, 2, 2]", "w": "2" }, "old": "2", "new": "1" }
}

Deterministic, reproducible from the seed, no LLM involved. Add one line to your AGENTS.md / CLAUDE.md and your agent has to prove its refactor preserved behavior before calling it done.

properties [planned]

An LLM proposes Hypothesis invariants for a function; they're written to .goalie/ as plain code you review and commit. Generation uses a model once — execution is deterministic forever.

mutate [planned]

Trust check for the property suite: inject small bugs (<= → <, max → min) and re-run. Properties that don't catch planted bugs won't catch real ones. Outputs a mutation score.

bench [planned]

pyperf old vs new with a regression threshold. Catches the O(n²) that passes every test.

Roadmap

v0 — diff-test end-to-end on pure-ish typed functions v0.1 — pipx install goalie, CI exit codes, kwargs/exceptions handling v1 — properties + mutate v1.x — MCP server mode, bench, stateful testing

Out of scope: sandboxing, non-Python, greenfield code with nothing to diff against.

Status

diff-test v0 implemented. Design spec: docs/superpowers/specs/2026-07-24-diff-test-design.md

Quickstart

uv sync
uv run goalie diff-test

Report written to .goalie/latest.json and .goalie/reports/. To waive an intentional behavior change:

uv run goalie waive path/to/file.py::function_name --reason "<why>"

Running tests

Agent-verifiable suite (green):

uv run pytest --ignore=tests/human -q

Gated suite — red until the maintainer implements the human-owned modules (src/goalie/discovery.py, src/goalie/orchestrator.py):

uv run pytest tests/human -v