goalie 0.2.0

Verification harness for AI-generated code

goalie

Status: learning project, not maintained. goalie was built to explore this idea end to end — design, implementation, dogfooding on itself, publishing — and it does what it says. It is not actively developed and issues will likely go unanswered. If you want this class of tool for real work, look at CrossHair's diffbehavior command (old-vs-new behavior comparison via symbolic execution) and Hypothesis's hypothesis write ghostwriter (property-test generation). Both are maintained and cover this space well.

Verification harness for AI-generated code. Checks that a diff actually does what it should, using oracles independent of whoever (or whatever) wrote it — and returns machine-readable verdicts an agent loop can act on.

Commands

goalie diff-test     # old vs new implementation on generated inputs
goalie properties    # generate + run property-based tests  [planned]

diff-test

For refactors, the old implementation is a free oracle. goalie extracts changed functions from the git diff, generates inputs with Hypothesis, and runs both versions:

{
  "target": "core/rolling.py::rolling_max",
  "verdict": "FAIL",
  "examples": 2000,
  "seed": 42,
  "counterexample": {
    "kwargs": { "xs": "[1, 2, 2]", "w": "2" },
    "old": "2",
    "new": "1"
  }
}

Deterministic, reproducible from the seed, no LLM involved. Add one line to your AGENTS.md / CLAUDE.md and your agent has to prove its refactor preserved behavior before calling it done.

properties

An LLM proposes Hypothesis invariants for a function; they're written to tests/properties as plain Python code you review and commit. Generation uses a model once — execution is deterministic forever. Automatic functions update detection by checking signature changes upon running the code. Properties are run via pytest as simple tests, without any additional infrastructure.

Roadmap

v0 — diff-test end-to-end on pure-ish typed functions ✅ v0.1 — pipx install goalie, CI exit codes, kwargs/exceptions handling ✅ v0.2 — properties ✅ v0.3 mutate vx.x — MCP server mode, bench, stateful testing

Out of scope: sandboxing, non-Python, greenfield code with nothing to diff against.

Status

diff-test v0 implemented. Design spec: docs/superpowers/specs/2026-07-24-diff-test-design.md

Quickstart

uv sync
uv run goalie diff-test

Report written to .goalie/latest.json and .goalie/reports/. To waive an intentional behavior change:

uv run goalie waive path/to/file.py::function_name --reason "<why>"

Running tests

Agent-verifiable suite (green):

uv run pytest --ignore=tests/human -q

Gated suite — behavioral tests for the human-owned modules (src/goalie/discovery.py, src/goalie/orchestrator.py), green since they were implemented:

uv run pytest tests/human -v