linejudge 0.2.0

The independent line judge for coding agents - runs your agent, then verifies its work itself. Never trust the player's call.

linejudge

The independent line judge for coding agents. Never trust the player's call.

Coding agents routinely claim success. linejudge runs your agent on a goal, then independently verifies the result — real commands, real diffs, real exit codes — and renders its own verdict. The agent's opinion of its own work is never consulted.

What it found

Eight agent-authored patches for eight real sqlite-utils issues, put through every gate the harness has:

evidence	result
Runs succeeded	7/8
Independently verified pass — judged by verifiers, not by the claim	8/8
Regression test proven — fails with the fix reverted	8/8
Diff reviewed against the issue it claims to close	6/8

Three machine gates, 8/8 on every one. Reading the diffs found two that don't fix the bug:

• #439 — the patch counts bytes as utf-8-sig. Fixes utf-16, regresses the default: a 708-byte plain-utf-8 CSV reports 1011. The added test covers utf-16-le only.
• #762 — the constraint parser masks string literals but not SQL comments, so a commented-out -- CHECK (id > 0) is re-emitted as an active constraint. A table that accepted id = -1 starts rejecting it.

Neither is reachable by a test suite that doesn't already know about the bug. Machine verification is necessary and it is not sufficient — so the scoreboard reports those levels separately instead of as one number.

Full evidence trail: PROOF.md. The long version, with both rejects walked through in detail: Eight patches passed every check I had. Two of them were wrong.

See it in 30 seconds, for free

$ python proofs/demo.py --root demo        # 3 tasks, one agent LIES
20260721T…-issue-101-config-loader…: SUCCESS
20260721T…-issue-102-add-json-flag…: FAILED     ← claimed success, wrote nothing
20260721T…-issue-103-docs-quickstart…: SUCCESS

The failed run's agent reported ## Status: SUCCESS. The files_exist verifier checked the filesystem and disagreed. The verdict, not the claim, decides the run status. No tokens spent — see the full sample PROOF.md.

Why

Every agent loop on the market grades its own homework: the model says "done" and the harness believes it. linejudge splits the roles. The agent plays the point; the harness calls the lines:

• Independent verification — a declarative verifier spec (command, files_exist, diff_constraints, http_check) executed by the harness, outside the agent session. Full spec.
• Blast-radius guarding — read-only directories are snapshotted (git status) before and after every run; any unexpected mutation fails the run with diagnostics.
• Verified-diff-only writes — write access goes through a git worktree on an unmerged linejudge/<run_id> branch. The terminal state is a reviewable, verified diff — never a silent merge into your working tree.
• Auditable cross-run learning — a second, tool-less agent call distills each run into a versioned markdown lesson; future runs retrieve the most relevant lessons by tag. No opaque memory, and a poisoning guard keeps rate-limit garbage out of the pool.
• Cost accounting — per-run token/dollar cost captured from the agent's own telemetry into run_cost.json and an append-only runs/ledger.jsonl.
• Local review dashboard — every run's full evidence trail (prompt, claim, diff, verdict, cost, lesson) with an approve/reject gate. The decision is written beside the evidence, never into it.
• Zero dependencies — Python stdlib only, every source file ≤300 lines. You can read the whole engine in an afternoon.

Works with Claude Code headless (claude -p) today; the adapter interface is agent-agnostic by design — run(prompt, cwd, timeout, …) -> RunResult is the entire contract.

Architecture

goal.md ─────► runner ─────► adapter ──────► agent (claude -p / mock / yours)
 (task +          │                             │
  verifiers)      │◄───── REPORT.md ────────────┘   ◄── the CLAIM
                  │
                  ├── guard: read_dirs snapshot before/after (trip ⇒ FAILED)
                  ├── write flow: git worktree ─► write_diff.patch ─► branch
                  ├── verifiers ─────────► verdict.json             ◄── the CALL
                  ├── distill (2nd call) ─► learnings/<id>.md
                  └── ledger ────────────► run_cost.json + runs/ledger.jsonl
                                                │
                              linejudge dashboard (review + approve/reject)

Quickstart

Needs Python 3.10+. Nothing else — no API key required for the mock pipeline.

pip install linejudge

# example goals + the mock demo live in the repo
git clone https://github.com/phillipmex/linejudge && cd linejudge

# 1. See exactly what the harness would send — zero spend
linejudge run goals/examples/hello.md --dry-run

# 2. Full pipeline on a scripted mock agent (one run lies and gets caught)
python proofs/demo.py --root demo

# 3. Review the evidence
linejudge dashboard --root demo        # http://127.0.0.1:8765

To run against a real agent, install Claude Code, set ANTHROPIC_API_KEY (directly or in a gitignored .env.local), and drop the --dry-run:

linejudge run goals/examples/hello.md --root .

Step-by-step walkthrough with a write-mode goal: docs/quickstart.md.

Goal files

A goal is one markdown file: a fenced header (deliberately not YAML — no nesting, no quoting rules, nothing to mis-parse) plus the prompt body.

---
name: fix-config-crash
tags:
  - widget
read_dirs:
  - /path/to/reference-repo      # guarded read-only
write_repo: /path/to/widget      # optional: write cycle via worktree
verifiers:
  - command: python -m pytest -q
  - files_exist: done.txt
  - diff_constraints: max_files=5 deny=**/*.env
timeout_secs: 1800
---
Fix the config loader crash on empty YAML files. …

Docs

Quickstart	install → mock demo → real run → write mode
Verifier spec	built-in verifiers, semantics, authoring custom ones
Adapter guide	plug in any agent backend
Comparison	vs bare claude -p loops, Aider, OpenHands, hosted agents
Governance templates	constitution + definition-of-done for agent fleets
ADR-0001	lineage and open-core rationale
PROOF.md	real run: 8 curated sqlite-utils issues, independently verified
Sample PROOF.md	the claim-vs-verdict gap, rendered from the mock demo

Proof harness

proofs/ turns real GitHub issues into goals and renders an honest scoreboard:

python proofs/generate.py --repo owner/name --limit 5 --verifier "command: pytest -q"
linejudge run proofs/goals/issue-….md --root proofs/root      # per goal
python proofs/stats.py --root proofs/root                     # → PROOF.md

PROOF.md reports runs succeeded and independently verified pass % as separate numbers, because they are separate facts.

Status

v0.1.0. 96 tests, CI on Windows + Ubuntu × Python 3.10/3.12, zero runtime dependencies.

License

Apache-2.0