Gauntlet

Adversarial multi-agent development harness. Every artifact — PRD, plan, and each implementation phase — runs the gauntlet of adversarial review before it ships: a builder agent implements, an independent reviewer agent attacks the result, a cheap triage model sorts the findings, the builder fixes, and the reviewer confirms the fix against the diff. A localhost judge service gates every tool call the agents make, failing closed.

A local-first, loopback-only console (gauntlet serve) makes every run visible, answerable, and recoverable from the browser, and the CLI exposes the same observability — live log tailing, machine-readable status, and guarded recovery — for headless use.

The canonical spec is PRD-gauntlet.md. The bootstrap plan is runs/gauntlet/plan.md.

Status: the bootstrap is complete — Gauntlet was built by running its own pipeline against itself (phases P1–P7, each adversarially reviewed and human-ratified). It is usable on other repositories via the steps below.

Table of contents

• How it works
• Prerequisites
• Install
  • macOS / Linux
  • Windows
• Configure credentials
• Quick start (≤ 3 commands)
• Authoring a PRD (the repo teaches you how)
• The run lifecycle
• Watching a run (console + observability)
• Command reference
• Configuration
• Safety model
• Development
• Troubleshooting

How it works

A pipeline (YAML) is a sequence of stages; each stage is built from a few step types:

The central invariant is that the working tree is clean and committed at every point where control passes to the reviewer — this is what makes review diffs meaningful and kill -9 resume safe.

Two pipelines ship by default: standard (for real work) and bootstrap (the self-hosting pipeline used to build Gauntlet itself).

Prerequisites

Gauntlet is a thin orchestrator that drives external agent CLIs and model APIs. You need:

The default agent profiles are: builder = claude (model opus), reviewer = codex (model gpt-5.5), triage/judge = gpt-5-mini, escalation = gpt-5. You can repoint any tier to a different provider in config (see Configuration); ANTHROPIC_API_KEY / GEMINI_API_KEY are only needed if you switch the API tiers to those providers.

Install

macOS / Linux

1. Install uv (if you don't have it):

curl -LsSf https://astral.sh/uv/install.sh | sh

2. Install the agent CLIs and sign in to each (follow each tool's own docs):

# Claude Code (builder) — see https://docs.claude.com/en/docs/claude-code
claude --version        # confirm it's on PATH
claude /login           # or however your install authenticates

# Codex CLI (reviewer) — see https://github.com/openai/codex
codex --version
codex login

3. Install Gauntlet as a global tool:

uv tool install gauntlet-spec       # from PyPI; or the git URL below for HEAD
# uv tool install git+https://github.com/johnpletka/gauntlet.git
gauntlet version

The PyPI package is gauntlet-spec, not gauntlet. The bare name gauntlet on PyPI is an unrelated (and broken) project. The installed command is still gauntlet — only the install name differs.

Python 3.10+ is required. If your default interpreter is older, uv will refuse with does not satisfy Python>=3.10. Add --python 3.10 (or newer) to the command and uv will fetch a suitable interpreter automatically.

This puts two console scripts on your PATH: gauntlet (the CLI) and gauntlet-judge-hook (the per-tool-call safety hook, wired automatically by gauntlet init).

Windows

Gauntlet itself is pure Python and runs natively on Windows via uv. Use PowerShell.

1. Install uv:

powershell -ExecutionPolicy ByPass -c "irm https://astral.sh/uv/install.ps1 | iex"

2. Install and authenticate the agent CLIs. Install claude (Claude Code) and codex per their official docs and confirm each is on your PATH:

claude --version
codex --version

Note on the agent CLIs: if a given CLI does not yet ship a native Windows build, install Gauntlet and that CLI inside WSL2 (Ubuntu) and follow the macOS / Linux steps there instead. The orchestrator, judge service (loopback HTTP on 127.0.0.1), and hooks are all cross-platform; the only platform-sensitive dependency is the agent CLIs themselves.

3. Install Gauntlet:

uv tool install gauntlet-spec
# or, for HEAD: uv tool install "git+https://github.com/johnpletka/gauntlet.git"
gauntlet version

The PyPI package is gauntlet-spec, not gauntlet — the bare name is an unrelated, broken project. The command is still gauntlet. If uv reports does not satisfy Python>=3.10, append --python 3.10 (or newer) and it will fetch a compatible interpreter.

Configure credentials

The API tiers (triage, judge, escalation) read credentials from the environment only — never from repo config (so keys never get committed).

macOS / Linux (add to ~/.zshrc / ~/.bashrc to persist):

export OPENAI_API_KEY="sk-..."

Windows — PowerShell (current session):

$env:OPENAI_API_KEY = "sk-..."

Windows — persist across sessions:

setx OPENAI_API_KEY "sk-..."
# then open a new terminal

Run gauntlet doctor (below) to verify everything resolves before your first run.

macOS — the sandboxed verifier and your claude login. The adversarial verifier runs claude in an isolated HOME (it hides ~/.aws/~/.ssh from un-hooked subprocesses). On macOS the claude login lives in the Keychain (no ~/.claude/.credentials.json), and that isolation breaks the CLI's Keychain lookup. The verifier handles this for you: it reads your existing login session from the Keychain and hands it to the sandboxed turn, so a normal claude /login just works — no extra setup.

If you'd rather not depend on the Keychain session (it holds a short-lived token the sandbox can't refresh, and CI has no interactive login), set an explicit long-lived token, which takes precedence:

claude setup-token                       # mints a long-lived OAuth token
export CLAUDE_CODE_OAUTH_TOKEN="sk-ant-oat-..."   # add to ~/.zshenv to persist

Either way this is the one claude credential the verifier is allowed to carry (the same class as the run's judge token); every other secret stays stripped from the sandbox. Linux hosts with file-based ~/.claude/.credentials.json are unaffected. If neither the session nor a token is available, the verifier parks the review closed with an actionable hook-probe message rather than running unauthenticated.

Quick start (≤ 3 commands)

From the repository you want Gauntlet to work on:

gauntlet init        # 1. scaffold config, pipeline, prompts, policy + wire hooks (idempotent)
gauntlet doctor      # 2. validate CLIs, auth, hook wiring, judge, API keys
gauntlet new myfeat  # 3a. scaffold .gauntlet/runs/myfeat/ with a PRD stub
#    ...author .gauntlet/runs/myfeat/prd.md...
gauntlet run myfeat  # 3b. start the pipeline

If the repository already carries committed Gauntlet assets (a teammate ran init before you), you only need to wire this machine's hooks:

gauntlet init --from-repo

gauntlet doctor reports actionable, per-check status — installed CLI versions vs. the verified pin file (.gauntlet/pins.yaml), authentication, hook wiring, judge startability, and ApiAdapter keys — and exits non-zero on any blocker.

Authoring a PRD (the repo teaches you how)

A Gauntlet run starts from a human-authored PRD. gauntlet init installs two committable aids so you don't have to carry the conventions in your head — and a teammate who clones the repo inherits both automatically:

• A Claude Code skill at .claude/skills/gauntlet-prd-author/SKILL.md. In a Claude session, a natural-language request like "help me write a PRD" or "start a Gauntlet run" triggers it; it routes you to this repo's authoring playbook (prompts/prd-author.md, under your asset_root) and the conventions for where the PRD lives and how to scaffold and launch it. It's a thin pointer to the playbook, not a copy, so there's one source of truth.
• A structured stub. gauntlet new <slug> writes a PRD stub with the playbook's full section skeleton and a one-line hint per section, so you start from the right shape. The stub is the committable template <asset_root>/prd-stub.md — edit it to change the house style for every future PRD.

The skill teaches and routes; it never authors the PRD for you. A human writes and ratifies it (FR-10.1): gauntlet run refuses to start while the file is still the stub (marker present, or no substantive content added), so an unfilled skeleton can't become a runnable non-PRD.

Both aids are idempotent and never-clobber: re-running gauntlet init leaves any customization byte-for-byte intact (only an unmodified generated file is ever refreshed, and only after a template version bump). gauntlet doctor includes a warn-only check that the skill is installed and well-formed — it never blocks a run, since the skill gates nothing.

The run lifecycle

A run advances automatically until it hits a human_gate, then parks for your decision:

gauntlet run myfeat              # start (parks at the first gate)
gauntlet status myfeat           # see current step + every step's state
gauntlet approve myfeat          # accept the parked gate; drive to the next one
gauntlet reject myfeat --notes "…"   # send the phase back for another fix round
gauntlet resume myfeat           # resume after an interruption (kill -9 safe)
gauntlet resume myfeat --response "…"   # decide an upstream conflict (see below)
gauntlet report myfeat           # per-step / per-agent cost + token breakdown

• Interrupted runs are resumable. State lives in the run's manifest.json; gauntlet resume re-enters at the last incomplete step. A step that wrote a dirty tree before dying is parked or reset rather than re-run blindly.
• Provider usage limits pause, they don't destroy. A quota/429/usage-limit hit mid-step — including inside a review cycle's sub-agents — parks the run (parked_usage_limit) with the worktree untouched and the agent session preserved; gauntlet resume continues the same session with a short continuation prompt instead of re-running the step. Cycle sub-steps checkpoint as they complete, so a resumed cycle re-enters at the first incomplete sub-step. Builders also commit P<N> wip: milestones inside a phase, bounding worst-case lost work to one milestone. Opt-in resume_on_quota: auto self-resumes at the provider's hinted reset time.
• Laptop sleep is survivable. A driver heartbeat detects host suspension and credits the slept time back to the running step's deadline (capped), so closing the lid neither silently stalls the run nor spuriously kills a healthy step; status reports detected suspensions. Opt-in keep_awake: true wraps the driver in caffeinate -i on macOS.
• Malformed structured artifacts self-repair. Agent-authored artifacts (like the plan's gauntlet-phases block) are validated in-step; the agent gets its own parse error back for a bounded repair loop, and if that fails the run parks (parked_artifact_invalid) for a sanctioned hand-edit — resume re-runs only the validator and audits the edit via content hashes.
• Approved artifacts are immutable. A later phase that finds an approved PRD/plan incomplete halts and surfaces the conflict rather than amending it. You resolve that conflict with gauntlet resume <slug> --response "…" (see Resolving an upstream conflict below), which routes any artifact change back through its own gate rather than letting the builder amend it in place.
• At the final gate a PR.md draft is written under .gauntlet/runs/<slug>/ (it is not opened or pushed — that stays a human action).
• After a run, gauntlet feedback <slug> captures your retrospective notes and triage corrections to feed the self-improvement loop.

Watching a run (console + observability)

A run advances on its own between gates, so the question is usually "where is it now, and does it need me?" Gauntlet answers that two ways — a browser console and CLI primitives that expose the same state for headless/CI use.

The console (gauntlet serve)

gauntlet serve                 # loopback-only, token-authenticated console
gauntlet serve --resume        # reuse/boot the console, open the browser, return
gauntlet run myfeat --watch    # boot/reuse the console, open the browser, then run

gauntlet serve starts a loopback-only, token-authenticated web console that runs strictly above the orchestrator: every control action it offers launches the same sanctioned gauntlet CLI verb you would type, so it inherits every safety invariant rather than being able to weaken one. It lists every run across all slugs with live status / current step / cost, drills into each step's prompt.md, rendered transcript.md, and events.jsonl (with live tailing for running steps), assembles the evidence behind a parked gate and offers Approve / Reject in one place, and classifies a failed/parked run into the action that actually applies. It can also launch and abort runs as supervised children and survive its own restart by re-attaching to live PIDs, and fire desktop / Slack / in-tab notifications on the four moments that need a human (gate reached, escalation parked, run failed, run completed).

gauntlet run --watch ensures the console is up (booting or reusing it), prints its URL, and opens the authenticated console in your browser before running in the foreground; pass --no-browser (on either command) to skip the launch. --console-host / --console-port override the bind (default 127.0.0.1:8765). gauntlet serve --resume does the same boot-or-reuse-and-open without holding the foreground.

CLI observability

gauntlet status myfeat              # driver liveness, run-state, next action
gauntlet status myfeat --json       # the same state as one machine-readable object
gauntlet logs myfeat                # a step's dir + transcript tail (read-only)
gauntlet logs myfeat --follow       # tail a running step's events.jsonl live
gauntlet recover myfeat             # terminate a verified-wedged driver (guarded)
gauntlet run myfeat --interactive   # detach the run, foreground a monitor agent

• status reports driver liveness, the computed run-state, and the next action / recovery hint; --json emits the same payload (schema schemas/status.json) for scripts and CI. The payload carries run elapsed time, token/cost totals (run-level and per agent profile), per-step duration_s/notes and engine-stamped halt_reason/parked_reason enums, heartbeat age with detected suspension intervals, and the quota reset time on a usage-limit park — every parked/halted/failed state is explainable from status alone, no transcript required. Additions are strictly additive (schema_version stays 1); a consumer pinning an older strict schema copy must re-pin on upgrade.
• logs is strictly read-only evidence-on-demand; --follow streams a step's events as they're written (paired with opt-in live step streaming).
• recover terminates a driver only after verifying it is genuinely wedged, then marks its step INTERRUPTED so a plain resume re-enters cleanly — it never kills a healthy run.
• run --interactive[=claude|codex] launches the run detached and hands the terminal to an interactive monitoring agent (wired to the run's judge as the operator's own session); status --interactive attaches the same monitor to an already-running run. An installed gauntlet-operator Claude Code skill routes a supervising session to this repo's recovery playbook.

Resolving an upstream conflict

When a builder finds that the approved PRD or plan is wrong or under-specified, it halts with an UPSTREAM CONFLICT instead of working around the approved artifact (FR-10.4). The step parks; the run is stuck until you decide. The standard, audited way to decide is:

gauntlet resume <slug> --response "<your decision, in plain text>"

The decision is recorded verbatim in the manifest (timestamped, attributed to your operator identity) and injected into a fresh builder run, which re-evaluates the conflict in light of it rather than re-surfacing it. The builder then emits one of three outcomes:

• Proceeds — the decision resolves the conflict within what the approved artifacts already allow (e.g. ratifying one of the options they leave open, or deferring out-of-scope follow-up to FUTURE.md). The run un-sticks and continues.
• Re-parks for an artifact amendment — the decision would require changing approved PRD/plan text (including "proceed even though this contradicts the plan"). There is no proceed-now-amend-later path: amend that artifact on its own branch, take it through its own review-and-gate cycle (FR-10.4), then resume again with a decision that no longer contradicts it.
• Re-parks for clarification — the decision was ambiguous; the builder names what it still needs. Supply another --response.

Notes:

• --response is required to resume a step parked on an upstream conflict. Other parks (e.g. a human_gate) are unaffected — use approve / reject for those, and a plain gauntlet resume for a non-conflict agent park.
• Conflicts do not consume the retry budget — only genuine failures do. You can supply as many --response cycles as it takes; you decide when to stop or abort.
• The whole history of your decisions is preserved in the manifest (steps[N].human_responses, append-only) and reaches git history, so the audit trail of who decided what, and when, is never lost.

Command reference

--no-judge disables the safety judge and is for testing only — it leaves agent tool calls ungated. Don't use it on real work.

Configuration

gauntlet init writes a .gauntlet/ directory in your repo:

• .gauntlet/config.yaml — agent profiles (adapter + model + flags), per-agent commit identities, run timeouts and budgets. References models, not credentials.
• .gauntlet/pins.yaml — the CLI versions and exact flags verified by the contract suite; doctor checks the installed CLIs against it.

Pipelines, prompt templates (versioned data, not code), structured-output schemas, and the judge fast-path policy.yaml all live under .gauntlet/ too — .gauntlet/pipelines/*.yaml, .gauntlet/prompts/, .gauntlet/schemas/, .gauntlet/policy.yaml. The config's asset_root (default .gauntlet in a scaffolded repo) is where the engine resolves them; everything is committable, so a teammate who clones the repo gets the identical workflow. (Gauntlet's own source repo sets asset_root: "." to keep these assets at the repo root as first-class source rather than tucked into a dotfile dir.)

To repoint a tier at a different provider, edit the agent profile's adapter and model in .gauntlet/config.yaml and set that provider's key in your environment (e.g. ANTHROPIC_API_KEY for an anthropic/* model). LiteLLM model naming applies to api adapter profiles.

Per-agent reasoning effort. Any profile (and any pipeline step, which wins over its profile) accepts an optional effort drawn from the canonical enum minimal / low / medium / high. The engine maps the canonical value to each adapter's real surface: claude-code → --effort (which accepts low/medium/high; canonical minimal remaps to low with a load-time warning), codex → -c model_reasoning_effort=…, api → the reasoning_effort param. A value an adapter/model cannot accept is a config-load error, never a silent drop. Optional and no-op when absent. A natural use is a cheaper fixer role for review-fix rounds while the initial builder runs at higher effort:

agents:
  builder:   { adapter: claude-code, model: opus,   effort: high }
  impl_fixer:{ adapter: claude-code, model: sonnet, effort: medium }
  reviewer:  { adapter: codex,       model: gpt-5.5, effort: high }

The judge_llm profile uses this same validated effort value. Its backward-compatible default is minimal; models that reject that tier can set effort: low (or another supported canonical tier). gauntlet doctor executes one live probe through the judge's actual classifier schema, timeout, and effort path, so an incompatible model/effort pair fails preflight instead of denying every agent tool call at runtime.

Mechanical emissions — commit-message drafting and resume-disposition output — run on a designated cheap mechanic: profile in the shipped config, so the builder's constrained provider window is spent on building.

Resilience & window knobs (all default to today's behavior; opt in per knob):

resume_on_quota: notify      # notify (default) | auto — self-resume a
                             #   usage-limit park at the provider's hinted reset
                             #   time (in-process; wants keep_awake or an
                             #   external scheduler re-invoking `resume`)
keep_awake: false            # true wraps the driver in `caffeinate -i` (darwin)
heartbeat_interval_s: 15     # driver heartbeat cadence (suspend detection)
suspend_credit_cap_s: 43200  # max slept time credited back to a step deadline
checkpoint_commits: keep     # keep | squash — builders' intra-phase `PN wip:`
                             #   milestone commits; the phase always ends in a
                             #   `PN:` commit and reviewers always see the
                             #   cumulative range diff either way
triage_concurrency: 4        # bounded pool for per-finding triage calls;
                             #   final triage.json is byte-identical to a
                             #   sequential run on all-success rounds
providers:                   # pre-step window admission (FR-10); absent = off
  anthropic:
    window_hours: 5
    window_budget: 1500000   # in budget_unit
    budget_unit: tokens      # tokens | cost
    enforce: false           # false = advisory warning; true = park pre-step
                             #   (`parked_usage_window`) with zero work in flight
    # fallback_estimate: 50000   # used when the ledger has no history yet

Admission estimates come from the machine-global usage ledger (~/.gauntlet/usage-ledger.jsonl, content-free counts only) that every run appends to; seed it from past runs with gauntlet ledger backfill. The ledger cannot see non-gauntlet usage, so admission is advisory by design — a wrong continue is survivable via the reactive usage-limit park.

Scoped context (pipeline-level). agent_task inputs accept a per-input mode so large artifacts travel by reference instead of being inlined into every prompt — the CLI agents read them in-session, where subsequent turns hit the provider prompt cache:

- id: implement
  type: agent_task
  agent: builder
  inputs:
    - { name: prd.md,  mode: reference }   # inject the path, agent reads it
    - { name: plan.md, mode: phase }       # inject only the current phase's
                                           #   plan section + the full-doc path
  # (bare `- prd.md` still means mode: inline, today's behavior)

reference/phase require a profile whose adapter can read the repo (api profiles can't; pipeline load fails closed, and doctor probes that a reference-capable profile's sandbox can actually read a repo file). Agent-task steps also accept validate: <name> (e.g. plan_phases) to check their output artifact in-step with a bounded self-repair loop.

Safety model

• Agent tool calls (e.g. the builder's shell commands and file writes) pass through a PreToolUse hook → localhost judge service. The judge decides via a deterministic policy fast-path, then an LLM classifier rung, and fails closed (deny) on timeout, parse error, or any unexpected outcome.
• The judge binds 127.0.0.1 only and rejects callers lacking the per-run token. Every decision is written to an audit log.
• The reviewer runs read-only (codex sandbox read-only); any worktree mutation by a reviewer is a detected process violation.
• Permission-bypass flags (e.g. --dangerously-skip-permissions) are rejected by config lint — they would disable the hook layer.

Development

Working on Gauntlet itself:

uv sync                       # create the venv, install deps + package (editable)
uv run pytest                 # unit suite (no credentials required)
uv run pytest -m integration  # contract tests against live CLIs/APIs (needs creds)
uv run gauntlet doctor        # validate your dev environment

uv run pytest runs unit tests only; the integration marker selects the live contract suite, which requires authenticated CLIs and API keys.

Troubleshooting

• gauntlet errors with ModuleNotFoundError: No module named 'gauntlet' (or gauntlet.main) — you installed the unrelated PyPI package via uv tool install gauntlet. Run uv tool uninstall gauntlet, then reinstall the correct package: uv tool install gauntlet-spec (add --python 3.10 if your default interpreter is older).
• A teammate who hasn't installed Gauntlet sees no hook errors. The wired PreToolUse command is an install-tolerant launcher: when gauntlet-judge-hook isn't on PATH it stands aside silently (exit 0) rather than emitting a per-call command not found notice — unless a gauntlet run is active. A shared repo can mix Gauntlet and non-Gauntlet developers freely.
• A run halts with gauntlet-judge-hook not on PATH during an active gauntlet run; failing closed — the hook console script isn't on the PATH the agent CLI sees inside a run, so the launcher fails closed (exit 2) rather than letting the run proceed ungated. Re-run gauntlet init (or gauntlet init --from-repo) and confirm uv tool's bin directory is on your PATH (uv tool update-shell, then open a new terminal). On native Windows, run inside WSL2 — the launcher is POSIX sh (see the install note above).
• doctor reports a stale CLI version — your installed claude / codex differs from .gauntlet/pins.yaml. Re-verify with the integration suite, or update the pin file if the new version is intended.
• A run parks unexpectedly / a step is failed — gauntlet status <slug> shows where; the step's transcript under .gauntlet/runs/<slug>/<run>/steps/ has the detail. gauntlet resume <slug> re-enters safely once the cause is cleared.
• An agent hits a provider session/usage limit mid-step — the engine fails the step closed (it does not fake success). Wait for the limit to reset, then gauntlet resume <slug>.