ConstraintLoop

ConstraintLoop is an evidence-based completion gate for AI coding agents. Instead of relying on a human to inspect every generated line, it requires the agent's work to pass an explicit, versioned contract of tests, static checks, metrics, artifacts, and independent model rubrics.

The central distinction is deliberate:

• Deterministic constraints produce reproducible evidence: exit codes, parsed metrics, and validated artifacts. Required deterministic failures block autonomous completion. A human may explicitly waive an exact local evidence snapshot with a non-empty reason; CI ignores every waiver and remains blocking.
• Non-deterministic constraints apply a written rubric through OpenAI, Anthropic, or any command that speaks ConstraintLoop's JSON protocol. They are advisory by default. A required rubric must run at least twice and declare a majority quorum.

ConstraintLoop supports Claude Code, Codex, and Gemini CLI through their hook lifecycles. CI is the final authority: it ignores local caches and human waivers.

Bounded convergence loops are implemented. The design keeps ConstraintLoop in control of evidence, budgets, and stopping while native Claude or Codex loops perform at most one requested repair per transition. See docs/convergence-loops.md.

Completion loops can also require the current coding session to generate and investigate N domain-grounded failure scenarios before stopping. Claude Code, Codex, and Gemini CLI use their existing session context and tools; this challenge gate requires no separate model evaluator. See session challenge gates.

At a glance

flowchart LR
    G[User goal] --> A[Coding agent]
    A --> C[Versioned contract]
    C --> D[Commands and metrics]
    C --> R[Optional rubric review]
    D --> E[Fresh evidence snapshot]
    R --> E
    E -->|pass| S[Completion allowed]
    E -->|fail| F[One focused repair]
    E -->|pending| W[Wait without repair]
    F --> A
    W --> E
    E -->|budget reached| H[Human decision]

Choose your path

Quick start

uv tool install constraintloop
# Or: pipx install constraintloop

constraintloop init
constraintloop setup --adapter all
# Or add --pre-push above to wire heavyweight push gates locally.
constraintloop run
constraintloop ci

constraintloop init detects existing Python and Node tooling and writes a plain constraintloop.yml. It does not install tools or silently invent gates. Review and commit the contract.

Install ConstraintLoop as an isolated tool instead of adding it to the target project's environment. This avoids dependency conflicts with the application. If you intentionally run setup through uvx, generated hooks pin the current ConstraintLoop version. You can choose another persistent invocation with, for example, constraintloop setup --hook-executable "pipx run constraintloop".

The five core commands above establish this flow:

sequenceDiagram
    participant U as User
    participant A as Agent
    participant CL as ConstraintLoop
    participant T as Project tools
    U->>CL: init + review contract
    U->>CL: setup hooks
    A->>CL: run change/stop/push phase
    CL->>T: execute ready constraints
    T-->>CL: exit codes, metrics, artifacts
    CL-->>A: pass, repair, wait, or escalate
    CL->>T: ci reruns without cache/waivers

Contract

version: 1
settings:
  max_auto_retries: 2
  hook_output_limit: 4096

constraints:
  tests:
    kind: command
    command: [python, -m, pytest, -q]
    phases: [stop, push, ci]
    watch: ["src/**/*.py", "tests/**/*.py", pyproject.toml]
    retry:
      max_attempts: 3
      exit_codes: []  # Do not retry assertion failures; retry process startup only.
      retry_start_errors: true
      delay_seconds: 2

  integration_tests:
    kind: command
    command: [python, -m, pytest, -q, -m, integration]
    phases: [push, ci]
    watch: ["src/**/*.py", "tests/**/*.py", pyproject.toml]
    needs: [tests]
    timeout_seconds: 900

  coverage:
    kind: metric
    command: [python, -m, pytest, --cov, "--cov-report=json:coverage.json"]
    parser:
      type: json
      source: file
      file: coverage.json
      path: totals.percent_covered
    threshold: {operator: gte, value: 85}
    needs: [tests]
    phases: [stop, ci]

  database_consumers:
    kind: ratchet
    description: Do not add legacy database consumers during migration
    command: [python, scripts/inventory_consumers.py, --json]
    parser: {type: json, path: counts.database_consumers}
    mode: must_not_increase
    phases: [stop, ci]

  design_review:
    kind: rubric
    enforcement: advisory
    evaluator: independent_review
    rubric: >
      Fail when the patch introduces an unjustified public API, crosses an
      existing architectural boundary, or omits handling for a named failure
      case. Cite concrete files in every finding.
    include: ["src/**/*.py"]
    phases: [stop, ci]

evaluators:
  independent_review:
    type: openai
    model: YOUR_PINNED_MODEL

See examples/constraintloop.full.yml for all constraint types.

Ratchets store their numeric baseline and evidence SHA-256 in the committed constraintloop-baselines.json; baseline changes are therefore explicit code review events. JSON artifact constraints can map selected dotted paths through evidence so human-readable runs and constraintloop status report useful counts and changes while the complete machine-readable run remains available through --json. The published JSON Schema provides editor validation and autocomplete for every contract field.

The pre-release engineering and open-source checklist is tracked in docs/release-readiness.md. Participation is governed by the Code of Conduct. Maintainer release setup and Trusted Publishing invariants are documented in RELEASE.md. The strict schema is documented in docs/configuration.md, and remote evaluator disclosure and cost controls are documented in docs/provider-privacy.md. OpenAI request-shape, failure, SDK-compatibility, and semantic-corpus checks are documented in docs/openai-evaluation.md. Optional local Codex and Claude Code command evaluators are documented in docs/native-cli-evaluators.md.

OpenAI evaluator setup

Install the optional provider SDK:

uv sync --extra dev --extra openai

For local development, paste the key into the gitignored .constraintloop/secrets.env file:

OPENAI_API_KEY=your-key-here

Process environment variables take precedence over the local file. In CI, use the CI platform's secret store and expose OPENAI_API_KEY; do not create or commit a credential file. ConstraintLoop parses the local file as plain KEY=VALUE data and never evaluates it as shell code. Agent hook writes to this file are denied.

This repository dogfoods an advisory native-agent design rubric. OpenAI and Anthropic remain optional provider integrations. Keep probabilistic gates advisory until their false-positive and false-negative rates are measured.

Lifecycle

1. SessionStart tells the coding agent which required gates exist.
2. The prompt hook records the user's goal as review evidence.
3. Before tool execution, agent attempts to edit the contract or create a waiver are denied.
4. After a main-agent tool execution, change gates run and fresh results are injected. Subagent tool and stop events are ignored because their working tree may be intentionally transient.
5. Before compaction, the completion policy is restated.
6. At Stop / AfterAgent, required stop gates block completion. The agent receives compact evidence and may repair the code a bounded number of times. ConstraintLoop defers this evaluation while background tasks or scheduled wakeups are active. Full retained output remains available through constraintloop debug ID.
7. Advisory failures require either passing fresh evidence or an explicit snapshot-bound explanation; delivery alone never counts as review.
8. Repeated required failure stops autonomous repair and requests a human decision. A trusted human can record a reasoned, snapshot-bound local waiver; hooks deny observed agent waiver commands, any relevant change invalidates it, and CI ignores it. The local CLI cannot authenticate whether its caller is human.
9. constraintloop ci reruns every CI gate without local evidence or waivers.

Every command constraint has a finite timeout (300 seconds by default). On POSIX, timeout cleanup terminates the entire spawned process group so TestContainers, Docker clients, and other descendants cannot keep inherited output pipes open. Commands run from their configured project-contained cwd, with the selected project root prepended to PYTHONPATH.

Evidence is keyed by the constraint definition and the bytes of every file matched by watch, plus the resolved project/worktree, branch, and HEAD. A source change or new HEAD therefore makes old evidence and waivers stale without a mutable invalidation list. Local state lives under the gitignored .constraintloop/state directory; set CONSTRAINTLOOP_CACHE_DIR to override it. Git projects use a separate state subdirectory per branch in each worktree. Use one worktree per concurrent line of work; doctor and explain show the active scope. See worktrees and concurrent sessions.

For stronger machine-local gates, create a gitignored constraintloop.local.yml. ConstraintLoop recursively merges mappings over the repository contract and rejects changes that could weaken committed gates. The authoritative constraintloop ci command ignores this overlay. init and setup add local state, the uninstall tombstone, generated agent hook settings, and overlay names to the selected project's .gitignore, and warn if state is already tracked. Claude uses its dedicated .claude/settings.local.json path. Explicit uninstall records a local tombstone, so a checkout that restores old committed hook wiring does not silently reactivate ConstraintLoop; setup clears the tombstone.

Verdicts and what they mean

Commands

• constraintloop init — generate a reviewable initial contract.
• constraintloop setup --adapter claude|codex|gemini|all [--pre-push] — merge agent hook entries while preserving existing hooks; optionally install an owned Git pre-push hook for push gates.
• constraintloop uninstall --adapter claude|codex|gemini|all — remove only ConstraintLoop hook entries while preserving unrelated settings; pass --pre-push to remove an owned Git hook too.
• constraintloop run --phase change|stop|push — run local gates with fresh caching. Push gates do not honor local waivers.
• constraintloop ci — authoritative, uncached, waiver-free run.
• constraintloop cycle NAME --json — execute one journaled loop transition.
• constraintloop supervise NAME — poll pending evidence under a recoverable single-writer lease and exit whenever repair or termination is required.
• constraintloop loop-prompt NAME --adapter claude|codex|gemini — print the bounded native-agent repair and challenge protocol without launching an agent.
• constraintloop challenge show NAME — inspect saved scenarios and the submission schema.
• constraintloop challenge submit NAME --file PATH — record session-authored discovery or verification for the current request and input snapshot.
• constraintloop status — inspect evidence without executing commands.
• constraintloop explain --phase change|stop|push|ci — show why each constraint runs or is skipped, including matched and changed watch paths, cache state, and dependency chains.
• constraintloop baseline update ID|--all — initialize or strengthen native ratchet baselines; weakening requires the explicit --allow-regression flag.
• constraintloop debug ID — explain evidence freshness, evaluator configuration, executable resolution, and native CLI availability without running an evaluator or consuming model quota.
• constraintloop acknowledge ID --reason "..." — record an explicit snapshot-bound advisory disposition without changing its verdict.
• constraintloop doctor — validate and fingerprint the contract; --deep also checks executables, Python invocations, referenced environment files and variables, worktree environment templates, virtual environments, container runtimes and daemons, ratchet baselines, empty watch globs, and local-state hygiene.
• constraintloop waive ID --reason "..." — human-local, snapshot-bound waiver for fresh non-passing deterministic evidence. Rubrics cannot be waived.
• constraintloop enhance — write a review-only proposal for stronger tooling.
• constraintloop author — write a review-only QA/test-authoring proposal.

enhance and author intentionally do not install dependencies or modify the active contract. Their proposal files make the future self-improvement path auditable.

Documentation

Evaluator command protocol

A command evaluator receives an EvaluationBundle JSON object on stdin and must write exactly one object to stdout:

{
  "verdict": "pass",
  "score": 0.91,
  "rationale": "The patch satisfies the rubric.",
  "findings": []
}

Valid verdicts are pass, fail, and uncertain. Provider errors and malformed responses become uncertain; a required rubric therefore fails closed.

Compatibility boundary

The supported v0.4 surfaces are the CLI and exit codes, configuration schema, evaluator command protocol, native hook responses, and schema-versioned evidence and cycle JSON. Python submodules are internal during initial development and are not covered by semantic-versioning compatibility promises. Migration notes will accompany changes to supported schemas and protocols.

Security model

Hooks are policy automation, not a security sandbox. A sufficiently privileged agent process can bypass local hooks or alter local files. The trusted boundary is a protected, reviewed contract plus an independent CI run. See docs/threat-model.md.

Frequently asked questions

Why not just tell the agent to run tests? Because a prompt is not durable policy. ConstraintLoop records which contract ran, which inputs it covered, and whether the evidence is still fresh.

Why do some constraints run after every action? Put only fast feedback in the change phase. Keep unit checks in stop; put heavyweight integration suites in push and ci.

Can I use Codex or Claude Code instead of an API evaluator? Yes. The native evaluator adapter prefers the active supported CLI and remains read-only.

How do I test failure behavior? Use deterministic commands or fixtures that return known failure, pending, malformed, timeout, or corruption outcomes. Do not spend provider quota merely to manufacture an error.

See the complete FAQ and troubleshooting guide.

License: MIT.