AgentLoop — provable correctness for autonomous coding agents.

A harness-agnostic verify-gate with a held-out oracle, so the agent can't fake 'done'.

agentloop "build a JSON linter" --verify "bash verify.sh"

Free. MIT licensed. Bring your own agent harness (OpenCode, Claude Code, Aider, Codex, Goose) — AgentLoop adds the verify-gate and held-out oracle.

When to use AgentLoop

AgentLoop shines where task correctness can be objectively verified with a held-out oracle or deterministic test suite:

1. Program Synthesis — CLI tools, parsers, compilers, validators, data converters.
2. Behavior-Preserving Refactoring — Large-scale refactors guarded by golden-file or property-based test capture.
3. Bug Fixes — Issues with clear, reproducible failing test cases.

When NOT to use AgentLoop

• Greenfield UI / web design exploration with no fixed visual contract.
• Exploratory prototyping or open-ended creative brainstorming.
• Anything without a definable, automated "correct" state.

Benchmarks — Proven Lift on SWE-bench-Verified

AgentLoop provides +16% to +22% lift over raw agent harnesses on SWE-bench-Verified by preventing false-greens and driving continuous retry until held-out verification passes.

See bench/README.md for full reproducible benchmark methodology and raw JSONL logs.

You already know the pain.

Coding agents are famous for stopping halfway and claiming they're done.

OpenCode issue #24685. The "stopped halfway" essays. You've been there:

• The agent writes 80% of the code, then quits. Your loop thinks it won.
• You bake test cases into your check script — now the agent overfits, passing those exact cases but failing on everything else. (This is the SWE-bench false-green bug, and it's real.)
• Your API bill keeps climbing while nothing ships.

You've probably thought: "I'll just write a 20-line bash loop."

That doesn't work either. A naive loop can't tell the difference between "the agent said DONE" and "the agent actually solved the problem." You need a verification oracle — a correctness gate the agent can't fake, edit, or overfit to.

That's what AgentLoop is.

The verification oracle is the product.

AgentLoop is not a coding agent and not another BYOK wrapper. It's a thin, harness-agnostic layer that wraps the agent you already use:

   goal + feedback ──► your agent edits the sandbox ──► verification oracle
          ▲                                                   │
          └────────────── (fail) ◄───────────────────────────┘
                         (pass) ──► DONE  (and a sealed oracle agrees)

Three things it adds that people keep rebuilding by hand:

• Continuity — loops until the goal is proven correct, not "please clarify"
• The verification oracle — a correctness gate, not "it runs and prints"
• Safety — your API key never reaches the agent; work is git-checkpointed and crash-resumable

"Wait — a naive loop can't tell if the work is correct."
Correct. That's why AgentLoop exists.

The held-out oracle (your moat against overfitting)

A plain verifier can still be gamed if the agent reverse-engineers the test cases. AgentLoop's sealed, held-out grading defeats that:

# 1) Auto-generate fresh test inputs from a reference program
python oracle.py gen --reference "python ref.py" --n 200 --out cases.txt --seed 42

# 2) Record the reference's behavior — split into visible + held-out (sealed)
python oracle.py record \
  --reference "python ref.py" --inputs cases.txt --visible 3 \
  --out .agentloop/oracle_sealed/oracle.json --seal "$ORACLE_SEAL"

# 3) The verifier grades the candidate against ALL inputs
#    The candidate only PASSES if it's correct on inputs it has NEVER seen
python oracle.py grade \
  --candidate "python sandbox/solution.py" \
  --oracle .agentloop/oracle_sealed/oracle.json --seal "$ORACLE_SEAL"

A wrong --seal makes grading report TAMPERED. The held-out file lives outside the sandbox — the agent can't read it, can't overfit to it. This is the feature that makes AgentLoop worth using over a hand-rolled loop.

Install in 5 seconds.

From PyPI (recommended):

pip install agentloop-cli
agentloop --init
./launch.sh

Or from source:

git clone https://github.com/instax-dutta/agentloop.git && cd agentloop
pip install -e "."
agentloop --init
./launch.sh

That's it. No API key. No .env to copy. No config to edit.

agentloop --init creates everything you need:

• goal.txt — a hello-world task
• verify.sh — a working verifier, ready to run
• .env — zero-config defaults

Need Windows?

.\launch.ps1       # start the loop (background job)
.\stop.ps1         # stop gracefully

Try a real example instead:

agentloop --init --example tax-demo
# Seeds: goal.txt + verify.sh from the tax-demo example

Or jump straight in:

agentloop "build a JSON linter" --verify "bash verify.sh"

Preview your setup before running:

agentloop --dry-run
# Shows: mode, agent command, verify command, goal, limits, version — no loop starts

Run multiple tasks at once — in parallel:

agentloop --run plan.md            # parallel, min(4, #tasks) workers
agentloop --run plan.md --workers 8

Parses any markdown plan — checklists, bullets, headings — and spawns one loop per task in parallel, each with its own sandbox (sandbox/task-N/) and namespaced state/log files. Add (after: #N) or (depends on: #N) to a task to express dependencies — AgentLoop builds a DAG, runs ready tasks in parallel, and skips downstream tasks when a dependency fails.

Parallel-mode verifiers: each worker's sandbox is sandbox/task-N/. Write your verify.sh against $AGENTLOOP_SANDBOX (set for every worker) so the oracle grades the right sandbox — the bundled examples do this automatically.

Check your version:

agentloop --version
# agentloop 0.5.0.dev0

It survives crashes.

Your laptop dies mid-run. Your SSH session drops. Your CI runner gets recycled.

AgentLoop resumes exactly where it stopped.

# Same command, same goal:
agentloop --verify "bash verify.sh"
# Output:
# resuming from iter 2 (cost so far: $0.10)

Every iteration is written atomically to agentloop.state.json. A crash leaves zero torn state. The wall clock tracks from the original start — not the resume time — so timeouts are fair.

Don't let a runaway API bill surprise you.

Set an estimated cost guard (CLI mode) / hard cap (direct mode with token-counted models) in dollars. If the agent exceeds it, the loop stops with status=over-budget:

MAX_COST_USD=5 agentloop "my task" --verify "bash verify.sh"

Tracks running cost in agentloop.state.json. In CLI mode, set ESTIMATED_COST_PER_ITER ($0.10 default). In direct mode, actual token counts are used automatically.

Logs rotate automatically.

Set LOG_MAX_MB in .env (default 10 MB) with 3 backup files via Python's RotatingFileHandler. No more gigabyte log files.

Know exactly what's happening.

agentloop --status       # terminal display: status, iters, elapsed, cost, PID
agentloop --serve        # web UI at http://localhost:8080 (auto-refresh 5s)
agentloop --serve --port 9090   # custom port

Get notified when it finishes.

Send terminal-state summaries wherever you work:

# Telegram
NOTIFY_TELEGRAM_BOT_TOKEN=123456:ABC-DEF
NOTIFY_TELEGRAM_CHAT_ID=-1001234567890

# Discord (webhook URL)
NOTIFY_DISCORD_WEBHOOK_URL=https://discord.com/api/webhooks/...

# Slack (webhook URL)
NOTIFY_SLACK_WEBHOOK_URL=https://hooks.slack.com/services/...

# Or any shell command
NOTIFY_CMD='curl -s -X POST https://hooks.example.com -d "{kind}: {msg}"'

Supports: completed, blocked, stopped, timeout, over-budget — each routed through your preferred channel.

Works with whatever agent you use.

No preset matching yours? Set AGENT_CMD to any command. The prompt is injected via $AGENTLOOP_PROMPT.

AGENT_CMD='my-agent run "$AGENTLOOP_PROMPT"'

If no preset is set, AgentLoop auto-detects an installed CLI with a version check — and warns you if it finds a broken binary.

Hidden strength: Because AgentLoop re-invokes the harness every iteration, a harness that "stops halfway" just becomes one failed iteration. The loop absorbs it and keeps going.

What makes AgentLoop different

Lots of tools loop a coding agent until it reports "done". Almost none can tell the difference between "the agent said done" and "the work is actually correct".

AgentLoop defines a new category: the verify-gate wrapper. Three things, in order of importance:

1. A held-out verification oracle. The agent is graded on inputs it has never seen, recorded from a trusted reference and sealed against tampering. This is the only mechanism that defeats the SWE-bench false-green problem — an agent that passes the exact test cases you baked in, then fails everything else.
2. A loop that only stops on proof. No DONE signal is trusted until VERIFY_CMD exits 0. Failures are fed back to the agent with the verifier's actual output, so every retry is informed.
3. Trustworthy plumbing. Crash-safe atomic state, real token-counted cost tracking, per-iteration git checkpoints, secret-scrubbed environments, and optional container isolation (--docker / --podman).

If your agent harness already edits code well, the only missing piece is knowing when it's actually done. That's the piece AgentLoop supplies.

What's in the box.

Real examples. Real verifiers.

The examples/ directory ships 10 working verifier samples, each teaching a different oracle pattern:

Each has a goal.txt (the task) and verify.sh (the oracle). Run ./verify.sh to see how the oracle works without the loop. Browse the full gallery — with runtime/cost estimates and how to pick a pattern — in docs/EXAMPLES.md.

Built a verifier worth sharing? Add it to the community-verifiers gallery — copy-paste-able verify.sh patterns with a merge checklist, so your oracle pattern becomes a template for everyone else.

Exit codes (for CI / scripting).

The fine print.

• The wrapped agent runs with cwd=sandbox. For hard isolation, use --docker or --podman (see ISOLATION.md).
• Without VERIFY_CMD, the loop falls back on a DONE/BLOCKED signal from the agent — use a real verifier.
• Each CLI iteration is a fresh agent invocation. Continuity is maintained by feeding the goal + last failure back.
• Keep the held-out case file outside the sandbox. The seal is a tamper signal, not absolute security.
• API keys and ORACLE_SEAL are stripped from the agent environment. Never put secrets inside the sandbox.

Tests (deterministic, no LLM required)

python3 test_oracle.py   # verification gate + held-out oracle
python3 test_loop.py     # full loop with mock agent + cost cap + status

Both pass on every commit. CI runs them on Linux (Python 3.10/3.12/3.13).

Pre-commit hooks (for contributors)

pip install -e '.[dev]'
pre-commit install

End-to-End Testing (verified from PyPI)

All features work from a fresh pip install agentloop-cli in a clean virtual environment.

Results: All 13 E2E checks pass. The held-out oracle correctly passes correct code (score 1.00), fails broken code (score 0.05), and detects tampering when the seal is wrong.

# Run the E2E suite yourself:
pip install agentloop-cli
agentloop --init --example tax-demo
agentloop --dry-run
agentloop-oracle gen --reference 'python3 -c "import sys; print(int(sys.stdin.read().strip()) * 2)"' --n 20 --out cases.txt --seed 42
agentloop-oracle record --reference 'python3 -c "import sys; print(int(sys.stdin.read().strip()) * 2)"' --inputs cases.txt --visible 3 --out oracle.json --seal test-seal
agentloop-oracle grade --candidate 'python3 -c "import sys; print(int(sys.stdin.read().strip()) * 2)"' --oracle oracle.json --seal test-seal

AgentLoop is 0.5.0-dev. MIT licensed. Zero runtime dependencies. One purpose: make your agent actually finish.

agentloop "build a JSON linter" --verify "bash verify.sh"

Report an issue · Contributing guide · Changelog