Corrobo

When an AI agent says "done", Corrobo reads the real system and rules VERIFIED, SAY-DO GAP or UNVERIFIED.

An agent's tool call can report success and still not have done the thing. Corrobo wraps the tool, reads the real before/after state through its own read-only connection, and checks it against the effect the action promised. A tool that lies, or silently fails, is caught; one that genuinely worked is proven, with a signed verdict. It never trusts the tool's own response; it reads the world.

pip install corrobo

Proof, not a promise: a live AI receptionist, verified

On 2026-09-24 Corrobo was wired to a production booking agent (Gemini + a live Postgres) and run over eight real conversations. Four callers asked for a slot that was already taken.

Verdicts matched the table in 8 of 8 conversations. It also caught its author twice on the way: a contract that said "done = any booking on this slot" was ruled VERIFIED_VACUOUS because the effect already held before the action, and a claim-reader that counted "that slot has just been booked" as a booking claim was exposed by printing its disagreements. A verifier that flags a lazy contract is the point.

Measured before that on public agent benchmarks (τ-bench retail, AgentDojo, Agent-Diff, Box): on the Box suite 59% of the agent's success claims were false, and neither the tool's return value nor a transcript-reading judge caught them. Reading the database did.

Thirty seconds, no setup

import sqlite3
from corrobo import Corrobo, SQLiteStateSource

con = sqlite3.connect("demo.db")
con.execute("CREATE TABLE IF NOT EXISTS bookings (id INTEGER PRIMARY KEY, slot TEXT UNIQUE, who TEXT)")
con.commit()

def book(slot, who):                      # a tool that LIES: says ok even when the slot was taken
    try:
        con.execute("INSERT INTO bookings (slot, who) VALUES (?, ?)", (slot, who)); con.commit()
    except sqlite3.IntegrityError:
        pass
    return {"ok": True, "message": f"You're all set for {slot}!"}

vd = Corrobo(mode="monitor", state=SQLiteStateSource(path="demo.db"))

for who in ("Alex", "Jordan"):            # second caller gets the same slot
    vd.check("book", {"slot": "Tue 3pm", "who": who}, lambda: book("Tue 3pm", who),
             succeeded=lambda r: r["ok"],
             reads=lambda a: [("bookings", {"slot": a["slot"]})],
             effect=lambda before, after, a: any(r["who"] == a["who"] for r in after["bookings"].values()))
    print(who, "->", vd.last()["status"])

# Alex -> VERIFIED
# Jordan -> SAY-DO GAP        (told "all set"; the table says otherwise)

Start here: the six names you need

The package exports a lot (mining, invariants, attestations, the MCP proxy). Ignore all of it until you need it. Day one is:

help(Corrobo) and help(Corrobo.check) describe every argument.

Do not want to write the contract? Let it be proposed

Point mine_tool at your function with a handful of calls. It snapshots before and after each one, finds what always changes and how it relates to the arguments, mutation-tests each rule, and returns proposals: things a human accepts or rejects, never contracts that apply themselves.

from corrobo import Corrobo, SQLiteStateSource, mine_tool, render_proposals

vd = Corrobo(mode="monitor", state=SQLiteStateSource(path="app.db"))

def upgrade(user_id, plan):
    con.execute("UPDATE users SET plan=? WHERE id=?", (plan, user_id)); con.commit()

props = mine_tool(vd, "upgrade", lambda a: upgrade(**a),
                  [{"user_id": i, "plan": "pro"} for i in range(1, 7)],
                  reads=lambda a: [("users", {"id": a["user_id"]})])
print(render_proposals(props))
# [1] `upgrade` always set `plan` to whatever was passed as `plan` (6 calls) ...   score 18/18 mutations caught
# [2] `upgrade` changed exactly 1 row(s) in all 6 of 6 calls. Should more than 1 be refused?
# [3] `upgrade` only ever touched ['users']. Should touching anything else be refused?

# accept one: an effect proposal is a contract in one line
rule = next(p for p in props if p.kind == "effect").as_effect()
vd.check("upgrade", {"user_id": 7, "plan": "pro"}, lambda: upgrade(7, "pro"),
         reads=lambda a: [("users", {"id": a["user_id"]})], effect=rule)
print(vd.last()["status"])   # VERIFIED

It really runs the tool six times, so use a copy of the data. Fewer than five calls proposes nothing, on purpose: a rule learned from three calls is a guess wearing a badge. Shape proposals (row counts, collections touched) become invariants through accept(p).

from corrobo import Corrobo, RestStateSource

vd = Corrobo(mode="monitor", state=RestStateSource(
    "https://api.github.com",
    resources={"issue": {"path": "/repos/OWNER/REPO/issues/{number}", "pk": "number"}},
    headers={"Authorization": f"Bearer {TOKEN}", "Accept": "application/vnd.github+json"}))

vd.check("rename_issue", {"number": 1, "new_title": "Fixed"},
         lambda: github_patch(1, {"title": "Fixed"}),      # your real tool call
         succeeded=lambda r: r.ok,                           # HTTP 2xx = the tool claims success
         reads=lambda a: [("issue", {"number": a["number"]})],
         effect=lambda before, after, a: after["issue"][a["number"]]["title"] == a["new_title"])

print(vd.last().status)        # VERIFIED  |  SAY-DO GAP  |  UNVERIFIED
print(vd.last().explain())     # a human, secrets-free explanation:
#   Action:    rename_issue  (args: new_title, number)
#   Expected:  the declared effect to hold on issue[number] (read independently, before & after)
#   Observed:  real state changed [issue #1: comments, updated_at], but the promised effect did NOT hold
#   Status:    SAY-DO GAP — the real state did not match the expected effect (say-do gap)

The full stack — from "did it happen?" to "the agent can't fake success"

The say-do-gap check above is layer 1. The proven layers on top:

1. Independent verification — read real state; decide VERIFIED / SAY-DO GAP / UNVERIFIED (above).
2. Cross-system outcomes — one contract can require an effect across multiple systems of record (MultiStateSource + identity correlation). The whole invariant must hold, or it's a gap.
3. Independent verifier, out of process — run the engine as its OWN process with its OWN read-only credentials, contracts, and signing key. The agent sends only {contract_id, args} — it cannot supply the state source, the contract, or the key (VerifierService / VerifierClient).
4. Enforcement gate — an authoritative success is released only on an independent VERIFIED (OutcomeGate).
5. Signed success attestations — on a genuine VERIFIED the verifier mints a short-lived Ed25519 attestation bound to the operation; any consumer validates it with the public key (no secret). It can't be forged, modified, replayed, retargeted, or outlived (OutcomeGate.accept, verify_attestation).
6. Consumption-time re-verification — optionally re-check the effect at the moment of release to shrink the TOCTOU window (accept(..., reverify=True)).

Validated end to end on real GitHub REST, PostgreSQL, Jira Cloud, MCP (stdio), and a live receptionist agent (Gemini + Postgres), plus adversarial red-teams of the verifier trust boundary and the attestation.

Install

pip install corrobo                 # core — pure standard library, no dependencies
pip install "corrobo[postgres]"     # + PostgresStateSource   (psycopg 3)
pip install "corrobo[mcp]"          # + drive a real MCP server over stdio (mcp)
pip install "corrobo[attest]"       # + mint/verify Ed25519 success attestations (cryptography)

from corrobo import Corrobo, CorroboSayDoGap, CorroboUncertain, CorroboBlocked

Python 3.9+. The core imports nothing outside the standard library; the extras are imported lazily, so installing the core never pulls a driver you do not use.

How it works — two things you declare

Corrobo snapshots the footprint, runs the tool, snapshots again, and evaluates the effect against real state — not the tool's return value.

The verdicts

An honest tool failure is distinguished from a lie: pass succeeded=lambda r: ... (e.g. an HTTP 2xx check, or an MCP is_error flag) and a tool that reports failure becomes UNVERIFIED (action-failed), never a say-do gap.

Monitor vs enforce

• mode="monitor" (safe default) — run the tool, verify, record evidence, never interfere.
• mode="enforce" — a pre-check can block the action; a say-do gap raises CorroboSayDoGap (stop downstream); an unestablished result raises CorroboUncertain (fail-safe). An honest tool failure does not raise (its own failure result is returned).

Adapters (StateSource) — one engine, any backend

The engine reads state through a StateSource; only the adapter changes.

from corrobo import RestStateSource        # any REST/HTTP API (scoped GETs, Retry-After, retries)
from corrobo import PostgresStateSource     # real Postgres (consistent READ ONLY snapshot, PK auto-detect)
from corrobo import SQLiteStateSource       # a SQLite file or connection
from corrobo import CorroboMCPProxy         # sit in front of an existing MCP server; verify each tool call

Corrobo has been validated end-to-end against real GitHub REST, real PostgreSQL, and the real MCP SDK over stdio — the same core engine, no per-system verification logic.

Evidence

Every verification appends to a tamper-evident, hash-chained log. It stores only hashes, the argument keys, the scoped footprint (column names), and which columns changed — never raw values, never secrets, never the connection string.

vd = Corrobo(mode="monitor", state=..., evidence_path="corrobo.jsonl")
vd.evidence_log.verify()     # True — the chain is intact

Current guarantees (validated on real systems)

• VERIFIED is emitted only when the promised effect is actually present — decided by an independent read, never the tool's response body. Multi-condition effects and idempotent re-application verify correctly.
• Real say-do gaps are caught: wrong record, wrong field, partial effect, or a lying success response.
• Honest tool failures (401/403/404/HTTP errors, DB errors, tool-reported failure) → action-failed / UNVERIFIED, never a say-do gap.
• Fail-safe under unavailability: a read 404, a timeout, a dropped connection, or a resource that vanished → UNVERIFIED, never a false VERIFIED.
• Concurrency: a concurrent change to the effect-relevant state → UNVERIFIED / CorroboUncertain; unrelated concurrent churn (an extra comment, a bumped updated_at/LastModifiedDate, an unrelated row/column) does not cause a false failure.
• Direct-record reads are treated as strongly consistent; genuinely lagging reads are handled by a bounded settle window (settle_retries); rate limits honor a bounded Retry-After.

Limitations (honest)

• Concurrency detection is verdict-based, not field-traced: a concurrent write that changes a footprint row but does not flip the effect's verdict (e.g. a loose/inequality effect) is not flagged. Exact-value effects — the common case — have no blind spot.
• A read 404 is ambiguous (truly absent vs. no access), so a genuinely-absent target is reported as UNVERIFIED (fail-safe), not a definitive say-do gap.
• Corrobo is a verifier, not a lock manager or a rollback engine: it detects and (in enforce) blocks; it does not undo the tool's transaction.
• Postgres contracts must declare a reads footprint (whole-DB reads are refused for production safety), and every table read needs a primary key.
• Expiring-token refresh for REST auth is not built in yet (static headers today).

What ships in the package

corrobo/ only: the engine, the REST / Postgres / SQLite adapters, the MCP proxy, contract mining, invariants, frame conditions, drift detection, the retry and ambiguity gates, the out-of-process verifier, the enforcement gate and Ed25519 attestations. About 7,700 lines, with roughly 11,000 lines of tests behind them (83 files: engine tests plus live validation campaigns against GitHub, Postgres, Jira and MCP).

Status

Beta. The engine has had three adversarial review passes, a full CodeRabbit sweep, a public-benchmark campaign and one live production workload. The verdict logic is stable; what is still moving is deployment hardening (key distribution, a shared replay store, TLS, multi-instance) and the ergonomics people ask for. If you run an agent that writes to a real system, the most useful thing you can do is run Corrobo in monitor mode beside it for a week and tell the author what it found.

Feedback

Bugs, false verdicts, and "it should have caught this" reports are the most valuable thing you can send. Email melvinis@gmail.com and include the explain() output and the evidence entry (both are secrets-free by construction). Commercial licensing goes to the same address.

License

PolyForm Noncommercial 1.0.0. Free to use, copy, modify and share for noncommercial purposes: personal projects, research, teaching, and noncommercial organizations. Commercial use needs a separate license: contact the author. See LICENSE for the full terms.