Corrobo

When an AI agent says "done", Corrobo reads the real system and rules VERIFIED, SAY-DO GAP or UNVERIFIED.

An outcome verification layer for AI agents. The question it answers is not "did the tool return success?" but "did the promised outcome actually happen?" Let the agent call the tool exactly as it does today; Corrobo then independently asks the system of record whether the effect the call promised is really there. It never trusts the tool's own response. It reads the world.

To be precise about the word: Corrobo verifies observable effects. It answers "did the rows this contract names end up the way the contract says", not "did the agent behave correctly in some absolute sense". The contract is yours; Corrobo holds the agent to it.

Free for noncommercial use (PolyForm Noncommercial 1.0.0). Commercial use needs a license: see License.

The problem, in one table

Row three is the failure nobody's logs show. The constraint held, the tool said no, the log said DENIED, and the caller was still told "all set". Every layer below the reply did its job; none of them tests the sentence. Corrobo is the line that compares the sentence to the table.

See it in thirty seconds

No key, no network, nothing to configure:

pip install corrobo
corrobo demo

A receptionist agent, three callers, one booking table. For each caller it prints what your stack saw and what the caller heard:

3) a taken slot, the agent says it is booked anyway
   constraint:  UNIQUE(date, time)  -> held: insert refused
   tool said:   unavailable
   log said:    TOOL -> book_appointment DENIED (taken) date='2026-10-05' time='3:00 PM'
   agent said:  "You're all set! See you on October 5th at 3:00 PM."
   table says:  3:00 PM -> someone else
   CORROBO:     SAY-DO GAP  - the real state did not match the expected effect

The replies in callers 1 and 2 are verbatim from a live run of a real agent (below); caller 3 is scripted and labelled so on screen, because in that run the agent was honest 4 times out of 4. It is the conversation Corrobo exists for on the day the agent is not.

The whole idea, in code

Here is the entire idea: let the agent call the tool normally, then independently ask the database whether the promised outcome actually happened.

import sqlite3
from corrobo import Corrobo, SQLiteStateSource

con = sqlite3.connect("demo.db")
con.execute("CREATE TABLE IF NOT EXISTS bookings (id INTEGER PRIMARY KEY, slot TEXT UNIQUE, who TEXT)")
con.commit()

def book(slot, who):                      # a tool that LIES: says ok even when the slot was taken
    try:
        con.execute("INSERT INTO bookings (slot, who) VALUES (?, ?)", (slot, who)); con.commit()
    except sqlite3.IntegrityError:
        pass
    return {"ok": True, "message": f"You're all set for {slot}!"}

vd = Corrobo(mode="monitor", state=SQLiteStateSource(path="demo.db"))

for who in ("Alex", "Jordan"):            # second caller gets the same slot
    vd.check("book", {"slot": "Tue 3pm", "who": who}, lambda: book("Tue 3pm", who),
             succeeded=lambda r: r["ok"],                                   # what the tool CLAIMED
             reads=lambda a: [("bookings", {"slot": a["slot"]})],           # the rows this call touches
             effect=lambda before, after, a: any(r["who"] == a["who"] for r in after["bookings"].values()))
    print(who, "->", vd.last()["status"])

# Alex -> VERIFIED
# Jordan -> SAY-DO GAP        (told "all set"; the table says otherwise)

Two things you declare, and only two: reads(args), the exact rows the action touches, and effect(before, after, args), what "done" means on the real state. Corrobo snapshots the footprint, runs the tool, snapshots again, and judges the effect against real state, never the return value.

Proven on real agents

A live receptionist. On 2026-09-24 Corrobo was wired to a production booking agent (Gemini + a live Postgres) and run over eight real conversations, four of them asking for a slot that was already taken. Verdicts matched the table in 8 of 8; the table at the top of this page is from that run. It also caught its author twice on the way: a contract that said "done = any booking on this slot" was ruled VERIFIED_VACUOUS because the effect already held before the action, and a claim-reader that counted "that slot has just been booked" as a booking claim was exposed by printing its disagreements. A verifier that flags a lazy contract is the point.

Public benchmarks. The same check, with gemini-2.5-flash as the agent, over third-party agent benchmarks. A claim is the agent's final message asserting the task is complete; a claim is false when the benchmark's own state assertions fail. Corrobo's effect predicate is written independently of those assertions, so the two cannot agree by construction.

What the Box result taught: every one of the ten false claims changed real state. The agent acts, rows move, and the result is not what was asked: wrong file, wrong destination, half the job. So the cheap check "if it claims success, something must have moved" catches 0 of 10, and neither the tool's return value nor a transcript-reading LLM judge caught them either. A per-tool declared effect checked against the database caught 9 of 10. The retail number is low because most of what first looked like a gap (13%) was the benchmark's own user simulator renegotiating the task mid-conversation, which the transcripts show and the re-score excludes.

Limits of the measurement: one model, three suites here (the smaller AgentDojo suites had one to five claims each, too few to quote), and the tasks and grading are the benchmarks' own. Method, per-task records and full transcripts are committed under benchmarks/ in the repository (private during beta; email the author for access).

Real systems the engine has been validated against: GitHub REST, PostgreSQL, Jira Cloud, MCP over stdio, and the receptionist above.

Install

pip install corrobo                 # core: pure standard library, no dependencies
pip install "corrobo[postgres]"     # + PostgresStateSource   (psycopg 3)
pip install "corrobo[mcp]"          # + drive a real MCP server over stdio (mcp)
pip install "corrobo[attest]"       # + mint/verify Ed25519 success attestations (cryptography)

Python 3.10+ (tested on 3.10, 3.11, 3.12). The core imports nothing outside the standard library; the extras are imported lazily, so installing the core never pulls a driver you do not use.

How it works

agent calls tool
      |
      v
 read real state  (only the rows `reads` names, through Corrobo's own read-only connection)
      |
      v
 run the tool     (untouched; its return value is recorded, not trusted)
      |
      v
 read real state again
      |
      v
 evaluate `effect(before, after, args)`  +  the tool's own claim (`succeeded`)
      |
      v
 VERIFIED  |  SAY-DO GAP  |  UNVERIFIED  |  BLOCKED

An honest tool failure is distinguished from a lie: pass succeeded=lambda r: ... (an HTTP 2xx check, an MCP is_error flag) and a tool that reports failure becomes UNVERIFIED (action-failed), never a say-do gap.

Monitor vs enforce. mode="monitor" (the safe default) runs the tool, verifies, records evidence and never interferes. mode="enforce" lets a pre-check block the action, raises CorroboSayDoGap on a gap so the false "all set" never reaches the user, and raises CorroboUncertain when the outcome could not be established. An honest tool failure does not raise; its own failure result is returned.

Start here: the six names you need

The package exports a lot (mining, invariants, attestations, the MCP proxy). Ignore all of it until you need it. Day one is:

help(Corrobo) and help(Corrobo.check) describe every argument.

Integrations

Already have tools? Wrap them where they are

Nothing needs rewriting. Each adapter takes the tool you have and returns the same tool, verified; name, description, schema and return value are untouched. None of them import a framework.

from corrobo import wrap_function, wrap_langchain_tool, wrap_openai_function_tool, wrap_tools

BOOKING = dict(reads=lambda a: [("bookings", {"slot": a["slot"]})],
               effect=lambda before, after, a: any(r["who"] == a["who"] for r in after["bookings"].values()))

book  = wrap_function(vd, book, **BOOKING)                      # plain function, sync or async
tools = [wrap_langchain_tool(vd, book_tool, **BOOKING), search]  # LangChain @tool / StructuredTool
agent = Agent(tools=[wrap_openai_function_tool(vd, book, **BOOKING)])   # OpenAI Agents SDK
tools = wrap_tools(vd, tools, {"book": BOOKING, "cancel": CANCEL})      # a whole toolset, by name

wrap_tools raises if a contract names a tool that is not in the list: a misspelled contract that quietly verified nothing is the failure this library exists to catch. Async tools stay async. Details and per-framework notes: docs/INTEGRATIONS.md.

Where the truth lives: one engine, any backend

The engine reads state through a StateSource; only the adapter changes.

from corrobo import SQLiteStateSource       # a SQLite file or connection
from corrobo import PostgresStateSource     # real Postgres (consistent READ ONLY snapshot, PK auto-detect)
from corrobo import RestStateSource         # any REST/HTTP API (scoped GETs, Retry-After, retries)
from corrobo import MultiStateSource        # several systems of record in one contract
from corrobo import CorroboMCPProxy         # sit in front of an existing MCP server; verify each tool call

Any REST API, for example GitHub:

from corrobo import Corrobo, RestStateSource

vd = Corrobo(mode="monitor", state=RestStateSource(
    "https://api.github.com",
    resources={"issue": {"path": "/repos/OWNER/REPO/issues/{number}", "pk": "number"}},
    headers={"Authorization": f"Bearer {TOKEN}", "Accept": "application/vnd.github+json"}))

vd.check("rename_issue", {"number": 1, "new_title": "Fixed"},
         lambda: github_patch(1, {"title": "Fixed"}),      # your real tool call
         succeeded=lambda r: r.ok,                           # HTTP 2xx = the tool claims success
         reads=lambda a: [("issue", {"number": a["number"]})],
         effect=lambda before, after, a: after["issue"][a["number"]]["title"] == a["new_title"])

print(vd.last().status)        # VERIFIED  |  SAY-DO GAP  |  UNVERIFIED
print(vd.last().explain())     # a human, secrets-free explanation:
#   Action:    rename_issue  (args: new_title, number)
#   Expected:  the declared effect to hold on issue[number] (read independently, before & after)
#   Observed:  real state changed [issue #1: comments, updated_at], but the promised effect did NOT hold
#   Status:    SAY-DO GAP — the real state did not match the expected effect (say-do gap)

Your own system is a small class with one method, "read these things"; a conformance kit checks it returns the right shape (docs/STATESOURCE.md).

Do not want to write the contract? Let it be proposed

Point mine_tool at your function with a handful of calls. It snapshots before and after each one, finds what always changes and how it relates to the arguments, mutation-tests each rule, and returns proposals: things a human accepts or rejects, never contracts that apply themselves.

from corrobo import Corrobo, SQLiteStateSource, mine_tool, render_proposals

vd = Corrobo(mode="monitor", state=SQLiteStateSource(path="app.db"))

def upgrade(user_id, plan):
    con.execute("UPDATE users SET plan=? WHERE id=?", (plan, user_id)); con.commit()

props = mine_tool(vd, "upgrade", lambda a: upgrade(**a),
                  [{"user_id": i, "plan": "pro"} for i in range(1, 7)],
                  reads=lambda a: [("users", {"id": a["user_id"]})])
print(render_proposals(props))
# [1] `upgrade` always set `plan` to whatever was passed as `plan` (6 calls) ...   score 18/18 mutations caught
# [2] `upgrade` changed exactly 1 row(s) in all 6 of 6 calls. Should more than 1 be refused?
# [3] `upgrade` only ever touched ['users']. Should touching anything else be refused?

# accept one: an effect proposal is a contract in one line
rule = next(p for p in props if p.kind == "effect").as_effect()
vd.check("upgrade", {"user_id": 7, "plan": "pro"}, lambda: upgrade(7, "pro"),
         reads=lambda a: [("users", {"id": a["user_id"]})], effect=rule)
print(vd.last()["status"])   # VERIFIED

It really runs the tool six times, so use a copy of the data. Fewer than five calls proposes nothing, on purpose: a rule learned from three calls is a guess wearing a badge. Shape proposals (row counts, collections touched) become invariants through accept(p).

Beyond one check: an independent trust boundary around the agent

The say-do-gap check is layer 1. What makes Corrobo more than "a library that diffs database state" is that the agent can be kept out of the loop that judges it:

1. Independent verification: read real state; decide VERIFIED / SAY-DO GAP / UNVERIFIED (above).
2. Cross-system outcomes: one contract can require an effect across multiple systems of record (MultiStateSource + identity correlation). The whole invariant must hold, or it is a gap.
3. Independent verifier, out of process: run the engine as its OWN process with its OWN read-only credentials, contracts, and signing key. The agent sends only {contract_id, args}. It cannot supply the state source, the contract, or the key (VerifierService / VerifierClient).
4. Enforcement gate: an authoritative success is released only on an independent VERIFIED (OutcomeGate).
5. Signed success attestations: on a genuine VERIFIED the verifier mints a short-lived Ed25519 attestation bound to the operation; any consumer validates it with the public key, no secret. It cannot be forged, modified, replayed, retargeted, or outlived (OutcomeGate.accept, verify_attestation).
6. Consumption-time re-verification: optionally re-check the effect at the moment of release to shrink the TOCTOU window (accept(..., reverify=True)).

Also in the package, each with its own doc: invariants and frame conditions (what must NOT have changed: blast radius), an ambiguity gate (refuse to act when the target is a guess), confirmation binding (verify against what a person approved), a retry gate, per-step chain verification, and drift detection against a baseline.

Evidence

Every verification appends to a tamper-evident, hash-chained log. It stores only hashes, the argument keys, the scoped footprint (column names), and which columns changed: never raw values, never secrets, never the connection string.

vd = Corrobo(mode="monitor", state=..., evidence_path="corrobo.jsonl")
vd.evidence_log.verify()     # True: the chain is intact

Current guarantees (validated on real systems)

• VERIFIED is emitted only when the promised effect is actually present, decided by an independent read, never the tool's response body. Multi-condition effects and idempotent re-application verify correctly.
• Real say-do gaps are caught: wrong record, wrong field, partial effect, or a lying success response.
• Honest tool failures (401/403/404/HTTP errors, DB errors, tool-reported failure) become action-failed / UNVERIFIED, never a say-do gap.
• Fail-safe under unavailability: a read 404, a timeout, a dropped connection, or a resource that vanished gives UNVERIFIED, never a false VERIFIED.
• Concurrency: a concurrent change to the effect-relevant state gives UNVERIFIED / CorroboUncertain; unrelated concurrent churn (an extra comment, a bumped updated_at, an unrelated row or column) does not cause a false failure.
• Direct-record reads are treated as strongly consistent; genuinely lagging reads are handled by a bounded settle window (settle_retries); rate limits honor a bounded Retry-After.

Limitations (honest)

• Corrobo verifies "did the tool do what this call promised", not "was this call what the user asked for". An agent that confidently acts on the wrong order id, spoken by the customer and acted on faithfully, passes. That needs a check before execution (the ambiguity gate and confirmation binding are the start of one), not a better check after it.
• Concurrency detection is verdict-based, not field-traced: a concurrent write that changes a footprint row but does not flip the effect's verdict (a loose or inequality effect) is not flagged. Exact-value effects, the common case, have no blind spot.
• A read 404 is ambiguous (truly absent vs. no access), so a genuinely absent target is reported as UNVERIFIED (fail-safe), not a definitive say-do gap.
• Corrobo is a verifier, not a lock manager or a rollback engine: it detects and, in enforce, blocks; it does not undo the tool's transaction.
• Postgres contracts must declare a reads footprint (whole-DB reads are refused for production safety), and every table read needs a primary key.
• Expiring-token refresh for REST auth is not built in yet (static headers today).

What ships in the package

corrobo/ only: the engine, the SQLite / Postgres / REST adapters, the framework adapters, the MCP proxy, contract mining, invariants, frame conditions, drift detection, the retry and ambiguity gates, the out-of-process verifier, the enforcement gate and Ed25519 attestations. About 8,000 lines, with roughly 11,500 lines of tests behind them (87 files: engine tests plus live validation campaigns against GitHub, Postgres, Jira, MCP, and the real LangChain and OpenAI Agents packages).

Status

Beta. The engine has had three adversarial review passes, a full CodeRabbit sweep, a public-benchmark campaign and one live production workload. The verdict logic is stable; what is still moving is deployment hardening (key distribution, a shared replay store, TLS, multi-instance) and the ergonomics people ask for. If you run an agent that writes to a real system, the most useful thing you can do is run Corrobo in monitor mode beside it for a week and tell the author what it found.

Feedback

Bugs, false verdicts, and "it should have caught this" reports are the most valuable thing you can send. Email fahadrahiman10@gmail.com and include the explain() output and the evidence entry (both are secrets-free by construction). Commercial licensing goes to the same address.

License

PolyForm Noncommercial 1.0.0. Free for noncommercial use: personal projects, research, teaching, evaluation, and noncommercial organizations. Commercial use needs a separate license: contact the author. See LICENSE for the full terms.