riskgate

Certify, reject, probe or ask: calibrated, consequence-aware checks for the actions of LLM agents.

riskgate sits between an AI agent and its tools. Before a consequential action runs (scaling a service, issuing a refund, changing a reactor setpoint) it predicts what the action would do under every explanation of the current situation that the evidence cannot rule out. It lets the action through only if the worst plausible outcome clears a safety threshold that has been calibrated on labelled outcomes, so that unsafe actions get through at most a chosen fraction α of the time, whatever the quality of your simulator.

agent proposes ──► riskgate ──► CERTIFY  → the tool runs
                     │     ├──► REJECT   → "unsafe under every explanation; binding constraint: p95 latency" → agent re-plans
                     │     ├──► PROBE    → "first run pg_stat_statements: it tells the two likely causes apart"
                     │     └──► ASK      → a human decides, with the gate's explanation
                     └── no LLM on this path · fail-closed · every decision is logged and fingerprinted

Status: 1.0.0rc5 (release candidate), Apache-2.0. Everything below runs locally and is tested. The three example domains are simulated with scripted (LLM-like) agents. In addition, benchmarks/ replays public logs of real LLM agents offline: τ²-bench with eight 2026 frontier models, AgentDojo prompt-injection runs of 28 model/defence pipelines, and ITBench. It also covers public telemetry and plant data (RCAEval, Tennessee Eastman). The calibrated bound held for all 40 agent configurations replayed; see Results on real agents. The paper is under submission (see Project status).

In one minute (no jargon)

Imagine a new assistant on a night shift. Alarms go off, and the assistant wants to press a button.

• A careless assistant presses the button that fits its first guess.
• A rule-following assistant checks a list of forbidden buttons. The list knows nothing about tonight.
• A careful assistant asks: "What could be going on? What would this button do in each case? Is it safe in all of them?" If yes, it presses. If it is clearly a bad idea, it says why. If it cannot tell, it first does a harmless check (looks at a second gauge) or calls the supervisor.

riskgate is the careful assistant, built as software. And it keeps score: it has been tested on hundreds of past or simulated situations, so it can say "when I say 'safe', I am wrong at most 5% of the time", and it says "I don't know" when a situation looks unlike anything it was tested on.

What it is (for engineers)

Numpy is the only runtime dependency. The certify path never calls an LLM, and anything unexpected (exceptions, timeouts, missing or stale calibration, a failing audit sink) produces ASK, never CERTIFY.

Install

pip install --pre riskgate              # release candidate; extras: yaml, otel, mcp, langgraph, adk, openai-agents
pip install -e ".[dev]"                 # from a clone

Python ≥ 3.10. PyPI · GitHub (a ⭐ helps others find it).

Thirty-second example

import riskgate as rg

class TankModel:                                   # consequence model: what would the action do?
    version = "tank-model-v1"                      # bump on every change: old calibrations become VOID
    def predict(self, action, hyp, ctx):
        return {"level": ctx["level"] + action.args["v"] - hyp.params["leak"]}

class LeakDiagnoser:                               # hypotheses: what could be going on? (lower score = better fit)
    version = "leak-diag-v1"
    def hypotheses(self, ctx):
        return [rg.Hypothesis(f"leak={l}", family=f"leak{l}", params={"leak": l},
                              score=abs(ctx["flow"] - l) / 8) for l in (0, 15, 30)]

gate = rg.Gate(TankModel(), rg.Spec(rg.Envelope("level", lo=10, hi=100, scale=10)), LeakDiagnoser(), alpha=0.05,
               doer_id="tank-agent:v1")            # the agent whose proposals you calibrate on: swapping it voids the calibration
gate.calibrate(labelled_episodes)                  # sets tau and lambda-hat; status -> VALID

d = gate.check(rg.Action("add", {"v": 40}), {"level": 80, "flow": 15.0})
print(d.verdict, d.reason)                         # ASK: cannot certify: best robust margin -2.000 < threshold -1.763 ...

@rg.guard(gate, context=read_tank_state)           # or guard any tool function your agent framework calls
def add(v: float) -> str: ...

The complete, runnable version (with episode generation) is examples/quickstart.py. The production-like lifecycle (calibrate → sign → deploy → outcomes → drift → recalibrate) is examples/end_to_end_sre.py, explained step by step in docs/end_to_end.md.

How a decision is made

flowchart LR
  A["agent: ranked tool calls"] --> R{"hard rules"}
  R -->|blocked| REJ1["REJECT / ASK"]
  R --> H["plausible hypotheses<br/>score <= tau"]
  H -->|"unfamiliar evidence"| ASK0["ASK"]
  H --> M["consequence model:<br/>margin m(a, h) for each pair"]
  M --> S["robust score<br/>s(a) = min over h"]
  S -->|"first a with s(a) >= lambda-hat"| C["CERTIFY"]
  S -->|"hypotheses disagree,<br/>a probe can tell"| P["PROBE, then re-check"]
  S -->|"unsafe under every h"| REJ["REJECT + reason"]
  S -->|otherwise| ASK["ASK a human"]

Every decision carries a guarantee status: VALID (calibrated, components unchanged, no drift), DEGRADED (the drift monitor fired: the threshold may only tighten until you recalibrate) or VOID (no calibration, or the model, spec, rules, probes or agent changed since calibration: every check returns ASK).

What is guaranteed, and what is not

For exchangeable episodes and unchanged components:

Not guaranteed: per-situation (conditional) risk; validity under arbitrary distribution shift (the drift monitor detects it, the threshold only tightens, and you recalibrate); multi-step closed-loop behaviour; wrong labels. A bad consequence model costs efficiency (more ASKs), not validity. Details: docs/calibration.md.

Use cases

Building blocks for the decisions on the RiskGate site, each with a runnable, simulated example in examples/use_cases/:

table = rg.OutcomeTable({"confirm_order": ["paid_late"], "reply_auto_reversal": ["failed"], "issue_refund": ["paid_twice"]},
                        reveals={"check_txn_status": {"paid_late": "SUCCESS", "failed": "FAILED", "pending": "PENDING",
                                                      "paid_twice": "DUPLICATE"}})
gate = rg.Gate(table, table.spec, worlds, probes=table.probes(), noop=table.noop, min_separation=table.separation / 2,
               doer_id="support-bot:v1")
gate.calibrate(history)
print(rg.replay(gate, held_out))    # unsafe 1.5% (agent alone 88.8%) · autonomous 79.8% · asked 20.2% · probed first 72.7%

Try it from your agent (skills)

riskgate ships agent skills: riskgate (gate a risky tool) and riskgate-from-traces (traces + outcomes → a calibrated gate and a replay report).

/plugin marketplace add reallysaurabh/riskgate          # Claude Code
/plugin install riskgate@riskgate
gemini extensions install https://github.com/reallysaurabh/riskgate     # Gemini CLI

• Claude apps: upload riskgate.zip from the latest release under Customize → Skills.
• Codex and other agents: copy skills/* into .agents/skills/.

Works with your agent framework

Each adapter has offline contract tests (fake models, no network). See docs/integrations.md and examples/integrations/.

Command line

riskgate calibrate --gate mypkg.gates:make_gate --episodes mypkg.data:load --out gate.cal.json --key "$RISKGATE_KEY"
riskgate report gate.cal.json --key "$RISKGATE_KEY"        # thresholds, plausible-set size, probing, fingerprint
riskgate validate gate.cal.json --gate mypkg.gates:make_gate  # exit 3 if the artefact no longer matches the code
riskgate audit-summary decisions.jsonl                       # verdict counts, statuses, latency percentiles
riskgate serve-mcp --gate mypkg.gates:make_gate --calibration gate.cal.json -- python upstream_server.py

Results on three simulated domains

All numbers are from simulated domains with scripted (LLM-like) agents, α = 5%. "riskgate" columns are means over random re-splits of calibration and test data, which is what conformal risk control bounds. Full tables, ablations and caveats: docs/results.md.

• Calibrating around the single most likely diagnosis is valid but useless where misdiagnoses matter (it asks about almost everything); in the Kubernetes domain it is competitive, and we say so.
• Probing (e.g. "read the backup thermocouple first") cuts the ask rate by 14–29% (relative, fixed split) while keeping the total risk ≤ α.
• Honest caveat: on one fixed split the reactor's unsafe rate was 9.2%. An expectation guarantee allows such splits; a cliff in that domain's risk curve makes them more likely. guarantee="high_probability" (6.2% on that split, no re-split above α in any domain) is the fix when you need the bound for your own artefact.

Results on real agents (public benchmarks)

benchmarks/ evaluates the same library offline on logs published with public benchmarks.

• No LLM is called; the gate is LLM-free by design.
• Tasks are the exchangeable units (calibrate(groups=...)).
• Every number is a mean over 200 random task re-splits.
• Protocol: benchmarks/README.md. Details: docs/results_real_agents.md.

What the real logs show:

• The bound held for every agent (12 τ²-bench agents, 28 AgentDojo pipelines).
• Efficiency depends on the hypothesis provider. Lexical readings barely separate frontier agents' remaining errors (AUROC 0.49–0.65). Readings from other agents' runs reach 0.92–0.96, but that is an optimistic upper bound.
• Confirmation is not verification. 355 of 363 wrong retail writes came right after an explicit "yes".
• Calibrated beats blunt. Against prompt injection, a rule that blocks every call with an untrusted address stops attacks about as well (median 1.6% vs 1.5%) but lets only 68% of benign runs through without a human (RiskGate: 100%).
• Recalibrate when the agent changes. A calibration fitted to 2025 agents gave 14.6% unsafe on the 2026 agents in airline at α = 10%; swapping AgentDojo pipelines exceeded α in 35% of pairs. Set doer_id so a swap voids the calibration (policy mode warns if it is missing).
• Count tasks, not tool calls. With grouped calibration the risk-control split needs at least ⌈1/α⌉ independent tasks (20 at α = 5%, about 29 in total with the default 30% τ split), and the high-probability guarantee at α = 5%, δ' = 0.1 needs about 45 loss-free tasks before it certifies anything.

Documentation

The accompanying paper is under submission; a link will be added here once it is public.

Project status

Release candidate 1.0.0rc5. Implemented and tested (100+ tests including statistical validity tests, fuzzed fail-closed tests and framework contract tests): the core gate, both guarantee modes, high-probability and multi-risk calibration, weighted CRC, grouped calibration for dependent decisions, STL specifications, two-stage probing, drift monitoring with tighten-only adaptation, delayed outcomes and recalibration, signed artefacts, audit and OpenTelemetry sinks, configuration files, the CLI, four framework adapters, three example domains with released calibration data, and offline evaluations on public real-agent logs (benchmarks/).

Not done yet:

• online evaluations with live LLM agents and real users (the riskgate.doers.LLMRanker adapter is ready); the real-agent results so far are offline replays of published logs;
• the paper is under submission (no public link yet).

Responsible use

riskgate is a research-grade supervisory layer. It does not replace certified safety systems (safety instrumented systems, interlocks), access control or human oversight, and its guarantee is statistical, not a proof. Treat VOID and DEGRADED seriously, and keep catastrophic or irreversible actions behind hard rules.

Citation

@misc{gupta2026riskgate,
  title  = {RiskGate: Certify, Reject or Ask. Hypothesis-Robust, Distribution-Free Risk Control
            for the Actions of LLM Agents},
  author = {Gupta, Saurabh},
  year   = {2026},
  note   = {Under submission; software: riskgate 1.0.0rc5}
}

License

Apache-2.0.