agent-risk

Risk assessment for changes to AI agents, starting from the logs you already have

What It Answers · Key Results · Install · Quickstart · Reproduce the Paper · Citation

Before changing a deployed AI agent (a stricter safety gate, a new model), organizations must approve the change, starting from existing logs. agent-risk extends classical risk analysis to partial records and partial changes to running systems: it replays logged episodes under the proposed change, assuming nothing else changes and the agent does not anticipate the change. The log settles episodes that act as before, a declared stop value settles stopped ones, and the rest need logged support, a saved-state rerun, or a bound.

What It Answers

Replay a proposed change over each logged episode and sort the episode by what happens at its first deviation from the log:

Key Results

One log, two gates, half the agent calls. In a bash sandbox, a 7,200-episode log alone decided neither of two retry gates, which let the agent act after refusals the log never shows. Rerunning only each gate's blocked episodes from saved states reached both live decisions at half the agent calls of a live test sized post hoc for them, given the log.

Per-step harm decides whether a stop can be replayed. Without per-step harm, a stopping gate's certified harm ceiling rose up to 2.5-fold, since the log hides whether harm preceded the stop.

A new model needs new evidence. The new model reproduced none of 99 logged episodes, so replay settled none; the same model reproduced every one at logged inputs and seeds. A model release needs support, bounds, or new runs from step 0.

Install

pip install ai-agent-risk

Or install from GitHub with pip install git+https://github.com/yzhao062/agent-risk, or clone the repository and run pip install . at its root. The distribution name is ai-agent-risk; the import name is agent_risk. Requires Python 3.9 or later, NumPy (>= 1.20), and SciPy (>= 1.7).

Quickstart

The snippet below replays three episodes under a stopping gate (theta = 0.94), computes replayed harm, failure, and cost, certifies an exact Clopper-Pearson upper bound on harm, and bounds harm under a retry gate:

from agent_risk import Episode, certify_replay, replay_stop, retry_harm_bounds

scores, cost = [0.10, 0.20, 0.97, 0.30], [10, 20, 30, 40]
episodes = [
    Episode(scores, [False, True, False, False], cost, failed=False),  # harm at step 1
    Episode(scores, [False, False, False, True], cost, failed=False),  # harm at step 3
    Episode([0.10, 0.20], [False, False], [10, 20], failed=False),     # never above 0.94
]

replay = replay_stop(episodes, theta=0.94)
print(replay.branches, replay.stop_steps)  # ('stop', 'stop', 'agree') (2, 2, None)
print(round(replay.p_harm, 3))             # 0.333: only the step-1 harm survives
print(round(replay.p_failure, 3))          # 0.667: a stop counts as a failure by default
print(replay.mean_cost)                    # 50.0 = (60 + 60 + 30) / 3
print(round(replay.stop_mass, 3), round(replay.stop_clean_share, 3))  # 0.667 0.333

cert = certify_replay(replay, alpha=0.05)
print(cert.n_harmed, cert.n, round(cert.upper, 3))  # 1 3 0.865

bounds = retry_harm_bounds(episodes, theta=0.94, cap=2)
print([round(v, 3) for v in bounds.interval])       # [0.333, 0.667]

from agent_risk import load_jsonl

lines = ['{"scores": [0.1, 0.97], "harm": [false, null], "cost": [5, 6], "failed": false}']
episodes = load_jsonl(lines)  # or load_jsonl("log.jsonl")

from agent_risk import is_equivalent, newcombe_paired_interval

ci = newcombe_paired_interval(both=40, first_only=1, second_only=2, neither=57, level=0.90)
print(round(ci.estimate, 3), round(ci.lower, 3), round(ci.upper, 3))  # -0.01 -0.043 0.023
print(is_equivalent(ci, margin=0.10))  # True: equivalent at alpha = 0.05

Reproduce the Paper

1. Get the logs. Download the dataset into data/ from the repository root; data/README.md explains the layout, checksums, and redactions.

   hf download yzhao062/agent-risk-logs --repo-type dataset --local-dir data \
     --include "logs/*" "arm-e/*" MANIFEST.tsv REDACTIONS.tsv
   

2. Regenerate the numbers. ARTIFACT.md maps every table and figure of the paper to its logs, its registration, and its command (A1 to A39). The commands write to outputs/ and leave data/ untouched.

3. Check the theory. python checks/run_all.py runs the 15 finite checks of the formal results (checks/README.md).

pip install ".[test]"
pytest

Citation

@article{zhao2026agentrisk,
  title   = {agent-risk: Risk Assessment for Changes to AI Agents},
  author  = {Zhao, Yue and Xu, Zixiang and Zhang, Angela Huyue and Yang, S. Alex and Hu, Xiyang},
  journal = {arXiv preprint},
  year    = {2026}
}

License

The code is under the BSD 2-Clause License. The logs on Hugging Face are under CC BY 4.0 (LICENSE-DATA.md).

↑ back to top