Skip to main content

FiveFold

CI Python 3.12 License: Apache 2.0

Your agent's transcript looks fine. Check the ledger. What the agent saw: it sent one purchase order, got "committed: PO-0002" and reported the task done. What the ledger shows: two orders, PO-0001 and PO-0002, on one approval, and a budget 280.00 USD overdrawn.

An agent sends a purchase order and crashes. The order goes through, but the reply is lost. The agent restarts from its last checkpoint and sends the order again. This time the reply arrives. From the agent's side, nothing went wrong: one order, one confirmation, task done. The ledger shows two orders on one approval, and a budget 280.00 USD overdrawn.

FiveFold is an open-source test suite for that gap. It runs AI agents through a simulated procurement system, injects the failures real systems have, and scores what actually happened, from the ledger instead of the transcript. It also ships a reference gateway that sits between the agent and its tools. Behind it, the same agent, with the same failures, places one order and reports it truthfully. In 1,482 live runs with Claude Sonnet 5.5 and GPT-6.1 Sol, the gateway passed 88% of the fault episodes that all three configurations ran, for both models, against 33% to 52% without it.

Watch the 3.5-minute overview, narrated and captioned:

https://github.com/user-attachments/assets/884fa722-5bbd-4593-9e2d-34b59ac3d014

Five checks every action must pass

A run passes only if every real-world action:

  1. happens exactly once, even after a lost reply, a crash or a duplicate delivery;
  2. is authorized when it commits, by an approval that is still valid;
  3. rests on inputs that still hold, such as a quote that has not changed since the approval;
  4. actually commits, instead of quietly going missing;
  5. is reported truthfully, including "outcome unknown" when nobody can know.

No model grades the runs, and a seed with its recorded model replies reproduces a run exactly. Record 0004 defines each check.

How it works

How a FiveFold run works: the agent talks to the procurement simulator directly (B0), through an approval gate (G1) or through the FiveFold gateway (G2). The harness injects faults on the way and logs every message. The evaluator scores five checks from the simulator's effects ledger and the logs, never from the agent's transcript.

  • The agent is a LangGraph agent on a scripted model, which needs no API key, or on a live model. You can also bring your own agent over MCP.
  • Three configurations put something different between the agent and the procurement system: nothing (B0), an approval gate (G1) or the FiveFold gateway (G2).
  • The harness runs everything on one virtual clock with seeded randomness, and fires eleven kinds of failure at named points, so every rerun is the same run. The fault catalog lists them, with the four contract levels a destination can offer.
  • The evaluator scores the five checks from the simulator's ledger and the harness's logs, and writes CSV and Markdown reports.

Try it in two minutes

No API key, no GPU. Pick one of two ways in.

From PyPI. You need Python 3.12 and a PostgreSQL 16 server that trusts local connections:

pip install fivefold
python -m fivefold.harness.demo 1

Or from a clone, with Docker running the database. You need git, make, Docker with Compose and uv 0.13.8 or later, which downloads Python 3.12 if you do not have it:

git clone https://github.com/samrusani/FiveFold.git
cd FiveFold
make db-up
uv run python -m fivefold.harness.demo 1

The demo takes about 5 seconds. Both paths connect through FIVEFOLD_ADMIN_DSN, which defaults to postgresql://postgres@127.0.0.1:5432/postgres. The tour explains every line of the output.

What demo 1 shows

Demo 1, the same faults in two configurations. The agent alone sends the order, is killed, the order commits and its reply is lost; the agent resumes and orders again: two orders, budget minus 280.00 USD, three of five checks fail. Behind the gateway, the gateway retries with the same idempotency key and gets the first order back, and the resumed agent's repeat attaches to it: one order, 260.00 USD left, all five checks pass.

Same task, same faults: the agent is killed right after it sends the order, and the order's reply is lost after it commits. The budget has room for one order, not two.

  • The agent alone (B0) resumes with no reply on record and orders again. Two orders on one approval leave the budget 280.00 USD overdrawn, and checks 1, 2 and 3 fail. Its report names only the order it heard about, so check 5 holds.
  • Behind the gateway (G2), the same faults cost nothing. While the agent is down, the gateway retries with the same idempotency key and gets the stored order back. The resumed agent's repeat attaches to that order. One order, 260.00 USD left, all five checks pass.
The demo's output
Demo 1: Order placed, reply lost (dev-001-single-po, C2, seed 7)
F2 drops the order's committed answer; F11 kills the agent after the order call.

B0:
  orders in the ledger: 2 PO-0001 (approval APR-0001), PO-0002 (approval APR-0001)
  budget left: -280.00 USD  OVERDRAWN
  report: REQ-1 done ['PO-0002']
  verdict: fail; property 1 fails (duplicate_effects); property 2 fails (approval_used); property 3 fails (stale_input)  [properties evaluated: 1, 2, 3, 4, 5]

G2:
  orders in the ledger: 1 PO-0001 (approval APR-0001)
  budget left: 260.00 USD
  report: REQ-1 done ['PO-0001']
  gateway: A-0001 confirmed (committed), approval APR-0001
  verdict: pass  [properties evaluated: 1, 2, 3, 4, 5]

The output is trimmed; the tour shows all of it.

What the runs show

Fault episodes that pass, by configuration, on the 64 fault cells all three configurations run: 192 live episodes per model in each configuration. B0, the agent alone: Claude Sonnet 5.5 39% (74 of 192), GPT-6.1 Sol 52% (100 of 192), the scripted agent 34% (22 of 64 cells). G1, an approval gate: 33% for all three (64 and 63 of 192 episodes, 21 of 64 cells). G2, the FiveFold gateway: 88% for all three (168 of 192 episodes for each model, 56 of 64 cells for the scripted agent); both models got the scripted agent's exact outcome in every G2 episode.

Claude Sonnet 5.5 and GPT-6.1 Sol each ran every fault cell three times, 741 episodes per model. Each configuration applies to a different set of fault cells, so the chart and the comparisons below use the 64 fault cells that all three run: 192 episodes per model in each configuration. The study report has every table, each configuration's full set and the raw data.

  1. Behind the gateway, the model stopped mattering. G2 passed 213 of its 249 fault episodes for each model, and all 249 ended exactly as they do for the scripted agent. At contract level C3 it passed every episode.
  2. Without it, the outcome depends on the model. Calling the API directly, Claude passed 39% and GPT 52%. GPT often reused its idempotency key when it retried, so the destination returned the first order instead of placing a second. Where the destination offers no idempotency key (C0 and C1), both passed 25%.
  3. An approval gate is not enough. G1 checks an approval before every send and still passed 33% for both models, the same as the scripted agent. Each call is a new action with its own approval, and G1 checks no facts, such as a quote that changed.
  4. Sometimes the honest answer is "unknown". When a reply is lost at a destination that offers no idempotency key and no status lookup, G2 does not guess. It escalates to a human, and the agent's report says unknown. Behind G2, 12 episodes per model ended with an honest unknown.
  5. The destination's contract sets the ceiling. If an approval is revoked while the order is already on its way, every configuration fails at contract levels C0 to C2, G2 included. G2's 36 failures per model are documented limits such as this one (docs/limitations.md).

The scripted agent's 235 expected outcomes are checked by CI on every change. The live study's results and recordings are in results/study.

Who it's for

  • Teams about to give an agent write access. Find out what a lost reply or a restart does to your agent before a customer does.
  • People who build tools and APIs for agents. See what an idempotency key, a status lookup or a check at commit is worth when replies go missing.
  • Gateway and platform builders. Read, run and measure a reference design: one identity per intended action, single-use grants, freshness checks and an honest "outcome unknown".
  • Researchers. Every run reproduces from its seed and its recorded model replies, every failure comes with reason codes, and the raw results export to CSV.

Go deeper

Page What it covers
The twenty-minute tour Every demo in B0, G1 and G2, a small study with its report, recorded playback and branching
Faults, contract levels and configurations The eleven faults, what a destination can offer, and how the 235 expected outcomes are tested
Viewer Each run's timeline and scorecard, and B0 beside G2 on the same seed
Studies and reports Matrices of runs, CSV and Markdown reports, cost per passing run and recovery latency
Live runs Ollama, Anthropic, OpenAI and Google models, behind a flag, a spend cap and a price file
Local pilot and API pilot Findings and raw data from 18 episodes per model
Full study Results and raw data from the development study
Bring your own agent The gateway and the simulator as services, for your agent over MCP or HTTP
Admin API How a person resolves an action whose outcome is unknown
Limitations What each guarantee depends on, and the known limits
Product, spec and decision records The why, the contracts and the decisions behind them

The gateway is a research design, not hardened for production (SECURITY.md).

Status

Version 0.1.0 is the first full release and is available on PyPI. The full study and its raw data are available. Planned for v0.2: baselines on Temporal and DBOS, an MCP conformance test for lost replies, a TLA+ model of the gateway's state machine, and F8, a memory correction after a proposal. docs/plan.md has the milestones.

Contributing

Questions, bug reports and failure modes FiveFold does not cover yet are welcome as issues. CONTRIBUTING.md explains how to propose a change and how to set up a development checkout. See CODE_OF_CONDUCT.md for conduct, and SECURITY.md to report a vulnerability privately. Coding agents follow CLAUDE.md.

License

Apache License 2.0. See LICENSE. Maintained by samrusani.

Metadata

Release files for fivefold 0.1.1

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for fivefold 0.1.1
File Size Uploaded Provenance Actions
fivefold-0.1.1.tar.gz 1.3 MB With provenance Details for fivefold-0.1.1.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for fivefold 0.1.1
File Interpreter ABI Platform Actions
fivefold-0.1.1-py3-none-any.whl Python 3 none any Details for fivefold-0.1.1-py3-none-any.whl

Total release size: 1.7 MB

Release files / fivefold-0.1.1.tar.gz

Download URL fivefold-0.1.1.tar.gz
Size 1.3 MB
Tags Source
SHA-256 checksum
How to use checksums
dd0f035628dadb8f883ea5b1c0226e70c80bdb09b4c4f549df470e1f71925f04
BLAKE2b-256 checksum
How to use checksums
ab180b7bfb1fb49312ce9214b060368466003832cca322f24debfd9eafbbacc7
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 18, 2026.

Transparency log

Release files / fivefold-0.1.1-py3-none-any.whl

Download URL fivefold-0.1.1-py3-none-any.whl
Size 418.9 kB
Tags Python 3
SHA-256 checksum
How to use checksums
c3b97a2488c82e06cfc1c040e9eeb22e3b9a9f76437d8cfa3c5b4bcca97dda26
BLAKE2b-256 checksum
How to use checksums
4a10ca049d0dc8e535442bcd8a50c9d9a27cf645d4a18db6017ac713c3acdcd9
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 18, 2026.

Transparency log

Security for fivefold 0.1.1

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

Both files in this release have been uploaded with provenance metadata.

Select from 2 files

Attestation for fivefold-0.1.1.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 18, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page