resilientforge 0.3.0

Persistent, cross-run failure memory for tool-calling agents.

ResilientForge

ResilientForge is a small, framework-agnostic Python library that sits on top of an agent's existing tool-calling loop and adds persistent, cross-run failure memory.

When a wrapped agent hits a tool failure or violates a defined invariant, ResilientForge captures the failure, checks a local failure oracle for a previously-successful fix, applies it if found — no model call needed — and, if it has to generate a new fix via a model call, writes that fix back to the oracle so the next occurrence of the same failure shape is resolved without another model call, even if the specific values involved are different. Once a fix has proven itself reliably enough times, it's promoted into a standing guard that fixes the arguments before the tool is even called — the failure stops recurring at all, instead of recurring once and then being fixed on retry.

Status: v0.2.0, Alpha — not yet published to PyPI. Full phase-by-phase development history is in CHANGELOG.md; see docs/architecture.md for how it actually works under the hood, and "Validation" below for how it's held up against a real external agent.

Quickstart

pip install resilientforge

from resilientforge import wrap

def create_event(date: str, title: str = "Event") -> dict:
    import re
    if not re.match(r"^\d{4}-\d{2}-\d{2}$", date):
        raise ValueError(f"could not parse date '{date}'")
    return {"date": date, "title": title, "status": "created"}

def reflect(context):
    # Stand-in for a real model call — see integrations/raw_tool_loop.py's
    # create_anthropic_reflect() for a real Anthropic-backed one.
    return {
        "strategy": "reformat_argument",
        "transforms": [{"argument": "date", "transform": "parse_relative_date_to_iso"}],
    }

wrapped = wrap(create_event, reflect=reflect)

wrapped.invoke(date="next Friday", title="Standup")
# -> {"date": "2026-07-17", "title": "Standup", "status": "created"}
# (recovered: date wasn't ISO 8601, reflect() proposed reparsing it, retried, succeeded)

wrapped.invoke(date="next Tuesday", title="Retro")
# -> {"date": "2026-07-21", "title": "Retro", "status": "created"}
# (recovered via the recipe learned above — reflect() was NOT called again)

That's the raw tool-calling loop integration — the simplest entry point, and the reference implementation. For a real Anthropic tool-use loop, an OpenAI function-calling loop, or a LangGraph ToolNode, see integrations/raw_tool_loop.py / integrations/langgraph_adapter.py and the runnable demos in examples/:

python examples/raw_loop_demo.py
python examples/langgraph_demo.py
python examples/guards_demo.py

Does it work? Real numbers, not a claim

The table below is generated by pytest tests/failure_injection — seven scenarios reproducing the real-world failure patterns this project targets (malformed tool-call JSON, a missing required field, a natural-language date where ISO 8601 was expected, a transient timeout, a wrong-typed argument, a longer-running recurring-date scenario built specifically to demonstrate guard promotion, and a scenario whose correct fix can't be guessed from the arguments alone, built to demonstrate speculative branching), each run once with no recovery mechanism (baseline) and once wrapped with ResilientForge, with different literal values per trial:

Scenario	Trials	Baseline recovery	Recovery (ResilientForge)	Avg attempts to recovery	Oracle hit rate (after 1st occurrence)	Guard promoted	Prevention rate	Avg candidates considered
malformed_json_args	5	0%	100%	0.6	100%	yes	100%	0.2
missing_required_field	5	0%	100%	0.6	100%	yes	100%	0.2
natural_language_date	5	0%	100%	0.6	100%	yes	100%	0.2
transient_timeout	5	0%	100%	1.0	100%	no	0%	0.2
wrong_type_argument	5	0%	100%	0.6	100%	yes	100%	0.2
recurring_date_guard	8	0%	100%	0.4	100%	yes	100%	0.1
ambiguous_fix_candidates	6	0%	100%	2.2	0%	no	0%	2.2

The "oracle hit rate" column is Phase 1's proof: after the first occurrence of a given failure shape is recovered once (which needs one model call), every later occurrence — even with completely different literal values — resolves via the oracle with zero additional model calls. The next two columns are Phase 2's proof: once a fix has been applied reliably enough times (3, by default), it's promoted into a standing guard that fixes the arguments before the tool is even called — transient_timeout is the one scenario where no guard is promoted, because its correct "fix" is just a blind retry with no argument to change. recurring_date_guard is the dedicated demonstration: 8 trials, the first 3 cross the promotion threshold reactively, the remaining 5 use dates never seen before and are all prevented outright, not merely recovered from. The last column is Phase 3's: ambiguous_fix_candidates still recovers 100% of trials via real, per-candidate verification, at a real, reported cost — unlike every other scenario, its oracle hit rate is 0%, because considering multiple candidates means a model call is made every round regardless of whether a recipe already exists. Regenerate this table yourself with pytest -s tests/failure_injection (also written to tests/failure_injection/reports/latest.md) — these numbers were regenerated fresh against the current code, including the signature- normalization fixes described below (they didn't move: none of these seven scenarios exercise HTTP status text or hex byte literals).

Validation: tested against a real external agent, not just our own scenarios

The numbers above come from scenarios this project wrote itself. To check whether the oracle's recovery and signature-matching logic hold up against real, unscripted failures, we ran 3 rounds of validation against langchain-ai/react-agent — a real external LangGraph agent, wrapped with a real web-search tool and a real URL-content-extraction tool, zero engineered failures.

• Round 1 found a real bug before the actual research question could even be tested — async tool support was completely missing from integrations/langgraph_adapter.py. Fixed, then confirmed cross-session recurrence-recognition on the one real failure shape that showed up (a network timeout).
• Round 2 widened the real failure surface (a second, real tool) and found two opposite bugs in signature normalization: a false-merge (two genuinely different real HTTP errors collapsed into one signature) and a missed-match (one real problem split across two signatures) — plus a third finding, a fix recorded as "recovered" that was actually structurally inert.
• Round 3 fixed and confirmed all three against the exact real cases that exposed them, and in the process found a fourth instance of the same inert-fix bug via a different code path — also fixed and confirmed, in an addendum to the same report.

Every real bug this process found was found by testing against something real, not synthetic — and every one was fixed and re-confirmed against the exact real case that exposed it, not just asserted fixed. This is validation against one real external agent, across 3 rounds — not a claim of broader battle-testing. Full detail, including what each round does and doesn't prove: round 1 · round 2 · round 3 + addendum.

Standing guards: prevention, not just recovery

Once a fix has succeeded reliably enough times (3, by default), it's promoted into a standing guard that proactively fixes the arguments before the tool is even called — the failure stops happening at all, instead of happening once and then being fixed on retry:

wrapped = wrap(create_event, reflect=reflect, guard_promotion_min_occurrences=3)

wrapped.invoke(date="next Friday", title="A")   # recovers via reflect()
wrapped.invoke(date="next Tuesday", title="B")  # recovers via the fast-path recipe
wrapped.invoke(date="next Monday", title="C")   # recovers, and this 3rd success promotes a guard

wrapped.invoke(date="next Wednesday", title="D")
# -> succeeds on the FIRST attempt — no failure is even recorded for it,
#    and reflect() is not called: the guard fixed the date before the call.

Guards are stored the same way recipes are — locally, per oracle — and can be inspected/revoked with the CLI, or described as text for you to splice into your own system prompt (ResilientForge never touches a prompt itself — see the "Standing guards" section in docs/architecture.md):

resilientforge guards list
resilientforge guards describe

Add invariants to catch failures that don't raise an exception (e.g. a result that's missing a required field):

from pydantic import BaseModel
from resilientforge import Invariant, wrap

class EventResult(BaseModel):
    title: str
    attendees: list[str]

wrapped = wrap(
    my_agent,
    invariants=[Invariant.from_pydantic_model("valid_event", EventResult)],
)

See docs/writing_invariants.md for the full interface (deterministic, Pydantic-schema, and LLM-judged invariants, plus on_violation="recover"|"abort"|"warn").

Speculative branching: considering more than one fix

By default, a failed call gets exactly one candidate fix per attempt. Ask for more with num_branches, and the tool is still only ever called for real once per attempt — candidates are ranked by a recipe's real success_rate when one exists, so this never risks a duplicate real-world side effect:

wrapped = wrap(create_event, reflect=reflect, num_branches=3)

If your tool has no problematic real-world effect regardless of which arguments it's called with (a pure computation, a read-only lookup, a validation — not anything that books, sends, charges, or deletes for real), opt in to side_effect_free=True and ResilientForge will actually try each candidate for real, in ranked order, until one fully passes your invariants — genuine verification, not a guess:

wrapped = wrap(compute_something, reflect=reflect, num_branches=3, side_effect_free=True)

See the "Speculative branching" section in docs/architecture.md for the full design, including why the flag isn't called idempotent.

Sandboxed isolation: a hang or crash shouldn't take down your agent

isolate=True runs every real tool call in a freshly-spawned subprocess, with an optional wall-clock deadline — a hang or a crash becomes a normal recoverable failure instead of blocking or killing your process:

wrapped = wrap(create_event, reflect=reflect, isolate=True, call_timeout=5.0)

This protects the caller, not the world the tool touches — it cannot undo a real-world side effect the tool already performed before it hung or crashed (no code-level sandbox can). It also requires tool_fn to be picklable (checked eagerly, at wrap() time): a module-level function or a bound method works out of the box; a locally-defined closure or lambda needs pip install resilientforge[isolation] (adds cloudpickle, which can serialize those too — falls back to it automatically when installed). POSIX-only, best-effort resource ceilings are available too:

wrapped = wrap(create_event, reflect=reflect, isolate=True,
                max_memory_mb=512, max_cpu_seconds=10)

See the "Sandboxed isolation" section in docs/architecture.md for the full design, including why resource caps are best-effort even on POSIX systems (an empirically-confirmed, not hypothetical, caveat) and why this isn't available through the LangGraph adapter.

Local dashboard: the oracle's contents, in a browser

pip install resilientforge[dashboard]
resilientforge dashboard --oracle-path .resilientforge

A small, read-only, GET-only view of the same recipes/guards/failure history the CLI already exposes — binds to 127.0.0.1 by default, fastapi/uvicorn are only pulled in by this extra. Try it against seeded example data:

python examples/dashboard_demo.py
resilientforge dashboard --oracle-path examples/.resilientforge_dashboard

Every number in that screenshot is real, produced by actually running the recovery loop against examples/dashboard_demo.py's seeded data, not mocked for the picture — recipes and their success rates, which guards have been promoted, and the raw failure history including the one call ResilientForge correctly gave up on instead of fabricating a fix. See the "Local dashboard" section in docs/architecture.md for the full design.

Observability: watch the recovery loop as it runs

The dashboard shows the oracle's contents after the fact. metrics is a live counterpart — a callable that sees events as they actually happen: every real tool call, how a recovery ultimately resolved, and when a guard fires, gets promoted, or gets revoked.

from resilientforge import wrap
from resilientforge.telemetry import LoggingMetricsHook

wrapped = wrap(create_event, reflect=reflect, metrics=LoggingMetricsHook())

LoggingMetricsHook is a zero-dependency reference implementation using stdlib logging — configure the resilientforge.metrics logger the normal way to send it wherever your logs already go, or write your own MetricsHook (any Callable[[MetricEvent], None]) for Prometheus, Datadog, OpenTelemetry, or anything else. See examples/walkthrough_demo.py for it running live alongside the printed recovery narration, and the "Observability" section in docs/architecture.md for the full event list.

Inspecting the oracle

resilientforge list                    # recipes learned so far
resilientforge list --failures         # raw failure history
resilientforge inspect <signature>     # full detail for one recipe (prefix match OK)
resilientforge prune --dry-run         # preview what pruning would remove
resilientforge stats                   # counts + resolution-status breakdown
resilientforge dashboard               # read-only web view of all of the above

resilientforge guards list             # standing guards
resilientforge guards inspect <tool> <argument>
resilientforge guards revoke <tool> <argument>
resilientforge guards prune --dry-run  # unattended maintenance -- see docs/architecture.md
resilientforge guards describe         # text to splice into your own system prompt

Installation

pip install resilientforge               # raw Anthropic/OpenAI tool-loop adapter
pip install resilientforge[langgraph]    # + the LangGraph adapter
pip install resilientforge[dashboard]    # + the local web dashboard
pip install resilientforge[isolation]    # + isolate=True support for closures/lambdas
pip install resilientforge[semantic]     # + an optional semantic embedder (~1GB — see docs/architecture.md
                                          #   before installing: it didn't outperform the default on our own benchmark)

Development

pip install -e ".[dev,langgraph,dashboard,isolation]"
pytest tests/unit tests/integration          # fast, no network — default CI gate
pytest tests/failure_injection                # the recovery-rate proof above
pytest -m live                                 # opt-in, real API calls

See CONTRIBUTING.md for more, including how to add a new failure-injection scenario.

License

Apache 2.0 — see LICENSE.