bendex-resilience 0.1.0

Tests whether an AI agent or stateful system recovers or cascades after a perturbation, using twin-trajectory divergence analysis. Works with minimal historical data and no model of the system's internals.

Bendex Resilience

Tests whether an AI agent (or any stateful system) recovers or cascades after a perturbation — a corrupted memory entry, a failed tool call, a config change — using twin-trajectory divergence analysis.

What makes this different

Most AI agent testing tools inject a failure and check one thing: did the task still complete? That's a pass/fail light switch — it can't tell the difference between "acting weird but fine" and "acting weird and actually breaking."

This tool instead:

1. Clones your system into two copies.
2. Applies a small, controlled perturbation to one copy.
3. Runs both copies through the same real workload.
4. Watches whether their behavior stays in sync (recovers) or drifts apart (cascades).

No historical baseline required before you can test it. No model of your system's internals needed. It works by comparing the two live copies to each other, not by comparing to weeks of past data.

Validated results

Tested against a real language model (Qwen2.5-0.5B-Instruct), not just synthetic data — see colab_shareable_demo.py, runnable yourself in one click:

Metric	Result
Accuracy (40 trials: 20 healthy, 20 real injected failures)	95.0% (38/40)
Healthy systems correctly left alone	18/20 (2/20 false alarms)
Real failures caught	20/20

Also validated on systems where classical stability-analysis methods don't apply at all — nonlinear systems (90% accuracy against an exact known answer) and systems with memory/delay (100% accuracy against an exact known answer). Also validated to work with far less historical data than standard covariance-based drift detection methods require.

Known limitations — stated plainly, not hidden

• Detection quality depends on calibration data size. Measured: 62% accuracy at 7 calibration observations, 83% at 25, 95% at 40+. Use at least ~40 observations where possible.
• Does not yet automatically find your system's worst-case vulnerability. You tell it what scenario to test; it doesn't discover unknown weaknesses on its own. This is open, unsolved work.
• Does not localize which specific subsystem is failing — tells you that something broke, not which part of a multi-component system.

Quick start

from resilience_v2 import ResilienceDetector

detector = ResilienceDetector()
detector.calibrate(nominal_trajectory)   # your system's normal behavior
result = detector.check(test_trajectory) # behavior after a perturbation
print(result)  # {'verdict': 'RECOVERS' or 'CASCADES', 'flagged_at_step': ..., ...}

Or run it as an API — see api.py (FastAPI, single /analyze endpoint, stateless).

Files

• resilience_v2.py — core detector
• api.py — REST API wrapper
• colab_shareable_demo.py — real-LLM validation demo, runnable in Colab
• tests/ — stress tests (scale, edge cases, false-positive rate)

License

MIT