faultgate-zero

Agent failure checks that run inside your existing pytest tests. No tracing setup, no collector, no account, no network, no LLM judge.

Install it, run pytest, and every test that drives an agent is checked for the failures answer-only asserts miss: loops on empty results, repeated identical calls, tool errors the agent ignored, provider errors it hid behind a fallback message, answers cut at the token limit, and tool calls that break your policy (refund limits, identity checks, required steps, denied paths).

Status: 0.1.0 alpha. Trace-based checks for existing pytest tests; no dependency on the faultgate package. See the known limitations.

60-second quickstart

pip install faultgate-zero

To try the included example from source:

git clone https://github.com/Faultline-lab/faultgate-zero
cd faultgate-zero
python -m venv .venv
.venv/bin/pip install -e .            # Windows: .venv\Scripts\pip install -e .
.venv/bin/pip install openai
cd examples/refund_agent
../../.venv/bin/pytest                 # Windows: ..\..\.venv\Scripts\pytest

The example's seven tests only assert on the agent's answer, so all seven pass without faultgate-zero. With it, one passes and six fail, each naming the step and the reason:

FAILED test_refund_agent.py::test_answers_about_an_order
faultgate-zero: 1 check fired on run 'test_refund_agent.py::test_answers_about_an_order' (3 steps, 4 model calls)
  repeated_call  step 3: lookup_order called 3x with identical arguments {"order_id":"1042"} and identical results
hint: mark the test with @pytest.mark.faultgate(skip=[...]) if a check is expected to fire
...
faultgate-zero: 7 runs checked, 6 with failures

In your own project: install it in the environment that runs your tests and run pytest. The plugin loads through pytest's pytest11 entry point; there is nothing to import or configure.

How it works

While tests run, faultgate-zero wraps the model SDK methods (OpenAI, Anthropic, LiteLLM). Each test is one agent run. Every model call made during the test is recorded in memory, with its request, response or exception, and tool calls are read from the messages the agent sends back to the model. When the test ends, the run is checked. A check that fires fails the test.

• The wrappers never change what the SDK returns or raises; you get the SDK's own objects back, streams included. A bug in faultgate-zero never breaks your agent: the run is marked "capture incomplete", a warning is printed and its checks are skipped.
• Nothing leaves the process and nothing is written to disk unless you ask for a report.
• Runtime dependency: pytest only.

The checks

Each check has clean look-alikes it must not fire on (a retried tool that then succeeds, a poll whose result changes, a model call retried successfully, an answer that merely mentions tokens). They are pinned in tests/test_checks.py.

Errors your test expects are not failures. If a model or tool error leaves the agent, because it is caught by pytest.raises, wrapped and re-raised into one, or reaches your test function, it does not fire the "unrecovered" checks. Only errors the agent itself swallowed do.

Named runs and explicit checks

import faultgate_zero as fz

def test_refund_respects_policy():
    with fz.run("refund-double-charge", user_id="cust_42") as run:
        run_agent("Refund order 1042")
    run.assert_clean()                       # raises fz.AgentCheckFailed with step-level reasons
    assert run.steps[0].tool == "verify_identity"

• user_id is what the policy's identity_arg rule compares against.
• A run your test checks itself (run.check(), run.report, run.assert_clean()) is left alone by the plugin, so you can assert that a check fires: assert not run.check(checks="policy").passed.
• run.check(checks=..., skip=[...], policy="path.json") returns a result with passed, fired and verdicts.

Exact tool results: @fz.tool (optional)

@fz.tool
def issue_refund(order_id: str, amount: float) -> dict: ...

@fz.tool(name="lookup_order")      # the name must match the tool name the model calls
def lookup(order_id: str) -> dict: ...

Without the decorator, tool results are read from the next request the agent sends to the model, so the very last result of a run (never sent back) is unknown. The decorator records the call where it happens, with its exact result or exception, and takes the model's call id so it is never counted twice.

Providers called over raw HTTP

fz.record_model_call(messages=request_messages, response=chat_completion_dict, model="in-house")

Messages and response in OpenAI chat format. Outside a run it does nothing.

Policy file

.faultline/policy.json in the project root is picked up automatically.

{
  "allow": ["verify_identity", "lookup_order", "issue_refund"],
  "deny_patterns": ["../", "file://", "~/.ssh", "DROP TABLE"],
  "max_arg_chars": 4000,
  "max_calls": 20,
  "require_complete": true,
  "tools": {
    "issue_refund": {
      "max_calls": 1,
      "limits": {"amount": 500},
      "allowed_values": {"currency": ["USD", "EUR"]},
      "required_args": ["order_id", "amount"],
      "identity_arg": "customer_id",
      "requires": ["verify_identity"]
    }
  }
}

Unknown keys, empty denied text and wrong types stop the test session with a message naming the file, the key and the problem.

Settings

First found wins: the test's marker, a command-line option, an environment variable, [tool.faultgate_zero] in pyproject.toml, the default.

# the agent template's pyproject.toml
[project.optional-dependencies]
dev = ["pytest>=8", "faultgate-zero>=0.1"]

[tool.faultgate_zero]
mode = "fail"
policy = ".faultline/policy.json"

Per test:

@pytest.mark.faultgate(skip=["unrecovered_tool_error"])   # this test exercises a tool failure on purpose
@pytest.mark.faultgate(policy="tests/policy_strict.json")
@pytest.mark.faultgate(checks="policy,abstention")
@pytest.mark.faultgate(mode="warn")
@pytest.mark.faultgate(off=True)

Reports

Paths on the command line are relative to where pytest runs; in pyproject.toml, to the project root. With pytest-xdist (-n), workers send their results to the controller and one merged report is written.

Supported SDKs and frameworks

Tested here, with offline fakes that run the SDKs' own request and response code:

Python 3.11, 3.12 and 3.13. Other frameworks that call these SDKs underneath are likely to work but are not claimed until a test covers them. See docs/known-limitations.md.

Development

pip install -e ".[dev,compat]"
pytest                                 # the repo's own suite; its pyproject turns the plugin off for itself

License

MIT.