xpbt

xpbt is a Python library for statistical testing of AI agents. It provides FSM-based hard-invariant monitors, Wald SPRT deployment gates, and Bayesian production monitoring.

Features

• FSM hard-invariant monitor — intercepts tool calls at runtime and raises HardInvariantViolation before an illegal state transition executes.
• SPRT deployment gate — Wald's Sequential Probability Ratio Test as a CI/CD gate; no fixed sample size required.
• Beta posterior credible intervals — calibrated uncertainty reporting for production quality monitoring.
• QualityScorer protocol — injection point for LLM-as-judge scorers; swap in a lambda for deterministic tests.
• Zero ML dependencies (only scipy).

Installation

pip install xpbt

Built for Python 3.12 or above.

Quick Start

1. Hard-invariant monitor (FSM)

from xpbt import FSMMonitor, HardInvariantViolation

monitor = FSMMonitor(
    states=["s_init", "s_bal_read", "s_posted"],
    initial_state="s_init",
    transitions={
        ("s_init", "read_trial_balance"): "s_bal_read",
        ("s_bal_read", "post_journal"): "s_posted",
    },
)

monitor.step("read_trial_balance")  # ok
monitor.step("post_journal")        # ok

monitor.reset()
try:
    monitor.step("post_journal")    # raises HardInvariantViolation
except HardInvariantViolation as e:
    print(f"Blocked: {e}")

2. SPRT deployment gate

import random
from xpbt import SPRTGate, SPRTDecision

gate = SPRTGate(theta_null=0.05, theta_alt=0.01)

# Simulate trajectories from a compliant agent (true violation rate 0.005)
rng = random.Random(42)
for _ in range(500):
    violation = rng.random() < 0.005
    decision = gate.update(violation)
    if decision == SPRTDecision.ACCEPT:
        print(f"Deploy: agent is compliant (stopped after {gate.n_samples} samples)")
        break
    elif decision == SPRTDecision.REJECT:
        print(f"Block: agent is noncompliant (stopped after {gate.n_samples} samples)")
        break

3. Bayesian production monitoring

from xpbt import beta_credible_interval, violation_rate_summary

lower, upper = beta_credible_interval(k=5, n=1000)
print(f"95% credible interval: [{lower:.4f}, {upper:.4f}]")

summary = violation_rate_summary(k=5, n=1000)
print(summary)
# {'point_estimate': 0.005, 'lower': ..., 'upper': ..., 'n_samples': 1000, 'n_violations': 5}

Using an LLM-as-judge quality scorer

The QualityScorer protocol accepts any callable (trajectory: str) -> float:

import anthropic
from xpbt import SPRTGate, SPRTDecision

client = anthropic.Anthropic()

def llm_judge(trajectory: str) -> float:
    response = client.messages.create(
        model="claude-sonnet-5",
        max_tokens=10,
        messages=[{"role": "user", "content": f"Rate 0-1: {trajectory}"}],
    )
    return float(response.content[0].text.strip())

gate = SPRTGate(theta_null=0.05, theta_alt=0.01)
agent_outputs = ["report text 1", "report text 2", "..."]  # your agent outputs here
for traj in agent_outputs:
    violation = llm_judge(traj) < 0.85
    if gate.update(violation) != SPRTDecision.CONTINUE:
        break

In tests, replace llm_judge with lambda traj: 1.0 or a unittest.mock.Mock.

API Reference

See API.md.

License

MIT License. See LICENSE.