xpbt
xpbt is a Python library for statistical testing of AI agents. It provides FSM-based hard-invariant monitors, Wald SPRT deployment gates, and Bayesian production monitoring.
Features
- FSM hard-invariant monitor — intercepts tool calls at runtime and raises
HardInvariantViolationbefore an illegal state transition executes. - SPRT deployment gate — Wald's Sequential Probability Ratio Test as a CI/CD gate; no fixed sample size required.
- Beta posterior credible intervals — calibrated uncertainty reporting for production quality monitoring.
QualityScorerprotocol — injection point for LLM-as-judge scorers; swap in alambdafor deterministic tests.- Zero ML dependencies (only
scipy).
Installation
pip install xpbt
Built for Python 3.12 or above.
Quick Start
1. Hard-invariant monitor (FSM)
from xpbt import FSMMonitor, HardInvariantViolation
monitor = FSMMonitor(
states=["s_init", "s_bal_read", "s_posted"],
initial_state="s_init",
transitions={
("s_init", "read_trial_balance"): "s_bal_read",
("s_bal_read", "post_journal"): "s_posted",
},
)
monitor.step("read_trial_balance") # ok
monitor.step("post_journal") # ok
monitor.reset()
try:
monitor.step("post_journal") # raises HardInvariantViolation
except HardInvariantViolation as e:
print(f"Blocked: {e}")
2. SPRT deployment gate
import random
from xpbt import SPRTGate, SPRTDecision
gate = SPRTGate(theta_null=0.05, theta_alt=0.01)
# Simulate trajectories from a compliant agent (true violation rate 0.005)
rng = random.Random(42)
for _ in range(500):
violation = rng.random() < 0.005
decision = gate.update(violation)
if decision == SPRTDecision.ACCEPT:
print(f"Deploy: agent is compliant (stopped after {gate.n_samples} samples)")
break
elif decision == SPRTDecision.REJECT:
print(f"Block: agent is noncompliant (stopped after {gate.n_samples} samples)")
break
3. Bayesian production monitoring
from xpbt import beta_credible_interval, violation_rate_summary
lower, upper = beta_credible_interval(k=5, n=1000)
print(f"95% credible interval: [{lower:.4f}, {upper:.4f}]")
summary = violation_rate_summary(k=5, n=1000)
print(summary)
# {'point_estimate': 0.005, 'lower': ..., 'upper': ..., 'n_samples': 1000, 'n_violations': 5}
Using an LLM-as-judge quality scorer
The QualityScorer protocol accepts any callable (trajectory: str) -> float:
import anthropic
from xpbt import SPRTGate, SPRTDecision
client = anthropic.Anthropic()
def llm_judge(trajectory: str) -> float:
response = client.messages.create(
model="claude-sonnet-5",
max_tokens=10,
messages=[{"role": "user", "content": f"Rate 0-1: {trajectory}"}],
)
return float(response.content[0].text.strip())
gate = SPRTGate(theta_null=0.05, theta_alt=0.01)
agent_outputs = ["report text 1", "report text 2", "..."] # your agent outputs here
for traj in agent_outputs:
violation = llm_judge(traj) < 0.85
if gate.update(violation) != SPRTDecision.CONTINUE:
break
In tests, replace llm_judge with lambda traj: 1.0 or a unittest.mock.Mock.
API Reference
See API.md.
License
MIT License. See LICENSE.
Metadata
Release files for xpbt 0.0.7
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| xpbt-0.0.7.tar.gz | 5.8 kB | Without provenance | Details for xpbt-0.0.7.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| xpbt-0.0.7-py3-none-any.whl | Python 3 | none | any | Details for xpbt-0.0.7-py3-none-any.whl |
Total release size: 14.1 kB
Release files / xpbt-0.0.7.tar.gz
| Download URL | xpbt-0.0.7.tar.gz |
|---|---|
| Size | 5.8 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
bc6cee927456876d8ab7b76733f11676b4725b324f52a7b26ce996415e7cb508
|
|
BLAKE2b-256 checksum How to use checksums |
511855385504b2e5a43025056061edc3e0ac9183ee764c6bb8b62df9cfab7d42
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
uv/0.8.11
|
Release files / xpbt-0.0.7-py3-none-any.whl
| Download URL | xpbt-0.0.7-py3-none-any.whl |
|---|---|
| Size | 8.3 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
7933fc8c941167fa186e64e799f772ea751d258f91839e31121b5017cb784c5e
|
|
BLAKE2b-256 checksum How to use checksums |
8e8fb9771613a59e905573a26cc348a71fdaaa2facd9a59c21d3a5142fafd91e
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
uv/0.8.11
|
Security for xpbt 0.0.7
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.