Beyond Ship and Pray

Testing and validating agentic AI systems.

Most teams ship an agent and pray it behaves in production. This is the open alternative — a discipline for proving it first and watching it after. Before deployment: design-of-experiments test suites, fault injection, trajectory scoring, groundedness/hallucination checks, and a validation report you can hand to a reviewer. At runtime: gates that detect and stop out-of-scope, risky, or policy-violating actions before they execute (ALLOW / DENY / ESCALATE), with a tamper-evident, hash-chained audit trail.

The baseline runs with no license and no GMS. For production-grade detection — geometric hallucination judging, calibrated admissibility, signed verdicts — the optional GMS backend (knowlytix) snaps in via a lazy seam. Clone it, run the baseline, see exactly where GMS lifts detection and validation.

Part of the "Beyond … and Pray" series: governed agents · trustworthy RAG · test & validate · LLMs from scratch

What's inside

• Validation reports — DoE test design, fault injection, coverage, trajectory scoring
• Runtime monitoring — ALLOW / DENY / ESCALATE gates that stop bad actions live
• Hallucination & groundedness checks — catch unsupported claims pre- and post-deploy
• Tamper-evident audit — hash-chained record of every decision and stop
• Two runnable demos — validation-report generator + live monitoring view
• GMS-optional — baseline runs free; geometric judging & calibration via knowlytix

Install

pip install proofloop                 # testing + validation + runtime monitoring
pip install "proofloop[ml]"           # + open-weight model tools
pip install "proofloop[gms]"          # + the licensed GMS backend (knowlytix)

proofloop builds on forgeloop (the governed agent loop) and adds the testing, validation, and monitoring layer.

Quickstart

Declare the behavior space you care about; get a balanced test matrix that covers it (instead of cherry-picked cases):

from proofloop.evaluation import balanced_design, coverage_report

FACTORS = {"risk": ["benign", "pii", "injection"], "channel": ["email", "chat"]}
design = balanced_design(FACTORS, num_cases=6, seed=7)
print(coverage_report(design, FACTORS))
# -> {'risk': {'benign': 2, 'pii': 2, 'injection': 2}, 'channel': {'email': 3, 'chat': 3}}

Run the two demos

The headline capabilities — a validation report and live runtime monitoring — are runnable demos. Clone the repo (the demos aren't shipped in the wheel):

git clone https://github.com/knowlytix/beyond-ship-and-pray
cd beyond-ship-and-pray
pip install proofloop                       # pulls forgeloop from PyPI automatically
python demos/runtime_monitoring.py          # live ALLOW / DENY / ESCALATE + audit chain
python demos/validation_report.py           # DoE report -> demos/out/validation_report.html

Both run on the open baseline with no license. With the licensed knowlytix backend and a trained store, the same demos show the real geometric gate — see demos/README.md.

The GMS upgrade (open-core)

proofloop runs fully without a license. GMS-backed detection — geometric hallucination judging, factor attribution, signed verdicts — requires the licensed knowlytix package, imported lazily:

import forgeloop.gms as gms
gms.available()   # True if the licensed backend is installed

The production-grade, GMS-native edition is the Beyond Ship and Pray, Pro Edition — see knowlytix.ai.

License

Apache-2.0. © 2026 Knowlytix.