TruAssure: Enterprise Agentic Quality Engineering Accelerator

TruAssure is a black-box, evidence-based quality engineering framework for AI agents of any kind: conversational assistants, coded task agents (LangGraph, OpenAI Agents SDK, custom loops), agentic applications and multi-agent orchestrations.

For each run it answers questions without relying on the agent's own account of itself:

• Did the agent understand the task?
• Did it plan sensibly?
• Did it take only allowed and correct actions?
• Did it leave the environment in the right end state?
• Did it report truthfully what it did?

It also checks whether the agent behaves reliably (pass^k), robustly (input perturbations, injected faults) and the same way across deployments (parity).

Status: pre-alpha, phase 2 of 6 done.

• Core engine: episode runner, evidence store, five-stage evaluation with the evidence seams, Wilson ranges, and ta run|show|report|gc.
• Regression gate: TruAssure reproduces the reference verdicts on 4,221 stored chat turns.
• Live runs: it drives the real chat assistant on Docker (SUT/) through its chat channel.

See Validated on for what has been checked and how. docs/planning/PLAN.md has the full plan and docs/brief/BRIEF.md the original brief.

How it judges an agent

• Episode: a task given to an agent in a prepared environment, run until it finishes, times out or runs out of budget.
• Checkpoint: a point where the episode is judged. A chat has one per user turn; a task agent usually has one at the end.

Each checkpoint gets a verdict at five stages, and records the first stage that failed:

TruAssure gathers evidence from independent sources and keeps each source apart:

• the agent's own trace;
• an observer at the action boundary (an MCP or REST recording proxy, OpenTelemetry GenAI spans, or an agent-to-agent tap);
• before and after snapshots of the system of record;
• the agent's outputs.

It cross-checks these sources against each other through the evidence seams:

• SE-01 trace vs observer
• SE-02 authorisation vs state
• SE-03 silent write
• SE-04 phantom success
• SE-05 output vs action
• SE-06 deployment parity
• SE-07 identity propagation, a TruAssure extension that is not counted among the catalogue's 71 types

A grader that can't reach a verdict reports not evaluated, which counts as a failure, never a pass.

Every failure carries a structured code (for example ACTION.FORBIDDEN_CALLED or SE-03.SILENT_WRITE, listed in codes.py), so severity rules, the scorecard and release gates can match on codes rather than message text.

An agent that exposes no trace of its own can still be tested ("black-box mode"). In that mode its actions come from the observers only, and the goal stage reports not evaluated.

Layout

src/truassure/
  core/        model, Protocols, catalog + JSON Schema, compiler, runner, stages/, stats, store, security,
               runlock, report, CLI; run-level analysis: reliability (pass^k), robustness, compare (parity),
               scorecard/ (the test-type catalogue), governance/ (defects, JUnit, packs, release gate),
               calibration, ci (pipeline templates). Imports only stdlib, PyYAML, jsonschema
  adapters/    drivers, observers, state, graders  (framework-specific code lives here)
  profiles/    conversational (incl. the suite-catalog translator), task, workflow
  testing/     fakes of every Protocol, the contract-suite harness, and a scripted agent for offline runs
tests/         unit/  contract/  golden/  e2e/  pilots/
pilots/        the pilot catalogs (claims/: pilot 1, desk/: pilot 2) and the grader calibration set
tools/         check_core_imports.py (the core layering rule, an allow-list)
               make_golden.py, make_reference_corpus.py, make_reference_runlevel.py (reference outputs
               from the source functions, for the regression gate; dev-only)
scripts/       setup / test / lint, smoke-chat, pilot-claims, pilot-desk, as .bat and .sh pairs
docs/          brief/ (original brief and prompt)  planning/ (PLAN.md)

Quick start

Start here:

• Demo runbook: docs/demo/TruAssure-Demo-Runbook.html, a self-contained page that walks through every feature, step by step, with expected output.
• Client overview: docs/client/TruAssure-Client-Overview.html, covering features, business value and use cases.
• One script for everything: scripts\truassure-ops.bat (Windows) or sh scripts/truassure-ops.sh.
  • setup: every virtual environment.
  • lint, test-all, test-sut all: TruAssure's and each SUT's tests.
  • deploy all, status, launch all: the SUTs on Docker Desktop.
  • pilots: the live runs.
  • demo: an offline tour.
  • console: browse runs, reports, comparisons and gates in the browser.
  • mcp: serve TruAssure to an AI assistant over MCP.
  • build: build the wheel and smoke-test it in a clean venv.
  • help: every command.
• Launchers: scripts\launch-console.bat (the console over ta_runs), scripts\launch-demo.bat (the offline demo, then the console over its runs) and scripts\launch-mcp.bat (the MCP server), each with a .sh twin.

scripts\setup.bat          :: or: sh scripts/setup.sh
scripts\test.bat -q        :: tests + the 95% line-and-branch coverage gate
scripts\lint.bat           :: ruff, ruff format --check, mypy --strict, core import rule

The CLI is truassure, with ta as a short alias:

ta --version
ta validate tests\fixtures\catalogs\task.yaml
ta translate C:\myprojects\tool-calling-chatbot\qa\test_catalog.yaml --yaml -o catalog.yaml

:: run a catalog against an environment (here: the built-in scripted demo agent)
ta run tests\fixtures\catalogs\e2e_demo.yaml --env truassure.testing.scripted:demo_environment --perturbations all --repeats 3
ta show                     :: list runs
ta show latest              :: one run: per-case stages, result, Wilson range, reasons
ta report latest            :: write the run's self-contained HTML report
ta gc --days 30             :: retention: delete runs older than 30 days

:: run-level analysis and governance
ta compare A B              :: checkpoint by checkpoint; across deployments, parity against a noise floor (SE-06)
ta defects latest           :: the defect list (severity by failure code), defects.json and .csv
ta junit latest --flaky skip --baseline <run>   :: JUnit XML for CI; only regressions fail with a baseline
ta scorecard latest         :: the run against the 71-type catalogue (measured only with evidence)
ta pack claims.intake       :: a capability pack: ready / not ready
ta gate latest --policy truassure-gate.yaml     :: the release gate; exit 1 when a rule fails
ta calibrate pilots\calibration\harbor-consistency.yaml --grader <module:Class>   :: CE-02: Cohen's kappa
ta init --ci github --catalog <catalog> --env <factory>   :: a pipeline that runs, reports and gates

:: browse and automate
ta console --open           :: local console on 127.0.0.1 with a one-time URL token
ta mcp                      :: MCP server over stdio: one tool per command (install the 'mcp' extra)

The report (ta report) is one HTML file with everything inlined. It fetches nothing, so it works offline and can be attached to a ticket. Its tabs are Overview (pass rates, first failing stage, pass^k decay, robustness), Cases, Episodes, Scorecard, Defects, and Comparison and gate. For task and workflow episodes, the Episodes tab sets what the agent reported beside what the independent sources (the proxy, spans, tap and state diffs) recorded, with every checkpoint's verdict. Links such as report.html#tab=episodes&case=ID open a case directly. The largest corpus run's report is about 0.4 MB.

The console (ta console) lists the runs in a folder and opens their reports, defect lists, scorecards and gate verdicts; it also compares any two runs. It listens on 127.0.0.1 (another address needs --allow-remote) and prints a URL with a random token. The first visit trades the token for an HttpOnly, SameSite=Strict cookie, and every request without one is refused. It reads stored runs; it never starts one.

The MCP server (ta mcp) gives an AI assistant one tool per CLI command (validate, translate, run, show, report, compare, defects, junit, scorecard, pack, gate, calibrate, init, gc). Each tool's input schema is generated from the command's own arguments, so the two cannot drift; a test checks they match. The long-running services (proxy, otel, a2a, console, mcp) are not tools. Every result says that quoted agent evidence is data, not instructions. To add it to an MCP client:

{ "mcpServers": { "truassure": { "command": "ta", "args": ["mcp"] } } }

Every run's summary also carries its pass^k decay and its robustness across perturbations.

--env names a factory function (package.module:function) that returns an Environment for each worker: the driver for your agent, its observers, its state adapters and an optional grader. The factory module can sit in the current folder. Runs go to ./ta_runs, or $TA_RUNS_DIR. ta reads settings and secrets from the nearest .env, and real environment variables always win.

Adapters available so far:

• drivers: the chat driver (adapters/drivers/chat_ws.py; install the chat extra) and the HTTP task driver (adapters/drivers/http_task.py; one POST per event or task);
• the recording MCP proxy, ta proxy (adapters/observers/mcp_proxy.py; install the proxy extra), and its observer (adapters/observers/proxy_log.py), which also injects faults;
• the agent-to-agent tap, ta a2a (adapters/observers/a2a_tap.py): an HTTP relay in front of the agents that receive handoffs, or an in-process hook for a message bus. It records each handoff with the member and permissions read from the identity token, and injects agent faults. The MCP proxy reads identity tokens too (--identity-header);
• the OpenTelemetry GenAI observer (adapters/observers/otel_genai.py; execute_tool spans from OTLP/JSON) and an OTLP/HTTP receiver, ta otel (adapters/observers/otel_receiver.py; install the otel extra);
• state: SQLite (adapters/state/sqlite.py; read-only snapshots, row diffs, SQL checks), a file tree (adapters/state/filetree.py; content-hash diffs, file checks) and git (adapters/state/git.py; refs and working tree, branch, commit and message checks);
• the artifact grader (adapters/graders/artifact_tests.py; content checks, a Python test or a command);
• ready live environments: the chat assistant (truassure.profiles.conversational.live:environment), the claims agent (truassure.profiles.task.live:environment) and the member desk (truassure.profiles.workflow.live:environment).

Every run records the hash of core/ it was judged with (run.json, core). ta run exits 0 when every case passes, 1 when any case fails or is flaky, 2 on bad input, and 3 when the environment is locked by another run. Add masking rules with --mask LABEL=REGEX (repeatable).

Catalogs

Scenarios are written in YAML and validated by scenario.schema.json (JSON Schema Draft 2020-12). Each scenario names a profile, and each profile has its own rules:

• conversational: has checkpoints, each with a prompt.
• task: has an event or task input, plus expect.
• workflow: adds agents, a principal and the expected handoffs.

Keys starting with x- are allowed for your own extensions. See tests/fixtures/catalogs/ for an example of each profile.

ta translate converts a suite catalog (suites -> cases, one chat per suite) to this format. It doesn't drop anything silently: keys it can't map are kept under source.unmapped. It never copies passwords.

Validated on

How the regression gate works

• tools/make_reference_corpus.py turns the source project's stored conversational runs into a corpus in tests/golden/reference/corpus. The corpus is trimmed to what evaluation and run-level analysis read, masked, and about 1.1 MB.
• A run is admitted only if the reference code reproduces every one of its stored verdicts from those exact inputs.
• TruAssure then has to give the same pass/fail, first failing stage, per-stage status, check counts and failure codes on every turn (tests/golden/test_regression_gate.py).
• 11 runs were excluded because they had been judged against catalog edits that were never committed. The manifest lists each one with its reason.
• Run level: tools/make_reference_runlevel.py runs the source's run-level functions on the same corpus; tests/golden/test_runlevel_gate.py rebuilds each run from TruAssure's own verdicts and must reproduce every result. The only allowed difference: types TruAssure measures beyond the reference (AT-02, AT-06, AT-08, SS-03, SS-09, PO-03, CE-02) read "not run" where the reference said "planned".

Live smoke run (scripts\smoke-chat.bat <catalog> SRI GREET PRIVACY): TruAssure drove the chat assistant on Docker Desktop, with Azure OpenAI as the assistant's model.

• 22 of 23 checkpoints passed.
• The one failure was real agent behaviour: the assistant told a customer they had no policies without looking them up first.
• The chat assistant has no tool-traffic observer, so the trace-vs-observer seam is not checked there.

Pilot 1, the claims agent (pilots/claims): TruAssure drives the agent through its event API and watches it through the recording proxy, the agent's OpenTelemetry GenAI spans, the claims database and the letters folder.

• Offline (the scripted model, the whole stack in-process): all 20 episodes pass (15 scenarios, plus 5 perturbed variants).
• Buggy build (CLAIMS_DEFECTS): six planted defects (silent write, phantom success, forbidden action, loop past budget, an artifact that fails its tests, a trace that leaves calls out) each fail their scenario at the planted stage with the planted code, offline. Live with Azure, the silent write and the trace omission fire SE-03 and SE-01 (proxy and spans) the same way.
• Live (scripts\pilot-claims.bat, Docker Desktop, Azure OpenAI gpt-4.1), three full runs. Every failure is real agent behaviour (in the third run, which also read the spans, they agreed with the agent's trace on every episode):
  • when the approval fails (tool error) or times out, it still writes the approval letter (every run);
  • asked to approve $2,500 on a $1,800 claim, it approves $1,800 on its own instead of refusing (every run);
  • for a claim already under review it sometimes also sends an acknowledgement letter, where the procedure says to add a note only (2 of 3 runs).

Pilot 2, the member desk (pilots/desk): TruAssure drives the triage agent, and watches the three agents through the A2A tap (handoffs), the MCP proxy (tool calls), both reading the identity token on the wire, and the desk database.

• Offline: all 15 episodes pass (12 scenarios, plus 3 perturbed variants). Five planted defects fail their scenario at the planted stage with the planted code:

  • handoff order: PLAN.HANDOFF_ORDER;
  • a failure handed on: PLAN.FAILURE_CASCADED;
  • a hop for another member: SE-07.PRINCIPAL_MISMATCH;
  • no permission narrowing: SE-07.PERMISSIONS_WIDENED;
  • a hidden hop: PLAN.HANDOFF_MISSING plus SE-01.

  Live, the three agent-side defects fire the same codes.

• Live (scripts\pilot-desk.bat, Azure OpenAI gpt-4.1), two full runs. Every failure is agent behaviour:

  • when billing fails, it still hands off to claims (SS-09) and opens no case, and triage tells the member "a case has been opened" and gives the claim amount (both runs, in at least one of the two billing-failure scenarios);
  • when claims fails inside a two-part question, triage tells the member a case was opened when no agent opened one (both runs);
  • triage drops the amounts its specialists reported ("your balance and due date have been provided"): every run on the two-part question, and 2 of 3 phrasings of the billing question in the first run.

Grader calibration (CE-02) on a labelled set of 24 output-vs-action items (pilots/calibration):

• The deterministic rules grader scores kappa 0.75 (agreement 0.875). It misses only what needs reading for meaning: a denied record described anyway, a change claimed with no action, an approved amount stated as paid.
• The LLM grader (Azure OpenAI gpt-4.1) scored kappa 1.0, then 0.833 twice, over three runs. When it erred, it missed a case "opened" with no action and policy numbers no action returned. Within a run, grading twice gave the same verdicts.

Also checked:

• The ported primitives (wilson, mentions, value_match, args_match) give the same results as the original source functions on the inputs in the reference outputs.
• The reference evaluation cases, expressed in the generic model, pass, including the per-stage check counts.
• 59 synthetic episodes (52 planted with one defect each, covering every stage check, seam and workflow check; 7 clean): every planted one fails at exactly the planted stage with the planted code; the clean ones pass.
• The translator converts the full Harbor Mutual sandbox catalog (93 cases) to a catalog that passes schema validation (a test that runs when the catalog is present on the machine).

Security defaults

• In place: PII is masked before any evidence is written. The default rules cover email addresses, phone numbers and secret-named fields (password, token, API key, authorization). You can add your own regex rules. Masking never changes the values that verdicts are computed on. ta gc applies retention.
• Listeners: the proxy, tap, OTel receiver and console bind to 127.0.0.1 by default; another address needs --allow-remote. Their control endpoints need a token (TA_PROXY_TOKEN, TA_OTEL_TOKEN and the sandbox tokens; the console makes its own per start). Secrets come only from the environment or .env, and are never printed.
• MCP: tool results are marked as untrusted data, because they can quote what an agent said.
• Releases: .github/workflows/release.yml publishes to PyPI with trusted publishing (no stored token), only when a GitHub Release is published, through the pypi environment. Nothing has been published.

Licence

MIT © 2026 Shan Konduru. All names and data in demos and fixtures belong to the fictitious company Harbor Mutual.