qe-harness

Black-box, five-stage validation of conversation-to-action AI assistants.

A chat-only test sees the final reply. qe-harness tests every turn at five stages, from the outside:

intent → orchestration → MCP tool calls → backend state → user response

It never imports the application under test, and it doesn't intercept anything. It signs in as test users and chats over the app's API, reads the app's per-turn traces, gets an independent record of every tool call from a recording proxy the sandbox routes its tool traffic through (stdio or HTTP), and compares read-only snapshots of the business data before and after each turn (see "How the evidence is collected" below). So it catches what a polite reply hides: a write the customer never approved, an amount no tool returned, a reply that says "done" when nothing was saved.

Install

pip install qe-harness            # core
pip install "qe-harness[judge]"   # plus the optional LLM grader (Azure OpenAI or Anthropic)

Python 3.11 or newer.

Quick start

qe init          # writes qe.yaml (sandbox connection) and qe_catalog.yaml (test cases) in this folder
# edit both files, then provide the two secrets the config names:
#   QE_SANDBOX_TOKEN  sandbox control token
#   QE_TEST_PASSWORD  test users' password
# (environment variables, or a .env file next to qe.yaml)
qe doctor        # connection checks: all green means testing can start
qe run --open    # run every suite, evaluate each turn, open the HTML report

qe finds its configuration from --config PATH, then $QE_CONFIG, then qe.yaml in the working directory. Relative paths in it start from the folder holding the file (or from base_dir: if set).

What the app's sandbox must provide

How the evidence is collected (nothing is intercepted)

The harness doesn't sniff network traffic, decrypt TLS or hook into the app. Each source of evidence is something the client switches on in their sandbox:

                     the client's sandbox (each source is switched on by the client)
  +------------+                    +-------------+   MCP    +-------------+   MCP   +-------------+
  |            | (1) chat as test   |             | -------> |  qe proxy   | ------> |             |
  |            |     users          |  app under  |          |  relays     |         | tool server |
  |            | -----------------> |  test       | <------- |  unchanged, | <------ |             |
  |            | <----------------- |             |          |  logs each  |         +------+------+
  | qe-harness |                    +------+------+          |  tool call  |                |
  |            | (2) per-turn trace API    |                 +------+------+                v
  |            | <-------------------------+                        |                +-------------+
  |            | (3) tool-call log, independent of the app's trace  |                |  business   |
  |            | <--------------------------------------------------+                |  data       |
  |            | (4) read-only snapshots of a replica, before and after every turn   |  (replica)  |
  |            | <-------------------------------------------------------------------+-------------+
  +------------+

1. Conversation: the harness signs in as test users and chats over the app's normal API, like a customer would.
2. Intent, plan and tool calls: the app reports them per turn through its trace API (the app describing itself).
3. Independent tool-call record: the app's tool traffic is routed through qe_harness.mcp_proxy by changing one setting in the sandbox. The proxy passes every MCP message through unchanged and writes each tools/call request and its result to a log. If the app's trace leaves out or misreports a tool call, this record disagrees, and the harness flags it ("trace and proxy disagree").
4. Business data: read-only snapshots of a replica of the system of record, before and after every turn.

Enabling the proxy

• stdio: one proxy process per tool-server process; the customer comes from COPILOT_CUSTOMER_ID in its environment.
• HTTP: a reverse proxy. Requests keep their path, headers (Mcp-Session-Id, Authorization, …) and body; event-stream responses are streamed through as they arrive. The customer comes from a request header the app sends (--customer-header, default X-Customer-Id) or a fixed --customer. Each request/response exchange is logged with its own id, so concurrent conversations can't be mixed up.
• Both modes write the same log format to <dir>, which must match tools.proxy_log_dir in the harness config.
• The proxy logs only tool calls and their results. It changes nothing, and if it's removed the app works exactly as before.

Other ways to get an independent record

If routing tool traffic through the proxy isn't possible, these give similar evidence, roughly from most to least recommended:

• API gateway or service-mesh access logs (Kong, Apigee, AWS API Gateway, Istio/Envoy) for the back-end calls the tools make: very independent of the app.
• Database change capture or audit logs (e.g. Debezium, database audit triggers): an independent record of every write; scales better than whole-database snapshots.
• OpenTelemetry traces the app already emits for model and tool calls: useful, but like the trace API it's the app describing itself.
• An audit log inside the tool server: independent of the agent, not of the client's code.
• Packet capture (tcpdump, mitmproxy with a trusted CA): a last resort. Traffic is usually TLS, it needs network access and the security team's consent, and it sees nothing at all when MCP runs over stdio.

Commands

What is checked per turn

Each failure records its first failing stage, so developers know where to look. With --repeats N, every case gets a pass rate with a 95% Wilson range and unstable cases are reported as flaky.

Parallel runs (one sandbox per worker)

Every run resets its sandbox, so one sandbox runs one test at a time (a lock enforces it). To go faster, give the harness several identical sandboxes: list the extra ones under workers: in the config, each with its own URL, read-only data path and proxy log folder.

workers:
  - {base_url: http://127.0.0.1:8101, sor_path: data/sandbox-2.db, proxy_log_dir: qe_runs/proxy-2}
  - {base_url: http://127.0.0.1:8102, sor_path: data/sandbox-3.db, proxy_log_dir: qe_runs/proxy-3}

qe doctor --workers              # check every sandbox
qe run --judge --parallel 3      # the main sandbox plus two workers

Work is split into (repeat, suite) units; each runs on the next free worker from a fresh reset of that sandbox, and the results merge into one run and one report. Suites must be independent for this: each is one test user's conversation, and its data checks should be scoped to that user's records. The sandboxes must report the same versions and clock, or the run refuses to start.

Intent coverage

List the app's intent families under intents: in qe.yaml. Each report shows, per family, the cases that test it, their pass rate and a status (covered, thin, failing, not tested), plus a families × runs grid of how coverage changes over time. qe coverage --trend shows the same in the terminal, as many recent runs as fit the window (--last N or --all to choose); in the report the grid keeps every run and scrolls sideways.

Defect list for the app team

Every report comes with a defect list next to it (defects.html and defects.csv, linked from the report header); qe defects [RUN] --open rebuilds and opens it. One entry per case that failed, cases that fail every time first, then flaky ones. Each entry has:

• what the customer said, what the catalog expected and why, and what went wrong at each stage;
• the app's reply and what it did (intent, tool calls, approval prompts it shouldn't have shown, data changed);
• how often it failed, in this run and across all earlier runs;
• how to reproduce it (which test user, which earlier messages of the conversation);
• a link to the evidence in the report (report.html#case=ID&repeat=N opens that case and try).

Severity comes from what went wrong: critical when the app did or tried to do something it shouldn't (a declined stray approval, a forbidden call, an unexpected data change, a trace that disagrees with the proxy); major when it didn't do what was asked, or the grader found the reply contradicts the tools; minor when the action was right but the reply missed something, or the request was misclassified. A catalog case can override it with severity: critical | major | minor. The CSV imports into Jira, Azure Boards or a spreadsheet.

In CI: JUnit, baselines and nightly pipelines

qe run --repeats 3 --judge --junit qe-results.xml --flaky skip --baseline qe-baseline.json

• --junit FILE writes JUnit XML: one test suite per catalog suite, one test case per case, the failure message is the first thing that went wrong, with a link to the evidence.
• --flaky fail|skip|pass decides how a case that failed only some tries counts (default fail).
• --baseline RUN|latest|summary.json fails the build only on regressions: cases that passed every try in the accepted baseline (or are new) and don't now. Cases that already failed there are reported as skipped ("known failure"), so they stay visible without turning every build red. Accept a baseline by committing a run's summary.json.
• The exit code is 0 when nothing counts as failed, 1 when something does, 2 when the run couldn't start.
• qe junit [RUN] --out FILE does the same for a run that has already finished.

qe init --ci github writes a nightly GitHub Actions workflow (.github/workflows/qe-nightly.yml) and qe init --ci azure an Azure DevOps pipeline (azure-pipelines-qe.yml). Both install qe-harness, wait for the sandbox with qe doctor, run the suites with a baseline, publish the JUnit results and keep the report and defect list as build artifacts. Add the secrets listed at the top of the file; use a self-hosted runner or agent if the sandbox is only reachable inside your network.

In AI assistants (MCP)

claude mcp add qe-harness -- qe mcp serve

For Cursor or VS Code, add a stdio server with command qe and args ["mcp", "serve"].

Tools: sandbox_status, sandbox_doctor, list_cases, get_case, run_tests (changes state; capped per call), get_run_status, list_runs, get_failure (masked evidence, marked as untrusted data), get_intent_coverage, get_intent_trend, get_defects (the defect list, filterable by severity). Prompt: triage_run.

LLM grader (optional)

qe run --judge asks a model whether each reply is consistent with what the tools actually did. It uses its own settings, separate from the app's: QE_JUDGE_PROVIDER = AZURE (with QE_JUDGE_AZURE_API_KEY, QE_JUDGE_AZURE_ENDPOINT, QE_JUDGE_AZURE_DEPLOYMENT) or ANTHROPIC (with ANTHROPIC_API_KEY, optional QE_JUDGE_MODEL).

Status

Alpha. Not yet included: fault injection, load and soak testing, simulated customers and robustness variant generation. The CI baseline is a regression gate; a gate on absolute thresholds (for example "no critical defects") is not built in yet.

License

MIT