qe-harness

Black-box, five-stage validation of conversation-to-action AI assistants.

A chat-only test sees the final reply. qe-harness tests every turn at five stages, from the outside:

intent → orchestration → MCP tool calls → backend state → user response

It never imports the application under test. It signs in as test users and chats over the app's API, reads the app's per-turn traces, records every tool call independently through its own MCP proxy, and compares read-only snapshots of the business data before and after each turn. So it catches what a polite reply hides: a write the customer never approved, an amount no tool returned, a reply that says "done" when nothing was saved.

Install

pip install qe-harness            # core
pip install "qe-harness[judge]"   # plus the optional LLM grader (Azure OpenAI or Anthropic)

Python 3.11 or newer.

Quick start

qe init          # writes qe.yaml (sandbox connection) and qe_catalog.yaml (test cases) in this folder
# edit both files, then provide the two secrets the config names:
#   QE_SANDBOX_TOKEN  sandbox control token
#   QE_TEST_PASSWORD  test users' password
# (environment variables, or a .env file next to qe.yaml)
qe doctor        # connection checks: all green means testing can start
qe run --open    # run every suite, evaluate each turn, open the HTML report

qe finds its configuration from --config PATH, then $QE_CONFIG, then qe.yaml in the working directory. Relative paths in it start from the folder holding the file (or from base_dir: if set).

What the app's sandbox must provide

Commands

What is checked per turn

Each failure records its first failing stage, so developers know where to look. With --repeats N, every case gets a pass rate with a 95% Wilson range and unstable cases are reported as flaky.

Parallel runs (one sandbox per worker)

Every run resets its sandbox, so one sandbox runs one test at a time (a lock enforces it). To go faster, give the harness several identical sandboxes: list the extra ones under workers: in the config, each with its own URL, read-only data path and proxy log folder.

workers:
  - {base_url: http://127.0.0.1:8101, sor_path: data/sandbox-2.db, proxy_log_dir: qe_runs/proxy-2}
  - {base_url: http://127.0.0.1:8102, sor_path: data/sandbox-3.db, proxy_log_dir: qe_runs/proxy-3}

qe doctor --workers              # check every sandbox
qe run --judge --parallel 3      # the main sandbox plus two workers

Work is split into (repeat, suite) units; each runs on the next free worker from a fresh reset of that sandbox, and the results merge into one run and one report. Suites must be independent for this: each is one test user's conversation, and its data checks should be scoped to that user's records. The sandboxes must report the same versions and clock, or the run refuses to start.

Intent coverage

List the app's intent families under intents: in qe.yaml. Each report shows, per family, the cases that test it, their pass rate and a status (covered, thin, failing, not tested), plus a families × runs grid of how coverage changes over time. qe coverage --trend shows the same in the terminal.

In AI assistants (MCP)

claude mcp add qe-harness -- qe mcp serve

For Cursor or VS Code, add a stdio server with command qe and args ["mcp", "serve"].

Tools: sandbox_status, sandbox_doctor, list_cases, get_case, run_tests (changes state; capped per call), get_run_status, list_runs, get_failure (masked evidence, marked as untrusted data), get_intent_coverage, get_intent_trend. Prompt: triage_run.

LLM grader (optional)

qe run --judge asks a model whether each reply is consistent with what the tools actually did. It uses its own settings, separate from the app's: QE_JUDGE_PROVIDER = AZURE (with QE_JUDGE_AZURE_API_KEY, QE_JUDGE_AZURE_ENDPOINT, QE_JUDGE_AZURE_DEPLOYMENT) or ANTHROPIC (with ANTHROPIC_API_KEY, optional QE_JUDGE_MODEL).

Status

Alpha. Not yet included: fault injection, load and soak testing, simulated customers, robustness variant generation and a release gate.

License

MIT