qe-harness

Black-box, five-stage validation of conversation-to-action AI assistants.

A chat-only test sees the final reply. qe-harness tests every turn at five stages, from the outside:

intent → orchestration → MCP tool calls → backend state → user response

It never imports the application under test. It signs in as test users and chats over the app's API, reads the app's per-turn traces, records every tool call independently through its own MCP proxy, and compares read-only snapshots of the business data before and after each turn. So it catches what a polite reply hides: a write the customer never approved, an amount no tool returned, a reply that says "done" when nothing was saved.

Install

pip install qe-harness            # core
pip install "qe-harness[judge]"   # plus the optional LLM grader (Azure OpenAI or Anthropic)

Python 3.11 or newer.

Quick start

qe init          # writes qe.yaml (sandbox connection) and qe_catalog.yaml (test cases) in this folder
# edit both files, then provide the two secrets the config names:
#   QE_SANDBOX_TOKEN  sandbox control token
#   QE_TEST_PASSWORD  test users' password
# (environment variables, or a .env file next to qe.yaml)
qe doctor        # connection checks: all green means testing can start
qe run --open    # run every suite, evaluate each turn, open the HTML report

qe finds its configuration from --config PATH, then $QE_CONFIG, then qe.yaml in the working directory. Relative paths in it start from the folder holding the file (or from base_dir: if set).

What the app's sandbox must provide

Commands

What is checked per turn

Each failure records its first failing stage, so developers know where to look. With --repeats N, every case gets a pass rate with a 95% Wilson range and unstable cases are reported as flaky.

Intent coverage

List the app's intent families under intents: in qe.yaml. Each report shows, per family, the cases that test it, their pass rate and a status (covered, thin, failing, not tested), plus a families × runs grid of how coverage changes over time. qe coverage --trend shows the same in the terminal.

In AI assistants (MCP)

claude mcp add qe-harness -- qe mcp serve

For Cursor or VS Code, add a stdio server with command qe and args ["mcp", "serve"].

Tools: sandbox_status, sandbox_doctor, list_cases, get_case, run_tests (changes state; capped per call), get_run_status, list_runs, get_failure (masked evidence, marked as untrusted data), get_intent_coverage, get_intent_trend. Prompt: triage_run.

LLM grader (optional)

qe run --judge asks a model whether each reply is consistent with what the tools actually did. It uses its own settings, separate from the app's: QE_JUDGE_PROVIDER = AZURE (with QE_JUDGE_AZURE_API_KEY, QE_JUDGE_AZURE_ENDPOINT, QE_JUDGE_AZURE_DEPLOYMENT) or ANTHROPIC (with ANTHROPIC_API_KEY, optional QE_JUDGE_MODEL).

Status

Alpha. Not yet included: fault injection, load and soak testing, simulated customers, robustness variant generation and a release gate.

License

MIT