Skip to main content

🧭 Evidence-Driven QA

From API surfaces to test stories to traceable evidence.

Discover · Question · Specify · Index · Audit

PyPI Python License: MIT MCP Agent Skills Open Source

An open-source Python CLI, local MCP server and portable Agent Skill for systematic QA planning.

Quick start · The methodology · Agent setup · MCP setup · Documentation


✦ The idea

Great integration tests begin before writing test code. First map the actual API surface. For every operation ask: who may do this, who must be denied, what conditions and relationships must hold, what changes, and what observable result would prove it? Convert the answers into test stories, assign durable case IDs, and then inspect the existing test suite.

    OPENAPI / SOURCE CONTRACT
              │
              ▼
   ① DISCOVER TESTABLE SURFACE
      endpoints · actors · states
              │
              ▼
   ② ASK & ANSWER QUESTIONS
      allowed? forbidden? conditions?
      relationships? effects? proof?
              │
              ▼
   ③ DESIGN TEST STORIES
      positive · negative · edge cases
              │
              ▼
   ④ CREATE INDEXED CASES
      stable IDs · expected evidence
              │
              ▼
   ⑤ AUDIT EXISTING TESTS
      reference matching (not a PASS)
              │
              ▼
   ⑥ IMPLEMENT → EXECUTE → REPORT
         planned next stages

The design is agent-assisted, deterministic at the tool boundary. AI agents reason about semantics and ask for missing product decisions. The Python core extracts the surface, validates reviewed stories, generates consistent case IDs and searches existing test sources. It never guesses an authorization policy from OpenAPI or treats a matching test ID as proof of a passing test.

✦ Quick start

Install the package (Python 3.11+):

pip install evidence-driven-qa

Or run without a permanent installation using uvx:

uvx --from evidence-driven-qa qa-toolkit --help

1 · Discover the HTTP API surface from a saved local OpenAPI JSON file:

qa-toolkit surface --openapi openapi.json --json > qa-surface.json

This produces an editable per-endpoint questionnaire: declared routes, input parameters, response codes, available testing topics, and questions about intended behavior.

2 · Answer the contract questions and write test stories. Give qa-surface.json and the application's requirements to your agent. Ask it to complete the contract and stories fields using evidence from your project's specification, and to flag ambiguous requirements. Review its answers before compiling them.

3 · Create indexed test cases:

qa-toolkit cases --answers qa-surface.json > INDEXED_TEST_PLAN.md

Or use --json to produce a machine-readable registry. Generated IDs are deterministic for each method + path + story key and every new case starts as specified_unverified.

4 · Audit the tests that already exist:

qa-toolkit audit --plan INDEXED_TEST_PLAN.md --tests tests/ --json

The audit finds potential matches in pytest/Vitest/Playwright test source files. A reference is marked referenced_unverified, not covered or passed. Assertion-level review and executed test evidence are separate stages.

Try the independent synthetic example: example API contract, OpenAPI file, reviewed answers. These are fictitious fixtures, not production data.

qa-toolkit surface --openapi examples/synthetic-api/openapi.json
qa-toolkit cases --answers examples/synthetic-api/answers.json

✦ How it works

The workflow is endpoint-by-endpoint, but not merely endpoint-by-endpoint testing. Resource relationships, states, and actors matter as much as HTTP routes.

Step Questions the agent answers Output
Surface discovery What endpoints, verbs, parameters, responses and security declarations exist? OpenAPI inventory
Intended behavior What user/business capability should this operation provide? Requirement-backed contract
Allowed vs forbidden Who is permitted? Who must be denied, including anonymous and foreign owners? Actor/access matrix
Conditions & relationships Which resource IDs, ownership, tenant, parent/child relations and lifecycle states must hold? Test-data dimensions
Side effects & observability What changes or stays unchanged? How do we prove that from HTTP and persistent state? Concrete assertion targets
Stories & case IDs Which positive, forbidden, invalid, missing and transition scenarios expose a meaningful bug? Stable indexed cases
Audit & evidence Which existing tests actually assert these outcomes and which have run? References now; verified execution planned

More than security testing

Test stories can target authentication and authorization, input validation, business rules, resource isolation, information disclosure, pagination and filtering, async queues and timeouts, idempotency and retries, concurrency, data integrity, error contracts, rate limiting, exports, compatibility, accessibility and failure recovery. Select applicable dimensions—not a mechanically generated Cartesian product of every possible input.

For example, a negative authorization story may require checking both the rejected status and that the response reveals nothing about a foreign record. For an asynchronous job, the required proof may include a real queued state followed by an eventual completion in a disposable environment. See the complete methodology.

✦ Agent setup

The skill supplies a reusable workflow for discovering contract gaps and writing meaningful, indexed test stories. It can work with any agent that supports the Agent Skills standard.

Install to supported coding agents:

npx skills add darizae/qa-toolkit --skill evidence-driven-qa

Or install manually:

# Claude Code — user-level
mkdir -p ~/.claude/skills/evidence-driven-qa
cp skills/evidence-driven-qa/SKILL.md ~/.claude/skills/evidence-driven-qa/

# Codex / cross-agent project-level
mkdir -p .agents/skills/evidence-driven-qa
cp skills/evidence-driven-qa/SKILL.md .agents/skills/evidence-driven-qa/

Suggested first agent instruction:

Use the evidence-driven-qa skill. Discover the testable API surface, inspect the requirements for each endpoint, answer who is allowed or denied and under which conditions, then design positive/negative/edge test stories. Produce stable case IDs and audit existing tests. Call out any unresolved business rules; do not assume implementation behavior is the intended contract.

Model and coding-agent ecosystem

The toolkit does not depend on a particular language model. Via compatible coding agents or MCP-capable hosts, it is suitable for workflows using these common model families:

Model family Example agent/host
OpenAI GPT models (GPT-6, GPT-5 series, GPT-4o) Codex and other MCP-aware clients
Anthropic Claude (Opus, Sonnet, Haiku) Claude Code, Claude Desktop
Google Gemini (Pro, Flash) Gemini CLI and compatible agent hosts
Meta Llama, Qwen, Mistral, DeepSeek Local or hosted models through MCP-capable agent runtimes
xAI Grok MCP-capable agent integrations

MCP integration is provided by the host or agent runtime, not by model weights themselves. The generic skill can also be read as instructions by agents without MCP support.

✦ MCP server setup

The Python distribution also provides qa-toolkit-mcp, a local, read-only stdio server using the same core as the CLI. No hosted account, API key, or commercial license is required.

Claude Code

claude mcp add --transport stdio qa-toolkit -- \
  uvx --from evidence-driven-qa qa-toolkit-mcp --project /absolute/path/to/your/project

Codex CLI

codex mcp add qa-toolkit -- \
  uvx --from evidence-driven-qa qa-toolkit-mcp --project /absolute/path/to/your/project

Generic MCP stdio client configuration (replace the project path):

{
  "mcpServers": {
    "qa-toolkit": {
      "command": "uvx",
      "args": [
        "--from", "evidence-driven-qa",
        "qa-toolkit-mcp",
        "--project", "/absolute/path/to/your/project"
      ]
    }
  }
}

The MCP server exposes four read-only tools:

Tool Purpose
qa_capabilities List operations and current evidence limits
qa_discover_api_surface Load a local OpenAPI JSON file and return the questionnaire
qa_compile_indexed_cases Compile reviewed answers into indexed case records
qa_audit_case_references Match case IDs against test-source references

All paths are restricted to the selected project root. The MCP tools do not launch applications, run shell commands, modify files, access credentials, or connect to external services. The local stdio package is distinct from a hosted HTTPS ChatGPT app.

✦ Open-source by design

MIT-licensed. Free to use, study, modify, fork, redistribute and build upon—including commercial use under the terms of the license. The MCP server is part of the same open-source project; no subscription or paid account is needed.

Contributions welcome: bug reports, adapters, security improvements, documentation and reproducible examples. Open an issue before substantial API or schema changes.

This is an early release of the discovery and planning workflow. Test generation into project-native source, controlled test execution, browser evidence ingestion and HTML reports are on the roadmap. No current command claims to verify or certify test coverage.

✦ Documentation

Document Why it matters
QA methodology API questions, equivalence classes and story generation
Product contract Scope and architecture
Case & evidence schema Traceability and status semantics
Roadmap Features and GitHub issue tracking
MCP distribution Registry and free local server strategy
Release checklist Release hygiene
Flowstead case study Private real-world validation plan

Development:

python -m pip install -e '.[dev]'
python -m unittest discover -s tests -v
ruff check src tests

Provenance: All distributed code and examples in this repository were authored independently. The public package contains no reconstructed archived source, archived documentation, or private Flowstead fixtures.


Better questions → better test stories → better evidence.

GitHub · Issues · MIT License

Metadata

Release files for evidence-driven-qa 0.1.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for evidence-driven-qa 0.1.0
File Size Uploaded Provenance Actions
evidence_driven_qa-0.1.0.tar.gz 17.8 kB Without provenance Details for evidence_driven_qa-0.1.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for evidence-driven-qa 0.1.0
File Interpreter ABI Platform Actions
evidence_driven_qa-0.1.0-py3-none-any.whl Python 3 none any Details for evidence_driven_qa-0.1.0-py3-none-any.whl

Total release size: 35.5 kB

Release files / evidence_driven_qa-0.1.0.tar.gz

Download URL evidence_driven_qa-0.1.0.tar.gz
Size 17.8 kB
Tags Source
SHA-256 checksum
How to use checksums
7d29ceb9d4511a785ac4c7c9acfada09475b13ee683fbb0647d2b50b30f65df3
BLAKE2b-256 checksum
How to use checksums
50816b1e03c259d7e44596e21ba2b3e0259f86eec48667b2d79f24d240ef9f38
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.13.13

Release files / evidence_driven_qa-0.1.0-py3-none-any.whl

Download URL evidence_driven_qa-0.1.0-py3-none-any.whl
Size 17.6 kB
Tags Python 3
SHA-256 checksum
How to use checksums
0c68aae9022117f2b444a5e2f5454316ab61391002d113449168f763e674dbbe
BLAKE2b-256 checksum
How to use checksums
a359308d500891d8cc48e6bbe53988417b948db501cf720567b4a678f140027f
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.13.13

Security for evidence-driven-qa 0.1.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.1.0 This release

2 release files No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page