🧭 Evidence-Driven QA
From API surfaces to test stories to traceable evidence.
Discover · Question · Specify · Index · Audit
An open-source Python CLI, local MCP server and portable Agent Skill for systematic QA planning.
Quick start · The methodology · Agent setup · MCP setup · Documentation
✦ The idea
Great integration tests begin before writing test code. First map the actual API surface. For every operation ask: who may do this, who must be denied, what conditions and relationships must hold, what changes, and what observable result would prove it? Convert the answers into test stories, assign durable case IDs, and then inspect the existing test suite.
OPENAPI / SOURCE CONTRACT
│
▼
① DISCOVER TESTABLE SURFACE
endpoints · actors · states
│
▼
② ASK & ANSWER QUESTIONS
allowed? forbidden? conditions?
relationships? effects? proof?
│
▼
③ DESIGN TEST STORIES
positive · negative · edge cases
│
▼
④ CREATE INDEXED CASES
stable IDs · expected evidence
│
▼
⑤ AUDIT EXISTING TESTS
reference matching (not a PASS)
│
▼
⑥ IMPLEMENT → EXECUTE → REPORT
planned next stages
The design is agent-assisted, deterministic at the tool boundary. AI agents reason about semantics and ask for missing product decisions. The Python core extracts the surface, validates reviewed stories, generates consistent case IDs and searches existing test sources. It never guesses an authorization policy from OpenAPI or treats a matching test ID as proof of a passing test.
✦ Quick start
Install the package (Python 3.11+):
pip install evidence-driven-qa
Or run without a permanent installation using uvx:
uvx --from evidence-driven-qa qa-toolkit --help
1 · Discover the HTTP API surface from a saved local OpenAPI JSON file:
qa-toolkit surface --openapi openapi.json --json > qa-surface.json
This produces an editable per-endpoint questionnaire: declared routes, input parameters, response codes, available testing topics, and questions about intended behavior.
2 · Answer the contract questions and write test stories. Give qa-surface.json and the application's requirements to your agent. Ask it to complete the contract and stories fields using evidence from your project's specification, and to flag ambiguous requirements. Review its answers before compiling them.
3 · Create indexed test cases:
qa-toolkit cases --answers qa-surface.json > INDEXED_TEST_PLAN.md
Or use --json to produce a machine-readable registry. Generated IDs are deterministic for each method + path + story key and every new case starts as specified_unverified.
4 · Audit the tests that already exist:
qa-toolkit audit --plan INDEXED_TEST_PLAN.md --tests tests/ --json
The audit finds potential matches in pytest/Vitest/Playwright test source files. A reference is marked referenced_unverified, not covered or passed. Assertion-level review and executed test evidence are separate stages.
Try the independent synthetic example: example API contract, OpenAPI file, reviewed answers. These are fictitious fixtures, not production data.
qa-toolkit surface --openapi examples/synthetic-api/openapi.json
qa-toolkit cases --answers examples/synthetic-api/answers.json
✦ How it works
The workflow is endpoint-by-endpoint, but not merely endpoint-by-endpoint testing. Resource relationships, states, and actors matter as much as HTTP routes.
| Step | Questions the agent answers | Output |
|---|---|---|
| Surface discovery | What endpoints, verbs, parameters, responses and security declarations exist? | OpenAPI inventory |
| Intended behavior | What user/business capability should this operation provide? | Requirement-backed contract |
| Allowed vs forbidden | Who is permitted? Who must be denied, including anonymous and foreign owners? | Actor/access matrix |
| Conditions & relationships | Which resource IDs, ownership, tenant, parent/child relations and lifecycle states must hold? | Test-data dimensions |
| Side effects & observability | What changes or stays unchanged? How do we prove that from HTTP and persistent state? | Concrete assertion targets |
| Stories & case IDs | Which positive, forbidden, invalid, missing and transition scenarios expose a meaningful bug? | Stable indexed cases |
| Audit & evidence | Which existing tests actually assert these outcomes and which have run? | References now; verified execution planned |
More than security testing
Test stories can target authentication and authorization, input validation, business rules, resource isolation, information disclosure, pagination and filtering, async queues and timeouts, idempotency and retries, concurrency, data integrity, error contracts, rate limiting, exports, compatibility, accessibility and failure recovery. Select applicable dimensions—not a mechanically generated Cartesian product of every possible input.
For example, a negative authorization story may require checking both the rejected status and that the response reveals nothing about a foreign record. For an asynchronous job, the required proof may include a real queued state followed by an eventual completion in a disposable environment. See the complete methodology.
✦ Agent setup
The skill supplies a reusable workflow for discovering contract gaps and writing meaningful, indexed test stories. It can work with any agent that supports the Agent Skills standard.
Install to supported coding agents:
npx skills add darizae/qa-toolkit --skill evidence-driven-qa
Or install manually:
# Claude Code — user-level
mkdir -p ~/.claude/skills/evidence-driven-qa
cp skills/evidence-driven-qa/SKILL.md ~/.claude/skills/evidence-driven-qa/
# Codex / cross-agent project-level
mkdir -p .agents/skills/evidence-driven-qa
cp skills/evidence-driven-qa/SKILL.md .agents/skills/evidence-driven-qa/
Suggested first agent instruction:
Use the evidence-driven-qa skill. Discover the testable API surface, inspect the requirements for each endpoint, answer who is allowed or denied and under which conditions, then design positive/negative/edge test stories. Produce stable case IDs and audit existing tests. Call out any unresolved business rules; do not assume implementation behavior is the intended contract.
Model and coding-agent ecosystem
The toolkit does not depend on a particular language model. Via compatible coding agents or MCP-capable hosts, it is suitable for workflows using these common model families:
| Model family | Example agent/host |
|---|---|
| OpenAI GPT models (GPT-6, GPT-5 series, GPT-4o) | Codex and other MCP-aware clients |
| Anthropic Claude (Opus, Sonnet, Haiku) | Claude Code, Claude Desktop |
| Google Gemini (Pro, Flash) | Gemini CLI and compatible agent hosts |
| Meta Llama, Qwen, Mistral, DeepSeek | Local or hosted models through MCP-capable agent runtimes |
| xAI Grok | MCP-capable agent integrations |
MCP integration is provided by the host or agent runtime, not by model weights themselves. The generic skill can also be read as instructions by agents without MCP support.
✦ MCP server setup
The Python distribution also provides qa-toolkit-mcp, a local, read-only stdio server using the same core as the CLI. No hosted account, API key, or commercial license is required.
Claude Code
claude mcp add --transport stdio qa-toolkit -- \
uvx --from evidence-driven-qa qa-toolkit-mcp --project /absolute/path/to/your/project
Codex CLI
codex mcp add qa-toolkit -- \
uvx --from evidence-driven-qa qa-toolkit-mcp --project /absolute/path/to/your/project
Generic MCP stdio client configuration (replace the project path):
{
"mcpServers": {
"qa-toolkit": {
"command": "uvx",
"args": [
"--from", "evidence-driven-qa",
"qa-toolkit-mcp",
"--project", "/absolute/path/to/your/project"
]
}
}
}
The MCP server exposes four read-only tools:
| Tool | Purpose |
|---|---|
qa_capabilities |
List operations and current evidence limits |
qa_discover_api_surface |
Load a local OpenAPI JSON file and return the questionnaire |
qa_compile_indexed_cases |
Compile reviewed answers into indexed case records |
qa_audit_case_references |
Match case IDs against test-source references |
All paths are restricted to the selected project root. The MCP tools do not launch applications, run shell commands, modify files, access credentials, or connect to external services. The local stdio package is distinct from a hosted HTTPS ChatGPT app.
✦ Open-source by design
MIT-licensed. Free to use, study, modify, fork, redistribute and build upon—including commercial use under the terms of the license. The MCP server is part of the same open-source project; no subscription or paid account is needed.
Contributions welcome: bug reports, adapters, security improvements, documentation and reproducible examples. Open an issue before substantial API or schema changes.
This is an early release of the discovery and planning workflow. Test generation into project-native source, controlled test execution, browser evidence ingestion and HTML reports are on the roadmap. No current command claims to verify or certify test coverage.
✦ Documentation
| Document | Why it matters |
|---|---|
| QA methodology | API questions, equivalence classes and story generation |
| Product contract | Scope and architecture |
| Case & evidence schema | Traceability and status semantics |
| Roadmap | Features and GitHub issue tracking |
| MCP distribution | Registry and free local server strategy |
| Release checklist | Release hygiene |
| Flowstead case study | Private real-world validation plan |
Development:
python -m pip install -e '.[dev]'
python -m unittest discover -s tests -v
ruff check src tests
Provenance: All distributed code and examples in this repository were authored independently. The public package contains no reconstructed archived source, archived documentation, or private Flowstead fixtures.
Better questions → better test stories → better evidence.
GitHub · Issues · MIT License
Metadata
Release files for evidence-driven-qa 0.1.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| evidence_driven_qa-0.1.0.tar.gz | 17.8 kB | Without provenance | Details for evidence_driven_qa-0.1.0.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| evidence_driven_qa-0.1.0-py3-none-any.whl | Python 3 | none | any | Details for evidence_driven_qa-0.1.0-py3-none-any.whl |
Total release size: 35.5 kB
Release files / evidence_driven_qa-0.1.0.tar.gz
| Download URL | evidence_driven_qa-0.1.0.tar.gz |
|---|---|
| Size | 17.8 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
7d29ceb9d4511a785ac4c7c9acfada09475b13ee683fbb0647d2b50b30f65df3
|
|
BLAKE2b-256 checksum How to use checksums |
50816b1e03c259d7e44596e21ba2b3e0259f86eec48667b2d79f24d240ef9f38
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.13.13
|
Release files / evidence_driven_qa-0.1.0-py3-none-any.whl
| Download URL | evidence_driven_qa-0.1.0-py3-none-any.whl |
|---|---|
| Size | 17.6 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
0c68aae9022117f2b444a5e2f5454316ab61391002d113449168f763e674dbbe
|
|
BLAKE2b-256 checksum How to use checksums |
a359308d500891d8cc48e6bbe53988417b948db501cf720567b4a678f140027f
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.13.13
|
Security for evidence-driven-qa 0.1.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.