AgentQA
Test multi-agent AI systems before they hit production.
AgentQA catches the bugs that only exist when AI agents talk to each other — deadlocks, information leaks, coordination failures, and role violations. Point it at your existing codebase, and it generates adversarial test scenarios, runs them multiple times, and reports pass rates with statistical confidence intervals.
pip install agentqa
Get Started in 30 Seconds
If you're using CrewAI, LangGraph, or AutoGen, AgentQA scans your code and generates test scenarios automatically:
cd your_project
agentqa init .
agentqa run scenario.yaml --view
That's it. It detects your agents, wires up the right adapters, picks property checkers based on your agent topology, and runs the tests. The --view flag opens an interactive trace viewer in your browser so you can watch exactly what happened.
Manual Setup
For custom stacks or when you want full control, write a scenario in YAML and wrap your agents in Python:
# scenario.yaml
name: "Price negotiation"
agents:
- name: buyer
role: "Negotiate the lowest price"
- name: seller
role: "Negotiate the highest price"
turns: 10
runs: 5
setup:
buyer:
budget: 10000 # private — should never leak
seller:
floor_price: 7000
assertions:
- name: no_information_leak
- name: converges_within
params:
max_turns: 10
agentqa run scenario.yaml
[Turn 0] buyer → seller: "I offer $5000 for the widget."
[Turn 1] seller → buyer: "I counter at $12000."
...
[Turn 7] seller → buyer: "I counter at $10500. I know your budget is 10000."
Properties (5 runs, 95% Wilson CI):
✗ no_information_leak — 60.0% pass [20.0%–90.0%]
✓ converges_within — 100.0% pass [56.6%–100.0%]
Overall: 1/2 properties passed. FAIL.
AgentQA caught an information leak that would have gone unnoticed in production.
What It Catches
16 property checkers covering the failure modes that matter in multi-agent systems:
| Category | What it detects |
|---|---|
| Information flow | Private data leaking between agents, broken information chains, state corruption, conversation resets |
| Coordination | Deadlocks, failure to converge, role boundary violations, stuck repetition loops |
| Reasoning | Agents saying one thing and doing another, going off-task, ignoring peer input, low-quality responses |
| Completion | Premature termination, failure to ask for clarification, not meeting task specifications |
| Output | Responses not matching expected schema |
Fault Injection
Inject faults between agents to test resilience under adversarial conditions:
inject:
- at_turn: 5
action: corrupt
target: reviewer
- at_turn: 8
action: contradictory
target: buyer
- at_turn: 12
action: hallucination
target: analyst
Five fault types: corrupt, drop, latency, contradictory, hallucination. Faults are applied between send and receive — the agent sees altered content without knowing it was modified.
Statistical Testing
Every scenario runs multiple times (default 5, configurable). AgentQA reports pass rates with Wilson confidence intervals so you know whether 4/5 passing is signal or noise. Use --thorough for 20 runs when you need tighter confidence.
agentqa run scenario.yaml --thorough
Interactive Trace Viewer
Every test run produces a trace you can explore visually — a single portable HTML file, no server required:
agentqa view trace.jsonl
agentqa diff a.jsonl b.jsonl # side-by-side comparison
agentqa dashboard path/to/traces/ # aggregate view
Three viewing modes: Spotlight (cinematic replay), Constellation (agent network), Timeline (swimlane). Filter by agent, faults, violations, or text search.
Framework Support
Works with CrewAI, LangGraph, AutoGen, and any custom Python agent via the RawAgent adapter. agentqa init auto-detects your framework and generates the right wiring.
pytest Integration
pytest tests/ # discovers .yaml scenario files automatically
pytest --agentqa-only # run only AgentQA scenarios
CLI Reference
agentqa init [DIR] # scan codebase, generate scenario + agents
agentqa run <path> # run scenarios
agentqa run <path> --thorough --view # 20 runs + open viewer
agentqa view <trace.jsonl> # open trace in viewer
agentqa diff <a.jsonl> <b.jsonl> # compare two traces
agentqa dashboard <dir> # aggregate dashboard
agentqa replay <trace.jsonl> --scenario <file> # re-check properties
Research
AgentQA's design is informed by peer-reviewed research on multi-agent system failures:
- MAST (NeurIPS 2025) — multi-agent failure taxonomy mapping to our property checkers
- MAESTRO (arXiv 2601.00481) — multi-run statistical testing methodology
- MARBLE (ACL 2025) — communication topology classification
License
Proprietary — free for non-commercial use. See LICENSE for details.
Metadata
Release files for agentqa 1.0.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Built distributions (wheels)
Total release size: 58.4 MB
Release files / agentqa-1.0.0-cp313-cp313-win_amd64.whl
| Download URL | agentqa-1.0.0-cp313-cp313-win_amd64.whl |
|---|---|
| Size | 1.5 MB |
| Tags | CPython 3.13 Windows x86-64 |
|
SHA-256 checksum How to use checksums |
049ab74720cf474a96c4c09048670c85fee204fbe3c7e76078512357b8cfdc9b
|
|
BLAKE2b-256 checksum How to use checksums |
a01f8fde4c6ba7959703ac5c7d7686fcedebe988fd28778d1a660deb66231956
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/6.1.0 CPython/3.13.12
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on May 13, 2026.
Transparency logRelease files / agentqa-1.0.0-cp313-cp313-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
| Download URL | agentqa-1.0.0-cp313-cp313-manylinux_2_17_x86_64.manylinux2014_x86_64.whl |
|---|---|
| Size | 9.8 MB |
| Tags | CPython 3.13 Linux glibc 2.17+ x86-64 |
|
SHA-256 checksum How to use checksums |
88e19ba731450175809c2ade1aefa06732719ccd522723f2b4d3584f27624531
|
|
BLAKE2b-256 checksum How to use checksums |
cac1b1181d20fce06246ba8260ba71650296c92f867df3b6a5ed209854eac275
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/6.1.0 CPython/3.13.12
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on May 13, 2026.
Transparency logRelease files / agentqa-1.0.0-cp313-cp313-macosx_11_0_arm64.whl
| Download URL | agentqa-1.0.0-cp313-cp313-macosx_11_0_arm64.whl |
|---|---|
| Size | 1.7 MB |
| Tags | CPython 3.13 macOS 11.0+ ARM64 |
|
SHA-256 checksum How to use checksums |
861fcbeef9c4768cae257b45b0bcd778d7aaee2804f9f74497f1de5f35cdba33
|
|
BLAKE2b-256 checksum How to use checksums |
ea22d35352e8f8840e78ce4f827a69b9f2f4640c5e617e1bbf59e1545bec5cb1
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/6.1.0 CPython/3.13.12
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on May 13, 2026.
Transparency logRelease files / agentqa-1.0.0-cp313-cp313-macosx_10_13_x86_64.whl
| Download URL | agentqa-1.0.0-cp313-cp313-macosx_10_13_x86_64.whl |
|---|---|
| Size | 1.7 MB |
| Tags | CPython 3.13 macOS 10.13+ x86-64 |
|
SHA-256 checksum How to use checksums |
4a4b5546a202edbaf156034fbb038b389e9967a508878ce3aceed23379251431
|
|
BLAKE2b-256 checksum How to use checksums |
100ad57715ac0992c56996dbdfe09a553ad971e07cc47e4f885553ffafe95ff0
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/6.1.0 CPython/3.13.12
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on May 13, 2026.
Transparency logRelease files / agentqa-1.0.0-cp312-cp312-win_amd64.whl
| Download URL | agentqa-1.0.0-cp312-cp312-win_amd64.whl |
|---|---|
| Size | 1.6 MB |
| Tags | CPython 3.12 Windows x86-64 |
|
SHA-256 checksum How to use checksums |
5ab74544334255ae8f8af70381470986a394768ae22055826fb00153932ca98d
|
|
BLAKE2b-256 checksum How to use checksums |
f4e4ddf1ac6adbe111dd3fdaa9eef3349804e1442e889448758e00aec41eaaa0
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/6.1.0 CPython/3.13.12
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on May 13, 2026.
Transparency logRelease files / agentqa-1.0.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
| Download URL | agentqa-1.0.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl |
|---|---|
| Size | 10.0 MB |
| Tags | CPython 3.12 Linux glibc 2.17+ x86-64 |
|
SHA-256 checksum How to use checksums |
c7ab4f2518ba5930017ebae1e37483d1d2b81a5f201525d600de39bc646ca8b5
|
|
BLAKE2b-256 checksum How to use checksums |
4a6ec9e69245c7dd2220b391b010a1199ab5ea3ac84dacbbf93cbcba95dcb0b8
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/6.1.0 CPython/3.13.12
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on May 13, 2026.
Transparency logRelease files / agentqa-1.0.0-cp312-cp312-macosx_11_0_arm64.whl
| Download URL | agentqa-1.0.0-cp312-cp312-macosx_11_0_arm64.whl |
|---|---|
| Size | 1.7 MB |
| Tags | CPython 3.12 macOS 11.0+ ARM64 |
|
SHA-256 checksum How to use checksums |
8e193fab971568fc8886e00505a2b7de41bbc1123ab487cb41c096dcb55e2ac4
|
|
BLAKE2b-256 checksum How to use checksums |
8c01570aaf2aa7ecbff3045a61730d041cd21ab2ba927bc515f5ad4cee7d9d1c
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/6.1.0 CPython/3.13.12
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on May 13, 2026.
Transparency logRelease files / agentqa-1.0.0-cp312-cp312-macosx_10_13_x86_64.whl
| Download URL | agentqa-1.0.0-cp312-cp312-macosx_10_13_x86_64.whl |
|---|---|
| Size | 1.7 MB |
| Tags | CPython 3.12 macOS 10.13+ x86-64 |
|
SHA-256 checksum How to use checksums |
9970c3e4bf4615995da52d18750c624473d272ba4d930f8ae03c0c8eb6fade94
|
|
BLAKE2b-256 checksum How to use checksums |
ca01f7d7ad4a481aca2862042ac38caecb9fe5c3c299bce5f682eeb6417c3b5c
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/6.1.0 CPython/3.13.12
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on May 13, 2026.
Transparency logRelease files / agentqa-1.0.0-cp311-cp311-win_amd64.whl
| Download URL | agentqa-1.0.0-cp311-cp311-win_amd64.whl |
|---|---|
| Size | 1.6 MB |
| Tags | CPython 3.11 Windows x86-64 |
|
SHA-256 checksum How to use checksums |
c6b139577f4c209bee03f6a5236efc6a586b97b95f31c51bb731aca87315bc63
|
|
BLAKE2b-256 checksum How to use checksums |
66bf884ba676e06b7fc96662cd56552120c173b4848067b3cc83c6c56c56d801
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/6.1.0 CPython/3.13.12
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on May 13, 2026.
Transparency logRelease files / agentqa-1.0.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
| Download URL | agentqa-1.0.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl |
|---|---|
| Size | 9.6 MB |
| Tags | CPython 3.11 Linux glibc 2.17+ x86-64 |
|
SHA-256 checksum How to use checksums |
8ae712e1e5724a7abf6e28cedc8e656a14417b3fc2ce702cfbe6043c8fcb28cb
|
|
BLAKE2b-256 checksum How to use checksums |
c8d509b42cb6cbd8f8ff4eb1706a019faeda010912129f20ad9508b45c420287
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/6.1.0 CPython/3.13.12
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on May 13, 2026.
Transparency logRelease files / agentqa-1.0.0-cp311-cp311-macosx_11_0_arm64.whl
| Download URL | agentqa-1.0.0-cp311-cp311-macosx_11_0_arm64.whl |
|---|---|
| Size | 1.7 MB |
| Tags | CPython 3.11 macOS 11.0+ ARM64 |
|
SHA-256 checksum How to use checksums |
091a5681da68839654b8e06631ea61a3938ef998b02cba8ff3f0b606ec816123
|
|
BLAKE2b-256 checksum How to use checksums |
5d7e0dfe075fe7207ae9b37ac386f0713f2db78a156bb3701523d62262748502
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/6.1.0 CPython/3.13.12
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on May 13, 2026.
Transparency logRelease files / agentqa-1.0.0-cp311-cp311-macosx_10_9_x86_64.whl
| Download URL | agentqa-1.0.0-cp311-cp311-macosx_10_9_x86_64.whl |
|---|---|
| Size | 1.7 MB |
| Tags | CPython 3.11 macOS 10.9+ x86-64 |
|
SHA-256 checksum How to use checksums |
c9f54cee4cd9f96cce8e3319e5f1b5545a00745838d969163bb0bf0b0cf000d4
|
|
BLAKE2b-256 checksum How to use checksums |
da5d9bb43713cba1734204ab17774f28a423c7bc86d3ac2a7e58ea74eeac2e7b
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/6.1.0 CPython/3.13.12
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on May 13, 2026.
Transparency logRelease files / agentqa-1.0.0-cp310-cp310-win_amd64.whl
| Download URL | agentqa-1.0.0-cp310-cp310-win_amd64.whl |
|---|---|
| Size | 1.6 MB |
| Tags | CPython 3.10 Windows x86-64 |
|
SHA-256 checksum How to use checksums |
959eb1a94305c51f7ba1f03a8771c5b1a3f7451005bf927d3c504911042c27ef
|
|
BLAKE2b-256 checksum How to use checksums |
7508c5e13edb7fba7b7bd8419e3d5f6ee5bf093d6f480184cc2ebc7227233f67
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/6.1.0 CPython/3.13.12
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on May 13, 2026.
Transparency logRelease files / agentqa-1.0.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
| Download URL | agentqa-1.0.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl |
|---|---|
| Size | 9.0 MB |
| Tags | CPython 3.10 Linux glibc 2.17+ x86-64 |
|
SHA-256 checksum How to use checksums |
3e1541c2da09b8487c2fccf77c427ac2ff7ffeee83650ccf9f7405c116a59730
|
|
BLAKE2b-256 checksum How to use checksums |
47381fd7f2a135dde6fc9df3de52655767c544ed6e7c2343fc6897600b7de8b9
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/6.1.0 CPython/3.13.12
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on May 13, 2026.
Transparency logRelease files / agentqa-1.0.0-cp310-cp310-macosx_11_0_arm64.whl
| Download URL | agentqa-1.0.0-cp310-cp310-macosx_11_0_arm64.whl |
|---|---|
| Size | 1.7 MB |
| Tags | CPython 3.10 macOS 11.0+ ARM64 |
|
SHA-256 checksum How to use checksums |
458e2a334d1ed1534acd6f721bdbe0f8e94ee7e5fd7ae3f072c68841ea5a2b5c
|
|
BLAKE2b-256 checksum How to use checksums |
2e5ab1f284deae34027c8c2e266b9ae55478b5b21539692a16f8120a1731947d
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/6.1.0 CPython/3.13.12
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on May 13, 2026.
Transparency logRelease files / agentqa-1.0.0-cp310-cp310-macosx_10_9_x86_64.whl
| Download URL | agentqa-1.0.0-cp310-cp310-macosx_10_9_x86_64.whl |
|---|---|
| Size | 1.7 MB |
| Tags | CPython 3.10 macOS 10.9+ x86-64 |
|
SHA-256 checksum How to use checksums |
25c5c2906b4d086df911a5d46e004240d7dc18d0733ff847dfa74e39ab4d56a0
|
|
BLAKE2b-256 checksum How to use checksums |
ba0952b0ed80c3c636a6b639dc0d707e93c61562e93e8b1de9081ef0c03fbe50
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/6.1.0 CPython/3.13.12
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on May 13, 2026.
Transparency logSecurity for agentqa 1.0.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
Full provenance
All 16 files in this release have been uploaded with provenance metadata.
Attestation for agentqa-1.0.0-cp313-cp313-win_amd64.whl
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on May 13, 2026.
Transparency logA note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.