AI Testing Swarm
AI Testing Swarm is a super-advanced, mutation-driven API testing framework (with optional OpenAPI + OpenAI augmentation) built on top of pytest.
It generates a large set of deterministic negative/edge/security test cases for an API request, executes them (optionally in parallel, with retries/throttling), and produces a JSON report with summaries.
Notes:
- UI testing is not the focus of the current releases.
- OpenAI features are optional and disabled by default.
Installation
pip install ai-testing-swarm
CLI entrypoint:
ai-test --help
Quick start (cURL input)
Create request.json:
{
"curl": "curl --location https://postman-echo.com/post --header \"Content-Type: application/json\" --data \"{\\\"hello\\\":\\\"world\\\",\\\"count\\\":1}\""
}
Run:
ai-test --input request.json
A JSON report is written under:
./ai_swarm_reports/<METHOD>_<endpoint>/<METHOD>_<endpoint>_<timestamp>.json
Reports include:
- per-test results
- summary counts by status code / failure type
- optional AI summary (if enabled)
Input formats
1) Raw cURL
{ "curl": "curl ..." }
2) Normalized request
{
"method": "POST",
"url": "https://example.com/api/login",
"headers": {"content-type": "application/json"},
"params": {"a": "b"},
"body": {"username": "u", "password": "p"}
}
3) OpenAPI-driven (optional)
{
"openapi": "./openapi.json",
"path": "/pets",
"method": "get",
"headers": {"accept": "application/json"},
"path_params": {"petId": "123"},
"query_params": {"limit": 10},
"body": null
}
- OpenAPI JSON works by default.
- OpenAPI YAML requires
PyYAMLinstalled. - Base URL is read from
spec.servers[0].url.- Override with
AI_SWARM_OPENAPI_BASE_URLif your spec doesn’t include servers.
- Override with
What test cases are generated?
The swarm always includes:
- happy_path (baseline)
Then generates broad coverage across:
- Method misuse: same path with wrong HTTP methods (GET/PUT/PATCH/DELETE etc.)
- Headers: missing/invalid
Content-Type, accept variations, and other header tampering - Auth (if
Authorizationheader exists): missing/invalid token tests - Body/query mutations (per field):
- missing / null / empty / whitespace
- type probes (int/bool/float/array/object)
- boundary inputs (very long strings, huge ints, negative values)
- unicode + special character payloads
- Security payload probes (per field): SQLi/XSS/path traversal/log4j patterns
- Whole-body mutations: null body, empty object, extra unexpected field
Output is deterministic unless OpenAI augmentation is enabled.
Safety mode (recommended for CI/demos)
Mutation testing can be noisy and may accidentally stress a real environment. To force safe demo runs only against public test hosts:
ai-test --input request.json --public-only
Or via env:
export AI_SWARM_PUBLIC_ONLY=1
Allowed hosts in public-only mode:
httpbin.orgpostman-echo.comreqres.in
Performance features
Parallel execution
- Enabled by default via thread pool.
- Control with:
AI_SWARM_WORKERS(default:5)
Retry + backoff (flaky endpoints)
- Retries on transient errors and status codes (408/429/5xx etc.)
- Control with:
AI_SWARM_RETRY_COUNT(default:1)AI_SWARM_RETRY_BACKOFF_MS(default:250)
Throttling (RPS)
- Global throttle to avoid hammering a target:
AI_SWARM_RPS(default:0= disabled)
Max test cap
- Avoids accidental DoS / CI timeouts:
AI_SWARM_MAX_TESTS(default:80)
Reporting
Reports include:
summary.counts_by_failure_typesummary.counts_by_status_codesummary.slow_tests(based on SLA)summary.risk_score+summary.risk_gradesummary.delta_vs_previous(from endpoint history)
SLA threshold:
AI_SWARM_SLA_MS(default:2000)
Security:
- Redaction is optional and OFF by default.
- Enable redaction with:
AI_SWARM_REDACT_ENABLED=1
- When enabled, sensitive headers are redacted (Authorization/Cookie/api tokens etc.)
- When enabled, sensitive keys in request params/body and response snippets are recursively redacted
- Add extra redaction keys with:
AI_SWARM_REDACT_KEYS(comma-separated, e.g.customerEmail,ssn)
HTML + history:
- Per run HTML:
ai_swarm_reports/<METHOD_endpoint>/<METHOD_endpoint>_<timestamp>.html - Endpoint latest:
ai_swarm_reports/<METHOD_endpoint>/latest.html - Endpoint history file:
ai_swarm_reports/<METHOD_endpoint>/history.json - Project dashboard:
ai_swarm_reports/index.html - Project manual-pass overrides:
ai_swarm_reports/manual_pass_overrides.json(shared across browsers/runs) - Detailed results support
MANUALLY_PASSEDfrom report UI, sourced only from project-levelmanual_pass_overrides.json(no browser fallback).- For button updates from HTML, serve reports over HTTP:
- Auto-start happens during report generation by default (
AI_SWARM_MANUAL_PASS_AUTOSTART=1) - Manual start (optional):
ai-test --serve-reports --serve-host 127.0.0.1 --serve-port 8787 - Open:
http://127.0.0.1:8787/index.html - If you open HTML via
file://, button will auto-targethttp://127.0.0.1:8787/api/manual-pass
- Auto-start happens during report generation by default (
- Optional API override:
AI_SWARM_MANUAL_PASS_API_URL=/api/manual-pass(default)AI_SWARM_REPORT_API_HOST=127.0.0.1AI_SWARM_REPORT_API_PORT=8787AI_SWARM_MANUAL_PASS_AUTOSTART=1
- For button updates from HTML, serve reports over HTTP:
- Owner & escalation links are configurable:
AI_SWARM_OWNER_MAP=GET_quickview=platform-api;*=qa-platformAI_SWARM_ENABLE_JIRA_ACTION=1(0disables Jira button in report UI)AI_SWARM_ENABLE_SLACK_ACTION=1(0disables Slack button in report UI)AI_SWARM_JIRA_URL_TEMPLATE=https://jira.example.com/create?summary={summary}&description={description}AI_SWARM_SLACK_URL_TEMPLATE=https://slack.example.com/notify?text={summary}- Jira/Slack templates support placeholders:
{method},{url},{run_time},{endpoint},{endpoint_name},{owner}{decision},{health},{risk_score},{risk_grade},{risk_label}{total_tests},{pass_count},{manual_pass_count},{effective_pass_count},{fail_count},{unknown_count},{unknown_status_count},{blocking_count}{summary}and{description}are URL-encoded{summary_raw}and{description_raw}are plain text
- Configure with:
AI_SWARM_HTML_REPORT=1(default enabled)AI_SWARM_REPORT_HISTORY_LIMIT=120AI_SWARM_OPEN_REPORT=off|dashboard|latest|both(defaultoff)AI_SWARM_OPEN_REPORT_FORCE=1to allow auto-open in CI/headless environments
Optional OpenAI augmentation (advanced)
A) Generate additional test cases (planner augmentation)
Enable:
export AI_SWARM_USE_OPENAI=1
export OPENAI_API_KEY=...
export AI_SWARM_MAX_AI_TESTS=30
B) Human-readable AI summary in report
Enable:
export AI_SWARM_USE_OPENAI=1
export AI_SWARM_AI_SUMMARY=1
export OPENAI_API_KEY=...
Model selection:
AI_SWARM_OPENAI_MODEL(default:gpt-4.1-mini)
CLI help
ai-test --help
Plain Pytest Integration Test (No Allure)
Run the offers-list swarm test without Allure dependency:
AI_SWARM_RUN_LIVE_TESTS=1 PYTHONPATH=src pytest -q tests/test_offers_list_api_swarm_plain.py -s
Public API plain tests (no Allure):
AI_SWARM_RUN_LIVE_TESTS=1 PYTHONPATH=src pytest -q tests/test_public_postman_get_swarm_plain.py -s
AI_SWARM_RUN_LIVE_TESTS=1 PYTHONPATH=src pytest -q tests/test_public_postman_post_swarm_plain.py -s
AI_SWARM_RUN_LIVE_TESTS=1 PYTHONPATH=src pytest -q tests/test_public_httpbin_anything_swarm_plain.py -s
For private/preprod offers tests, also set:
AI_SWARM_RUN_PRIVATE_TESTS=1
Release decisions
The swarm produces a release decision:
APPROVE_RELEASEAPPROVE_RELEASE_WITH_RISKSREJECT_RELEASE
The decision is derived from deterministic rules (not an LLM).
License
MIT
Metadata
Release files for ai-testing-swarm 0.1.23
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| ai_testing_swarm-0.1.23.tar.gz | 85.3 kB | Without provenance | Details for ai_testing_swarm-0.1.23.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| ai_testing_swarm-0.1.23-py3-none-any.whl | Python 3 | none | any | Details for ai_testing_swarm-0.1.23-py3-none-any.whl |
Total release size: 164.1 kB
Release files / ai_testing_swarm-0.1.23.tar.gz
| Download URL | ai_testing_swarm-0.1.23.tar.gz |
|---|---|
| Size | 85.3 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
085f12e7753b1960c94dee08607a5e33947b82cae884a32715a1cbcb1c784c62
|
|
BLAKE2b-256 checksum How to use checksums |
95828bbb41fc26e3305c2ad1cf428af589af4b151c11b61e5ffb6c46216b3e2e
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.2.0 CPython/3.11.4
|
Release files / ai_testing_swarm-0.1.23-py3-none-any.whl
| Download URL | ai_testing_swarm-0.1.23-py3-none-any.whl |
|---|---|
| Size | 78.8 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
0fd213ec942b9c09c21ea737a43fc8d8023c661538705490e99f0068aa742cbc
|
|
BLAKE2b-256 checksum How to use checksums |
9f269fbf634cffa444b06be8b4dc7e3c6b586c74d200245312acffa6be760e5e
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.2.0 CPython/3.11.4
|
Security for ai-testing-swarm 0.1.23
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.