AI Testing Swarm

AI Testing Swarm is a super-advanced, mutation-driven API testing framework (with optional OpenAPI + OpenAI augmentation) built on top of pytest.

It generates a large set of deterministic negative/edge/security test cases for an API request, executes them (optionally in parallel, with retries/throttling), and produces a JSON report with summaries.

Notes:

• UI testing is not the focus of the current releases.
• OpenAI features are optional and disabled by default.

Installation

pip install ai-testing-swarm

CLI entrypoint:

ai-test --help

Quick start (cURL input)

Create request.json:

{
  "curl": "curl --location https://postman-echo.com/post --header \"Content-Type: application/json\" --data \"{\\\"hello\\\":\\\"world\\\",\\\"count\\\":1}\""
}

Run:

ai-test --input request.json

A JSON report is written under:

• ./ai_swarm_reports/<METHOD>_<endpoint>/<METHOD>_<endpoint>_<timestamp>.json

Reports include:

• per-test results
• summary counts by status code / failure type
• optional AI summary (if enabled)

Input formats

1) Raw cURL

{ "curl": "curl ..." }

2) Normalized request

{
  "method": "POST",
  "url": "https://example.com/api/login",
  "headers": {"content-type": "application/json"},
  "params": {"a": "b"},
  "body": {"username": "u", "password": "p"}
}

3) OpenAPI-driven (optional)

{
  "openapi": "./openapi.json",
  "path": "/pets",
  "method": "get",
  "headers": {"accept": "application/json"},
  "path_params": {"petId": "123"},
  "query_params": {"limit": 10},
  "body": null
}

• OpenAPI JSON works by default.
• OpenAPI YAML requires PyYAML installed.
• Base URL is read from spec.servers[0].url.
  • Override with AI_SWARM_OPENAPI_BASE_URL if your spec doesn’t include servers.

What test cases are generated?

The swarm always includes:

• happy_path (baseline)

Then generates broad coverage across:

• Method misuse: same path with wrong HTTP methods (GET/PUT/PATCH/DELETE etc.)
• Headers: missing/invalid Content-Type, accept variations, and other header tampering
• Auth (if Authorization header exists): missing/invalid token tests
• Body/query mutations (per field):
  • missing / null / empty / whitespace
  • type probes (int/bool/float/array/object)
  • boundary inputs (very long strings, huge ints, negative values)
  • unicode + special character payloads
• Security payload probes (per field): SQLi/XSS/path traversal/log4j patterns
• Whole-body mutations: null body, empty object, extra unexpected field

Output is deterministic unless OpenAI augmentation is enabled.

Safety mode (recommended for CI/demos)

Mutation testing can be noisy and may accidentally stress a real environment. To force safe demo runs only against public test hosts:

ai-test --input request.json --public-only

Or via env:

export AI_SWARM_PUBLIC_ONLY=1

Allowed hosts in public-only mode:

• httpbin.org
• postman-echo.com
• reqres.in

Performance features

Parallel execution

• Enabled by default via thread pool.
• Control with:
  • AI_SWARM_WORKERS (default: 5)

Retry + backoff (flaky endpoints)

• Retries on transient errors and status codes (408/429/5xx etc.)
• Control with:
  • AI_SWARM_RETRY_COUNT (default: 1)
  • AI_SWARM_RETRY_BACKOFF_MS (default: 250)

Throttling (RPS)

• Global throttle to avoid hammering a target:
  • AI_SWARM_RPS (default: 0 = disabled)

Max test cap

• Avoids accidental DoS / CI timeouts:
  • AI_SWARM_MAX_TESTS (default: 80)

Reporting

Reports include:

• summary.counts_by_failure_type
• summary.counts_by_status_code
• summary.slow_tests (based on SLA)
• summary.risk_score + summary.risk_grade
• summary.delta_vs_previous (from endpoint history)

SLA threshold:

• AI_SWARM_SLA_MS (default: 2000)

Security:

• Redaction is optional and OFF by default.
• Enable redaction with:
  • AI_SWARM_REDACT_ENABLED=1
• When enabled, sensitive headers are redacted (Authorization/Cookie/api tokens etc.)
• When enabled, sensitive keys in request params/body and response snippets are recursively redacted
• Add extra redaction keys with:
  • AI_SWARM_REDACT_KEYS (comma-separated, e.g. customerEmail,ssn)

HTML + history:

• Per run HTML: ai_swarm_reports/<METHOD_endpoint>/<METHOD_endpoint>_<timestamp>.html
• Endpoint latest: ai_swarm_reports/<METHOD_endpoint>/latest.html
• Endpoint history file: ai_swarm_reports/<METHOD_endpoint>/history.json
• Project dashboard: ai_swarm_reports/index.html
• Project manual-pass overrides: ai_swarm_reports/manual_pass_overrides.json (shared across browsers/runs)
• Detailed results support MANUALLY_PASSED from report UI, sourced only from project-level manual_pass_overrides.json (no browser fallback).
  • For button updates from HTML, serve reports over HTTP:
    • Auto-start happens during report generation by default (AI_SWARM_MANUAL_PASS_AUTOSTART=1)
    • Manual start (optional): ai-test --serve-reports --serve-host 127.0.0.1 --serve-port 8787
    • Open: http://127.0.0.1:8787/index.html
    • If you open HTML via file://, button will auto-target http://127.0.0.1:8787/api/manual-pass
  • Optional API override:
    • AI_SWARM_MANUAL_PASS_API_URL=/api/manual-pass (default)
    • AI_SWARM_REPORT_API_HOST=127.0.0.1
    • AI_SWARM_REPORT_API_PORT=8787
    • AI_SWARM_MANUAL_PASS_AUTOSTART=1
• Owner & escalation links are configurable:
  • AI_SWARM_OWNER_MAP=GET_quickview=platform-api;*=qa-platform
  • AI_SWARM_ENABLE_JIRA_ACTION=1 (0 disables Jira button in report UI)
  • AI_SWARM_ENABLE_SLACK_ACTION=1 (0 disables Slack button in report UI)
  • AI_SWARM_JIRA_URL_TEMPLATE=https://jira.example.com/create?summary={summary}&description={description}
  • AI_SWARM_SLACK_URL_TEMPLATE=https://slack.example.com/notify?text={summary}
  • Jira/Slack templates support placeholders:
    • {method}, {url}, {run_time}, {endpoint}, {endpoint_name}, {owner}
    • {decision}, {health}, {risk_score}, {risk_grade}, {risk_label}
    • {total_tests}, {pass_count}, {manual_pass_count}, {effective_pass_count}, {fail_count}, {unknown_count}, {unknown_status_count}, {blocking_count}
    • {summary} and {description} are URL-encoded
    • {summary_raw} and {description_raw} are plain text
• Configure with:
  • AI_SWARM_HTML_REPORT=1 (default enabled)
  • AI_SWARM_REPORT_HISTORY_LIMIT=120
  • AI_SWARM_OPEN_REPORT=off|dashboard|latest|both (default off)
  • AI_SWARM_OPEN_REPORT_FORCE=1 to allow auto-open in CI/headless environments

Optional OpenAI augmentation (advanced)

A) Generate additional test cases (planner augmentation)

Enable:

export AI_SWARM_USE_OPENAI=1
export OPENAI_API_KEY=... 
export AI_SWARM_MAX_AI_TESTS=30

B) Human-readable AI summary in report

Enable:

export AI_SWARM_USE_OPENAI=1
export AI_SWARM_AI_SUMMARY=1
export OPENAI_API_KEY=...

Model selection:

• AI_SWARM_OPENAI_MODEL (default: gpt-4.1-mini)

CLI help

ai-test --help

Plain Pytest Integration Test (No Allure)

Run the offers-list swarm test without Allure dependency:

AI_SWARM_RUN_LIVE_TESTS=1 PYTHONPATH=src pytest -q tests/test_offers_list_api_swarm_plain.py -s

Public API plain tests (no Allure):

AI_SWARM_RUN_LIVE_TESTS=1 PYTHONPATH=src pytest -q tests/test_public_postman_get_swarm_plain.py -s
AI_SWARM_RUN_LIVE_TESTS=1 PYTHONPATH=src pytest -q tests/test_public_postman_post_swarm_plain.py -s
AI_SWARM_RUN_LIVE_TESTS=1 PYTHONPATH=src pytest -q tests/test_public_httpbin_anything_swarm_plain.py -s

For private/preprod offers tests, also set:

AI_SWARM_RUN_PRIVATE_TESTS=1

Release decisions

The swarm produces a release decision:

• APPROVE_RELEASE
• APPROVE_RELEASE_WITH_RISKS
• REJECT_RELEASE

The decision is derived from deterministic rules (not an LLM).

License

MIT