🧪 agentspec
Auto-test and validate any API — built for the agent era.
agentspec discovers your API endpoints, generates hundreds of tests automatically, and runs them — all from one command. Works with any backend framework. Integrates into Claude Code, Codex, and Cursor via MCP.
Installation · Quick start · Frameworks · MCP server · CLI reference
What it does
agentspec test app/main.py --base-url http://localhost:8000
agentspec v0.1.0
Discovering API endpoints...
Discovered 9 endpoint(s) in Task Management API
Generating tests...
Generated 128 test case(s)
Running tests against http://localhost:8000 (concurrency=10)...
✓ [CRITICAL] SQL injection in body field 'title' 39 ms [201]
✓ [ HIGH] Wrong method PUT on /tasks 25 ms [405]
✓ [ HIGH] Content-Type matches expected on GET /tasks 31 ms [200]
✓ [ MEDIUM] Enum value 'todo' for param 'status' 32 ms [200]
✓ [ MEDIUM] Request with all optional params 44 ms [200]
...
128 passed | 0 failed | 0 errors (4314 ms)
██████████████████████████████ 100.0%
One command. Zero config. 128 tests.
What gets tested
For every endpoint agentspec discovers, it generates tests across five categories:
| Category | What it checks |
|---|---|
| Happy path | Valid requests, all param combos, per-enum-value coverage |
| Edge cases | Empty strings, unicode, 10K-char strings, missing optional params |
| Error handling | Wrong HTTP methods, wrong types, unknown params, extra body fields |
| Security | SQL injection (8 variants), XSS, null bytes, path traversal, 1 MB payload bombs |
| Schema validation | JSON content-type, valid JSON body, response structure on every endpoint |
Installation
pip install agentspec
Requires Python 3.10+.
Quick start
# Start your API
uvicorn app.main:app --reload
# Test it — framework is auto-detected
agentspec test app/main.py --base-url http://localhost:8000
Works with any framework:
# Flask
agentspec test api/app.py --base-url http://localhost:5000
# Express / NestJS / Fastify
agentspec test src/app.ts --base-url http://localhost:3000
# OpenAPI spec
agentspec test openapi.yaml --base-url http://localhost:8000
# Scan an entire directory
agentspec test ./src --base-url http://localhost:3000
Supported frameworks
| Language | Framework | How it's discovered |
|---|---|---|
| Python | FastAPI | AST analysis |
| Python | Flask | AST analysis (Blueprints, MethodView, path converters) |
| Python | Django REST Framework | OpenAPI spec |
| JS / TS | Express.js | Regex analysis |
| JS / TS | NestJS | Decorator + controller analysis |
| JS / TS | Next.js (App Router) | Export handler + file path analysis |
| JS / TS | Fastify | Regex analysis |
| JS / TS | Hono | Regex analysis |
| Any | OpenAPI / Swagger | JSON or YAML spec |
MCP server
Add agentspec to Claude Code, Codex, or Cursor so your AI agent can test APIs directly:
{
"mcpServers": {
"agentspec": {
"command": "python",
"args": ["-m", "agentspec.mcp.server"]
}
}
}
Available tools:
| Tool | What it does |
|---|---|
test_api |
Discover, generate, and run all tests |
discover_api |
Discover endpoints without running tests |
register_api |
Test and register an API in the local registry |
list_registered |
List all registered APIs |
check_compatibility |
Detect breaking changes between API versions |
Example — Claude Code:
You: test my API at http://localhost:8000 using app/main.py
Claude: [calls test_api]
→ 128 passed, 0 failed. Here's the full report...
CLI reference
agentspec test
agentspec test <source> --base-url <url> [options]
Options:
--base-url TEXT Base URL of the running API [required]
--concurrency INT Parallel test runners (default: 10)
--timeout FLOAT Per-request timeout in seconds (default: 10.0)
--format TEXT text | json | html | junit (default: text)
--output PATH Write report to file
--autofix Show auto-fix suggestions for failures
--watch Re-run on source file changes
--config PATH Path to .agentspec.yaml
agentspec discover
Show all endpoints without running tests.
agentspec discover app/main.py
agentspec init
Generate a .agentspec.yaml config file.
agentspec init
agentspec register
Test and register an API in the local registry.
agentspec register app/main.py --base-url http://localhost:8000 --name "My API" --version 1.0.0
agentspec compat
Check for breaking changes between registered versions.
agentspec compat my-api:1.0.0 my-api:2.0.0
agentspec serve
Start the MCP server.
agentspec serve
HTML reports
agentspec test app/main.py --base-url http://localhost:8000 --format html --output report.html
Self-contained single-file HTML with donut chart, summary cards, sortable/filterable test table, and dark/light mode. No external dependencies.
Watch mode
Re-runs the full test suite whenever a source file changes:
agentspec test app/main.py --base-url http://localhost:8000 --watch
Auto-fix suggestions
agentspec test app/main.py --base-url http://localhost:8000 --autofix
When tests fail, the auto-fix engine analyzes failures and generates structured suggestions with code snippets — ready to apply directly or hand to your AI agent.
Config file
# .agentspec.yaml
base_url: http://localhost:8000
concurrency: 10
timeout: 10.0
format: text
skip_endpoints:
- GET /health
- GET /metrics
skip_categories:
- security
# Env var expansion supported
headers:
Authorization: "Bearer ${API_TOKEN}"
X-API-Key: "${API_KEY}"
CI integration
# .github/workflows/api-tests.yml
name: API Tests
on: [push, pull_request]
jobs:
test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: "3.12"
- run: pip install agentspec fastapi uvicorn
- run: uvicorn app.main:app --host 0.0.0.0 --port 8000 &
- run: sleep 2
- run: agentspec test app/main.py --base-url http://localhost:8000 --format junit --output results.xml
- uses: actions/upload-artifact@v4
with:
name: test-results
path: results.xml
API registry
agentspec keeps a local registry at ~/.agentspec/registry.json. Register multiple versions and diff them:
# Register v1
agentspec register app/main.py --base-url http://localhost:8000 --name myapi --version 1.0.0
# Register v2 after making changes
agentspec register app/main.py --base-url http://localhost:8001 --name myapi --version 2.0.0
# See what changed
agentspec compat myapi:1.0.0 myapi:2.0.0
Claude Code skill
Install the agentspec skill to teach Claude Code to automatically test APIs it generates:
cp -r skill ~/.claude/skills/agentspec
Once installed, Claude will suggest running agentspec after creating new endpoints and use auto-fix output to repair failures itself.
License
MIT
Metadata
Release files for agentspec-cli 0.1.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| agentspec_cli-0.1.0.tar.gz | 100.5 kB | Without provenance | Details for agentspec_cli-0.1.0.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| agentspec_cli-0.1.0-py3-none-any.whl | Python 3 | none | any | Details for agentspec_cli-0.1.0-py3-none-any.whl |
Total release size: 206.8 kB
Release files / agentspec_cli-0.1.0.tar.gz
| Download URL | agentspec_cli-0.1.0.tar.gz |
|---|---|
| Size | 100.5 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
e6a96403711ddd334e02865fcf9585acd63cff1c694c8734b1cfd7a712954782
|
|
BLAKE2b-256 checksum How to use checksums |
a2a9d1e82294afc732f84592f0e89dd703a6ac602941ed54e096b1f5a01ca3d7
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.12.14
|
Release files / agentspec_cli-0.1.0-py3-none-any.whl
| Download URL | agentspec_cli-0.1.0-py3-none-any.whl |
|---|---|
| Size | 106.3 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
c5cd71f6038c8eaab082b88a921f66cbb0a539da428367db5375ce01a3fc1c47
|
|
BLAKE2b-256 checksum How to use checksums |
00472e7fc338ad8eb13c1bc38c211b2201c60f8f69c711cbce6f032b853ceba1
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.12.14
|
Security for agentspec-cli 0.1.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.