Note. The xbow-validation/ folder is a git submodule containing the public XBOW validation benchmark suite (104 vulnerable web apps) used in the evaluation. Its contents are not included in the anonymized copy of this repository and appear as an empty folder. To reproduce the evaluation, obtain the public XBOW validation benchmarks, place them in xbow-validation/, then apply the environment fixes with benchmarks/patch_bitrot.sh (repairs aged base images for current Docker / Apple Silicon without changing any tested vulnerability or expected flag).

SwarmAttacker

Multi-methodology swarm penetration testing agent built with LangGraph. Part of a master's thesis on autonomous LLM-based penetration testing.

The core idea: instead of running one methodology at a time, SwarmAttacker deploys multiple attack agents in parallel (OWASP categories, vulnerability-specific specialists, custom attack chains) and aggregates results.

Requirements

• Python 3.12+
• uv (package manager)
• tmux (for agent session isolation) — install with brew install tmux
• Pentesting tools your agents will call: nmap, gobuster, sqlmap, nikto, curl — install with brew install nmap gobuster sqlmap nikto (or just run ./scripts/setup.sh below). Technology fingerprinting is done via curl -sI + the homepage HTML that fetch_page already pulls — whatweb is intentionally not required (it was dropped from Homebrew and added little over a header probe on our target workload).
• An LLM backend, one of:
  • ChatGPT Plus/Pro subscription (recommended, free with subscription — see LLM Provider: Codex below)
  • Anthropic API key
  • OpenAI API key
  • OpenRouter API key

Setup

uv sync                  # install Python deps + create .venv
./scripts/setup.sh                  # install pentesting tools (tmux, nmap, gobuster, sqlmap, nikto) + Playwright Chromium
./scripts/setup.sh --with-seclists  # ALSO clone SecLists (~1 GB) to ~/.swarmattacker/seclists for the gobuster "medium"/"big" presets
cp .env.example .env     # create .env (can stay empty if using Codex auth)
codex                    # one-time ChatGPT login (saves tokens to ~/.codex/auth.json)

Debug a single benchmark (recommended while iterating)

Use this when you want to watch the agent think on one XBOW challenge and see every tool call and output live in the terminal — the right rhythm while the agent itself is still being tuned.

# Daily 15 (resume-friendly):
uv run python -m benchmarks.xbow_runner --daily --resume --skip-build

# Single bench debug:
uv run python -m benchmarks.xbow_runner --bench XBEN-006-24 --skip-build

# Quiet for an overnight sweep:
uv run python -m benchmarks.xbow_runner --daily --resume --skip-build

# Loud for one-time deep debugging:
uv run python -m benchmarks.xbow_runner --bench XBEN-006-24 --verbose

What --verbose adds, streamed to stderr in real time:

• Every shell command the agent runs (with the agent's own reasoning).
• Every command's output, in full (no truncation).
• Every node transition with duration and a one-line summary.
• Every new AI message a node added, so the planner's decisions and the worker's reasoning land in the same terminal as the tool I/O.

When the run finishes you also get a per-run folder with all artifacts:

logs/run-XBEN-006-24-<ts>-<pid>/
  nodes.jsonl              # one JSON line per traced() node call, full result
  terminal_events.jsonl    # one JSON line per tool call/output (machine-readable)
  final_state.json         # graph.ainvoke return value, full state
  summary.md               # human-readable digest: timeline + findings + full
                           # message stream + per-node result dumps

summary.md is the file to open after the run. Pick a benchmark from benchmarks/daily_15.txt (or any XBEN-XXX-24), run with --verbose, read the summary, fix one thing, re-run. That's the loop.

Tip — second-terminal live-tail: if you ever need a structured view while a run is going (without --verbose), open another terminal and:

tail -f "logs/run-XBEN-006-24-"*"/terminal_events.jsonl" | jq

Quick start (local, vulnerable target)

When debugging the agent, point it at a known-vulnerable container running locally — not a real public site. Frontier models (especially the Codex backend) often refuse to attack real-looking domains regardless of what the prompt says, which makes it hard to tell whether the agent is broken or the model is just refusing.

# Start OWASP Juice Shop on port 3000 (or use the helper):
bash benchmarks/run_juice_shop.sh

# Then in LangGraph Studio chat:
#   target_url = http://localhost:3000
# Expected: chat shows initialize → recon → [sqli] tool calls →
#   [sqli] finding(s) → … → final report. No blank period > ~10s.
# Expected: at least one SQLi finding on /rest/user/login
#   (classic ' OR 1=1-- on the email field).

If you see ⚠️ [agent-id] model refused the task in chat, the LLM endpoint is the problem. Check the LLM provider initialized: … line that provider.py logs at startup — if provider=codex you're hitting ChatGPT's policy layer at chatgpt.com/backend-api/codex/responses, which is stricter than direct Anthropic. The authorization preamble in src/knowledge/prompts/base_rules.py reduces but doesn't eliminate these refusals; switching that agent's LLMConfig to Anthropic is the last-resort workaround.

Running

The canonical entry point while iterating is the benchmark runner — see "Debug a single benchmark" above. The graph is invoked in-process via graph.ainvoke(); no LangGraph Studio / dev server involved.

Live operator guidance

Real-target runs accept new instructions while the swarm is working. Type into the persistent bordered editor at the bottom of the terminal and press Enter. The text is queued without interrupting an active tool or worker, persisted as a real user message at the next safe graph barrier, and reviewed by the planner before it chooses another action. Multiple messages are applied in arrival order. The initial instruction and Continue-engagement update use the same editor. Command-V, mouse paste, and multiline bracketed paste are supported; pasted newlines remain part of one message until Enter is pressed. Shift-Enter or Option-Enter inserts a line break in terminals that report modified Enter keys; Ctrl-J is the portable fallback. Long text wraps inside the box, which expands to multiple rows and keeps the latest rows visible on a short terminal.

When Continue engagement is selected, the TUI also asks for an optional additional instruction before restarting. That instruction is appended to the preserved conversation and takes precedence over conflicting older guidance. Ctrl-C retains its existing behavior: once requests a graceful pause and twice forces exit from the latest durable checkpoint.

The langgraph dev Studio UI is currently disabled for this workflow. If you need to re-enable it (interactive node-by-node debugging in a browser), run langgraph dev --allow-blocking from the project root. --allow-blocking is required because the tmux-based terminal tool uses subprocess calls that LangGraph's blockbuster detector flags (they're already wrapped in asyncio.to_thread and don't actually block the event loop).

Testing

The suite runs in under a second and can be re-run as often as you like:

uv run pytest                       # full suite
uv run pytest -v                    # verbose, shows each test name
uv run pytest tests/test_skill_loader.py   # one file

Tests live in tests/ and mirror the src/ layout. Currently only Tier 1 (unit, no LLM, no network) tests exist:

File	What it pins down
tests/test_skill_loader.py	Every SKILL.md parses; every tool name in any frontmatter resolves via the registry; the planner's dispatch menu is correct
tests/test_tool_registry.py	Every registry entry is a real BaseTool; tool.name matches its registry key (otherwise the model emits unroutable tool calls)
tests/test_finding_parsers.py	Markdown **FINDING:** and JSON {"findings": [...]} extraction across the formats and edge cases the agent actually emits
tests/test_planner_decision_parser.py	Fenced / bare / multiple / malformed JSON decision blocks from the planner

Test-on-failure policy

This project does not add tests preemptively. A new test only gets written after a real failure has been observed and the fix is in. Every encountered failure is logged in tests/FAILURES.md even when no test is added — that file doubles as a thesis artefact (a real, dated record of agent failure modes encountered during development). The full policy lives in the project root CLAUDE.md / AGENTS.md under "Testing Policy".

Test tiers, in order of cost (always pick the cheapest one that would catch the failure):

1. Unit — pure functions only (parsers, loaders, registries). What's in tests/ today.
2. Node — inject a FakeListChatModel into BaseNode.run_skill_agent(llm=...). No real API call. Tests orchestration: did the node load the right skill, set the right state flags, propagate findings back?
3. Tool smoke — runs the actual binary (nmap, gobuster, ...) against a local target. Marked @pytest.mark.tools.
4. Live LLM — real model, real local target. Marked @pytest.mark.live, skipped by default.

Architecture

START → initialize → planner ←──────────────────────────┐
                      │                                  │
         ┌────────────┼────────────┬─────────────┐       │
         ↓            ↓            ↓             ↓       │
       recon       executor     web_search    END *      │
                 (×N parallel,                           │
                 Send() fan-out)                         │
                      │                                  │
                      └── all workers return ────────────┘

* The report node is currently bypassed: when the planner picks action="report" the graph routes straight to END. Run-folder artifacts (summary.md, nodes.jsonl, final_state.json, terminal_events.jsonl) are the source of truth instead. To re-enable report, edit _TERMINATE in src/edges/routing.py.

Supervisor-shaped graph: the planner node is the single decision-maker. Every worker edges back to it, and on each turn the planner emits a JSON directive picking the next action — recon, attack (with the exact list of executor dispatches to fan out), web_search, or report.

Nodes (src/nodes/):

• initialize — seeds stealth defaults and cleans leftover tmux state
• planner — supervisor LLM; decides the next action and, for attack, the list of executor dispatches (pre-built skills, custom_configs, or generic tasks) to run in parallel
• recon — reconnaissance agent (port scan, directory discovery, fingerprinting)
• executor — runs ONE dispatch from the planner. Resolves the dispatch's config_name to an AgentConfig (skill, custom_config, or synthesised generic-task) and runs it. The Planner+Executor split (Happe & Cito 2025; Fu et al. 2025) — the executor owns no decision logic, only execution
• web_search — looks up external facts on the planner's request
• report — aggregates all findings into a final report

Edges (src/edges/):

• route_after_planner — reads the planner's decision. Returns a node name (recon / web_search / report) or a list of Send() calls (for attack) that fan out to parallel executor runs

Key subsystems:

• agents/ — config-driven agent pattern. One function, different configs. 13 configs across 3 methodologies (OWASP, vuln-type, custom chains)
• knowledge/ — prompt rules + skill docs (the RAG layer is shelved, see below)
• loop/ — 4-strategy loop detection (hard cap, exact repeat, same-tool repeat, budget pressure)
• llm/ — provider-agnostic interface (Anthropic, OpenAI, OpenRouter, Codex)
  • llm/codex.py — self-contained LangChain chat model for the ChatGPT subscription / Codex backend. Handles OAuth token loading + refresh, Responses API SSE streaming, tool calls, all without any third-party library

Experimental subsystems (off by default): src/experimental/ holds research scaffolds that aren't part of the active agent loop. They are kept as evidence of design exploration; none are registered in the graph. Currently shelved:

• experimental/rag/ — knowledge vector store (FAISS)
• experimental/stealth/ — WAF/IDS detection (no evasion behavior)
• experimental/experience/ — cross-run guide store

Project structure

SwarmAttacker/
├── pyproject.toml              # Project config (uv + hatchling)
├── langgraph.json              # LangGraph Studio entry point
├── benchmarks/
│   ├── targets.yaml            # Benchmark target definitions (DVWA, Juice Shop, etc.)
│   ├── runner.py               # Benchmark runner
│   ├── ablation.py             # Ablation experiment runner
│   ├── multimodel.py           # Multi-model comparison
│   └── metrics.py              # Metric computation
├── src/                        # Main Python package
│   ├── graph.py                # LangGraph graph (pure wiring) + runtime config
│   ├── state.py                # Shared state schema + reducers
│   ├── cli.py                  # CLI entry point
│   ├── nodes/                  # Graph nodes (one file per node)
│   ├── edges/                  # Routing logic
│   ├── knowledge/              # Prompt rules + skill docs
│   ├── tools/                  # tmux-based command execution
│   ├── loop/                   # Loop detection
│   ├── experimental/           # Shelved scaffolds (rag/, stealth/, experience/)
│   └── llm/                    # Provider-agnostic LLM interface
└── tests/                      # See "Testing" section above
    ├── conftest.py             # Shared fixtures (and import-order warm-up)
    ├── FAILURES.md             # Dated log of every failure encountered
    └── test_*.py               # Tier 1 unit tests

Configuration

Runtime behavior is controlled by the config singleton in src/graph.py (budgets, verbosity). All settings are overridable via SWARM_* environment variables — see the _env_* helpers and describe_config() for the full list.

Local GGUFs via llama-server

Provider.LOCAL routes every LLM call through a local llama.cpp server (or Ollama) over its OpenAI-compatible endpoint, so any GGUF in ~/llms/ can be used without code changes.

brew install llama.cpp
llama-server -m ~/llms/Hermes-3-Llama-3.1-8B-Q4_K_M.gguf \
  --port 8080 --alias hermes-8b -c 32768
# in another shell:
SWARM_PROVIDER=local SWARM_LOCAL_MODEL=hermes-8b uv run swarm ...

Env vars: SWARM_PROVIDER=local, SWARM_LOCAL_MODEL=<--alias>, SWARM_LOCAL_BASE_URL=http://127.0.0.1:8080/v1 (Ollama: :11434/v1). Tool-call quality depends entirely on the GGUF — Hermes-3-8B and the abliterated gemma variants need a hand-crafted jinja template to call tools reliably (see tests/FAILURES.md 2026-05-17). The wiring works; swap models freely.

Dependencies

Package	Purpose
langgraph	Graph orchestration, state management, checkpointing, Studio
langchain-core	Base abstractions (messages, tools, chat models)
langchain-anthropic	Claude model integration
langchain-openai	OpenAI / OpenRouter model integration
pydantic	Data validation (used by LangChain internals)
libtmux	tmux session management for agent command isolation
pyyaml	SKILL.md frontmatter + benchmark targets parsing