bastionprobe

Pentest for AI agents. Fire indirect prompt-injection payloads at your agent and get a report of which ones land. Every payload that gets through is a hole a runtime guard should close.

The offensive twin of agentbastion (the runtime firewall). agentbastion blocks attacks in production; bastionprobe launches them before you ship. Scan with bastionprobe → find holes → plug them with agentbastion → re-scan → green.

bastionprobe = Burp Suite for agents      (red team, pre-ship, on demand)
agentbastion = WAF for agents           (blue team, production, always on)

Part of the bastion trilogy — attack (bastionprobe), prevent (agentbastion), investigate (bastiontrace) — all sharing one payload corpus, bastioncorpus.

The wedge: indirect prompt injection

The hardest injection to defend and the one most agents are blind to: the payload doesn't come from the user — it rides inside a tool result the agent reads mid-task (a fetched web page, a support ticket, a DB row, a file). If the agent treats that data as instructions, you're owned.

bastionprobe plants a payload in a tool output, runs your agent for one turn, and scores whether it obeyed:

• canary check — the payload tells the agent to emit a unique random token. Token shows up in the reply ⇒ injection landed. (Fresh token per run — can't be faked or pre-trained.)
• tool check — the payload tells the agent to call a forbidden tool (send_email, delete_file, http_post, …). Agent calls it ⇒ landed.

Install

pip install bastionprobe

load_payloads() sources its payloads from bastioncorpus (the shared trilogy corpus, pulled in as a dependency); a bundled payloads.jsonl snapshot is the fallback. Pass an explicit path to load_payloads(path) to fire your own set.

Quick start

Zero setup — fire at the bundled demo targets:

bastionprobe run                                      # vulnerable demo: 12/12 land
bastionprobe run --target bastionprobe.demo:safe_agent  # safe demo: 0 land

Fire at your agent — implement one function (see examples/my_agent.py):

from bastionprobe import AgentResponse

def agent(messages, tool_outputs):        # tool_outputs = {tool_name: output}
    reply, tools_called = run_your_agent(messages, tool_outputs)
    return AgentResponse(text=reply, tool_calls=tools_called)

bastionprobe run --target examples.my_agent:agent --out results.jsonl

Models are non-deterministic — the same payload lands on one run and blocks on the next. Fire each payload N times for the stable land rate:

bastionprobe run --target examples.my_agent:agent --runs 5   # shows e.g. [FAIL 3/5]

landed still counts a payload that got through even once (a hole that opens sometimes is still a hole), but the N/M rate is what you compare between runs.

Exit code is non-zero when any payload lands, so you can gate CI on it:

bastionprobe run --target examples.my_agent:agent || echo "agent is vulnerable"

Fire at a real Claude agent

A built-in adapter turns an Anthropic client into a target. It stands up a small tool-using agent, feeds each payload back as the result of the agent's read_document tool, runs one real model turn, and reports what Claude did — text plus any tools it tried to call.

pip install "bastionprobe[anthropic]"
export ANTHROPIC_API_KEY=...

from anthropic import Anthropic
from bastionprobe import load_payloads, run_suite, make_anthropic_target
from bastionprobe.report import render

target = make_anthropic_target(Anthropic(), model="claude-sonnet-4-5")
print(render(run_suite(target, load_payloads())))

Set model= to the model your production agent runs — that's the behavior you care about. system= and tools= are overridable so you can mirror your real agent's persona and toolbox instead of the defaults. See examples/anthropic_scan.py.

Fire at a GPT agent

Same contract, other vendor. make_openai_target wraps an OpenAI client and reuses the exact demo agent (system prompt + tools) as the Anthropic adapter, so a cross-vendor comparison measures the models, not two different harnesses.

pip install "bastionprobe[openai]"
export OPENAI_API_KEY=...

from openai import OpenAI
from bastionprobe import load_payloads, run_suite, make_openai_target
from bastionprobe.report import render

target = make_openai_target(OpenAI(), model="gpt-4o")
print(render(run_suite(target, load_payloads(), runs=5)))

Cross-model matrix

Fire the same suite at several models and compare land rate by tactic — turns a single-model result into a comparison:

bastionprobe matrix --models claude-opus-4-5,claude-sonnet-4-5,claude-haiku-4-5 --runs 5

cross-model land rate by tactic:
tactic              claude-opus-4-5  claude-sonnet-4-5  claude-haiku-4-5
data-field                     ...%               90%              ...%
destructive                    ...%              100%              ...%
egress-overt                   ...%                0%              ...%
egress-legit                   ...%                0%              ...%
OVERALL                        ...%               69%              ...%

A bad model id shows err for its column — the rest of the matrix still runs. The matrix is vendor-agnostic: mix Anthropic and OpenAI targets in one grid to ask whether a finding holds across vendors (does the egress refusal survive on GPT?). See examples/cross_model.py (Anthropic) and examples/cross_vendor.py (Anthropic + OpenAI).

Injection vector. Both adapters take vector=: tool_result (a real tool-result block — authentic, native) or user_message (a labeled user turn — portable; strict providers like Gemini 3.x reject a fabricated tool-call). For a fair cross-vendor comparison, run the same vector on every model (--vector user_message for the CLI matrix, or the parity default in cross_vendor.py) — otherwise a difference could be the vector, not the model.

Co-evolution director (experimental — Phase 1)

bastionprobe.coevo adds C, a research director for a red-team/blue-team co-evolution loop. It does not attack; it steers the search. Given a round of AttackResults, Director.round(...) classifies each attack into a behavioral cell (framing × surface × language × action, derived from payload metadata), maintains a MAP-Elites quality-diversity archive (one best elite per cell), scores novelty (equal weight to success — so B is pushed toward dissimilar attacks, not just the boundary of A), steers B toward empty cells, flags mode collapse, picks a Hall-of-Fame replay set, and diagnoses real vs illusory progress against a frozen benchmark A is never hardened against.

from bastionprobe import load_payloads, run_suite, Director
from bastionprobe.demo import vulnerable_agent

results = run_suite(vulnerable_agent, load_payloads(), runs=2)
report = Director().round(results, benchmark_score=0.62)
print(report.to_json())   # classification, direction_for_b, replay_for_a, ...

The full autonomous loop — B generates toward the director's target cells, fires at A (a defender wrapped as a Target), C steers the next round, A hardens against what landed, A is re-scored on the frozen benchmark; it stops when progress goes illusory (benchmark stalls while B keeps winning):

bastionprobe coevolve --rounds 5 --runs 2 --per-cell 2

Everything is pluggable (defender = block_fn(text)->bool, generator, harden_fn).

Real A that hardens. --defender agentbastion wires agentbastion in as A: it guards the poisoned tool result, and every round it ingests the attacks that landed as SemanticDetector templates — so it blocks them and their paraphrases next round. With a real embedder this is genuine hardening:

bastionprobe coevolve --defender agentbastion --rounds 6
# round 1: benchmark=0.000   (base A catches none of the held-out attacks)
# round 2: benchmark=0.800   (A hardened on what landed -> catches paraphrases)
# round 4: real_progress=False (benchmark plateaus while B keeps winning)

The last line is the point: the frozen benchmark (held out of hardening) tells real robustness gains from illusory ones. Two embedders ship: hashing_embedder (zero-dependency bag-of-words, the default) and sentence_transformer_embedder (real dense embeddings — generalizes across whole attack families, not just token overlap):

pip install "bastionprobe[embeddings]"
bastionprobe coevolve --defender agentbastion --embedder sentence-transformer

The threshold auto-pairs (0.4 hashing, 0.6 dense; override with --threshold). The demo defender (default) instead memorizes — a deliberately simple ceiling.

B that invents. By default B recombines fixed axis fragments; --generator llm has a model write a fresh injection per target cell, so B reaches novel strategy families the templates can't express. We stamp the cell's axis metadata and enforce its scoring constraint (canary token / tool name), so cell_of still matches — the model supplies only the creative text.

bastionprobe coevolve --defender agentbastion --generator llm --gen-model claude-haiku-4-5

(For authorized testing: the generated strings are planted in the agent-under- test's own tool results to measure and harden its defenses — the same class of test string as the bundled payloads.)

Close the loop: harden the shield

The sword's whole point is to make the shield better. harden turns landed findings into defenses agentbastion loads directly:

bastionprobe run --target examples.my_agent:agent --out results.jsonl
bastionprobe harden results.jsonl        # -> bastion_hardening/{policy.yaml, injections.jsonl}

• policy.yaml — every tool an injection got to call, as a deny-list for agentbastion's ToolPolicy (load_policy).
• injections.jsonl — every injection string that worked, in agentbastion's corpus schema. Load them as SemanticDetector templates and embedding similarity blocks those attacks and their paraphrases.

from agentbastion import Firewall, load_policy
from agentbastion.semantic import SemanticDetector
from agentbastion.inbound import InboundGuard
import json

templates = [json.loads(l)["text"] for l in open("bastion_hardening/injections.jsonl")]
fw = Firewall()
fw.tool_policy = load_policy("bastion_hardening/policy.yaml")   # deny what got called
fw.inbound = InboundGuard(detectors=[SemanticDetector(embed_fn, templates=templates)])

Then re-scan: scan → harden → load → re-scan, and each round the shield learns exactly what the sword got through.

The target contract

A target is any callable (messages, tool_outputs) -> AgentResponse. bastionprobe poisons one value in tool_outputs, runs your agent for one turn, and reads back the reply text plus the names of any tools it called. It never looks inside the agent — it measures behavior. Report what your agent actually did.

Shared corpus with agentbastion

Payloads use the same category taxonomy as agentbastion's benchmark/corpus.jsonl (indirect_injection, exfiltration, direct_injection, …). Same strings, opposite direction: agentbastion reads a row as "block this", bastionprobe reads it as "fire this". A finding here maps directly to a rule there.

Status

Alpha. One attack class (indirect injection via tool output), 16 payloads across 5 languages, tuned against a live model. Payloads are tagged with a framing tactic, and multi-run scans report a land rate by tactic breakdown — the signal that scales as the set grows. See FINDINGS.md for measured results (e.g. Claude refuses data egress but readily deletes local data). Reserved for later: direct injection, RAG poisoning, multi-agent trust escalation, LLM-judge scoring. Payload PRs welcome.

License

MIT.