agentsec-bench 0.1.2

AgentSec-Bench: a security benchmark for LLM agents and harnesses, built on Inspect AI

AgentSec-Bench

A security benchmark for LLM agents & harnesses. (Python package: agentsec.)

AgentSec-Bench measures how well LLM agents and their harnesses perform on security tasks. It is built on Inspect AI (UK AISI) and evaluates agents against live, sandboxed targets across four capability axes:

Axis	What it measures	Grading
offensive	Find & exploit vulns (CTF-style) against a live container; submit a flag.	Automated (flag match)
defensive	Detect insecure code and patch it without breaking functionality.	Hybrid (hidden security tests + judge)
secure_coding	Build a feature to spec without introducing vuln classes.	Hybrid (semgrep/bandit + judge)
harness_safety	Resist prompt injection, refuse misuse, respect sandbox boundaries.	Hybrid (judge + sandbox signals)

Difficulty tiers & partial credit

Challenges are tagged easy | medium | hard | stretch. Run a single tier (or several) and the hardest challenges award gradated partial credit — e.g. 0.33 for reaching stage 1 of a 3-stage exploit chain — so the benchmark discriminates across the capability range instead of collapsing to 0/1.

inspect eval src/agentsec/tasks/offensive.py -T difficulty=stretch --model google/gemini-2.5-flash
inspect eval src/agentsec/tasks/offensive.py -T difficulty=hard,stretch --epochs 3   # repeat stochastic hard tasks

A challenge declares subtasks (weighted exec/flag checks) for partial credit; one without subtasks scores all-or-nothing via the axis default scorer. See CHALLENGES.md for the schema. The hard/stretch set includes a multi-network SSRF→internal-metadata chain, weak-secret JWT forgery, a pickle-deserialization RCE fix, a functionally-graded secure file-upload, and an obfuscated multi-step prompt injection.

The differentiator: benchmarking harnesses, not just models

Most security benchmarks pin a harness and vary the model. agentsec does the opposite as well: via Inspect's agent bridge, external harnesses (Claude Code, Codex CLI, Gemini CLI) have their model calls proxied through Inspect's provider. Every harness therefore runs on the same underlying model, isolating harness quality from model quality. Swap --model to isolate model quality on a fixed harness.

agent harness (in sandbox)  ──API calls──►  localhost:13131 proxy  ──►  Inspect model provider

Install

# Option A — from PyPI (quickest; bundles the framework AND the built-in challenges):
pip install agentsec-bench

# Option B — from source (for development or adding challenges):
git clone https://github.com/Santhosraj/AgentSec-Bench.git
cd AgentSec-Bench
pip install -e ".[dev]"        # add ",scan" on Linux/macOS for semgrep

Running the live sandboxed challenges requires Docker (see Requirements). anthropic ships in core; add a provider extra for others (pip install "agentsec-bench[openai]").

Quickstart

export ANTHROPIC_API_KEY=...    # or GOOGLE_API_KEY / OPENAI_API_KEY, etc.

# Easiest — the bundled CLI (works whether you pip-installed or cloned):
agentsec run --model anthropic/claude-opus-4-8 --axes offensive --difficulty easy
agentsec report ./logs          # comparison table across runs

# Or call Inspect directly (source checkout — gives full -T/-S control):
inspect eval src/agentsec/tasks/offensive.py --model anthropic/claude-opus-4-8

# Same challenge driven by the Claude Code harness via the bridge
# (--solver overrides the task default; -S passes solver args — note -S, not -T):
inspect eval src/agentsec/tasks/offensive.py \
    --solver src/agentsec/solvers/bridge_harness.py -S harness=claude-code \
    --model anthropic/claude-opus-4-8

inspect view   # browse logs / transcripts

Providers

AgentSec-Bench is provider-agnostic — the model under test, the LLM judge, and the harness vendor are all swappable. anthropic ships in core so the examples above work out of the box; install an extra for any other provider:

pip install -e ".[openai]"     # or .[google] / .[mistral] / .[groq] / .[all]

Then point --model (and optionally the judge role) at it — the task and scorer code is unchanged:

inspect eval src/agentsec/tasks/offensive.py --solver builtin_agent --model openai/gpt-4o
inspect eval src/agentsec/tasks/defensive.py --solver builtin_agent \
    --model google/gemini-2.5-pro --model-role grader=openai/gpt-4o
# Local models work too: --model ollama/llama3.1  or  --model vllm/<model>

Because the agent bridge decouples a harness's API dialect from the backing model, you can also run a harness on a different vendor's model — e.g. the Claude Code harness backed by GPT-4o — to isolate scaffolding quality from the model:

inspect eval src/agentsec/tasks/offensive.py \
    --solver src/agentsec/solvers/bridge_harness.py -S harness=claude-code \
    --model openai/gpt-4o

Set the matching API key for whichever provider you use (OPENAI_API_KEY, GOOGLE_API_KEY, MISTRAL_API_KEY, GROQ_API_KEY, …).

Picking a model for the agent role (important)

The agent axes drive a multi-step tool-calling loop, so the agent model must reliably emit structured tool calls. Two tiers:

• ✅ Native tool-calling providers — Anthropic, OpenAI, Google, Mistral. Rock-solid; use these for the agent role when you want clean runs.
• ⚠️ Groq + open models (Llama/qwen/gpt-oss) — Groq re-parses model output into tool calls server-side and is strict about it. Llama-3.3 tends to emit <function=…> text tags and reasoning models emit <tool_call> blocks; Groq rejects both with a fatal 400 tool_use_failed. This is a provider quirk, not a model-capability gap (the model often solves the task right up to the bad tool call). --retry-on-error does not help because the format is deterministic, not flaky.

Want fully open-weight with reliable tools? Use Ollama locally — it parses each model's native tool format itself, so there's none of Groq's tool_use_failed and no API quotas.

# 1. Install Ollama (https://ollama.com/download)

# 2. Pull a tool-capable model. On CPU, prefer a smaller model for the multi-step agent loop:
ollama pull qwen2.5-coder:3b               # ~1.9 GB, decent tools, CPU-friendly
#   (qwen2.5-coder:7b is stronger but ~2-3x slower on CPU and needs ~8 GB RAM)

# 3. Run (--max-connections 1 keeps a local model from being flooded with parallel requests):
inspect eval src/agentsec/tasks/offensive.py -T difficulty=easy \
    --model ollama/qwen2.5-coder:3b --max-connections 1 --display plain

Notes for local models: inference is CPU-bound without a GPU, so each agent step takes seconds and a hard challenge can run several minutes — cap it with --message-limit 30. Small (3B) models legitimately score low on the hard/stretch tiers; that's a capability result, not a bug.

The grader role does no tool-calling, so any open model is fine there — pair a native agent model with an open-weight grader:

inspect eval src/agentsec/tasks/defensive.py \
    --model anthropic/claude-haiku-4-5 \
    --model-role grader=groq/llama-3.3-70b-versatile --display plain

No-Docker smoke test (sandbox_type=local)

Don't have Docker set up yet? The benign coding axes (defensive, secure_coding) can run in a local sandbox that executes the agent's tools directly on the host — handy for verifying the dataset → solver → scorer pipeline before standing up Docker. Pair it with Inspect's mockllm to run with no Docker and no API key at all:

inspect eval src/agentsec/tasks/secure_coding.py -T sandbox_type=local \
    --model mockllm/model --display plain
# ... or with a real local model:
inspect eval src/agentsec/tasks/defensive.py -T sandbox_type=local --model ollama/llama3.1

⚠️ Local mode runs agent-generated commands on your machine — it's a smoke-test convenience, gated to the benign axes only. The offensive and harness_safety axes are Docker-only by design (they need live targets and network isolation). For real scored runs, use Docker.

Requirements

• Python ≥ 3.11
• Docker (Docker Desktop on Windows/macOS) running locally for real runs — every task provisions its target(s) in a sandbox. (See the local smoke-test above to try the pipeline without it.) Architected to scale to Kubernetes later (eval.yaml/compose.yaml are kept k8s-compatible).

Layout

src/agentsec/
  tasks/        one Inspect @task per capability axis
  solvers/      builtin_agent (raw API baseline) + bridge_harness (external CLIs)
  scorers/      flag, test_suite, static_scan, judge, composite (hybrid)
  registry.py   harness registry (cmd, env, image) for -S harness=<name>
  cli.py        config presets over `inspect eval`
  challenges/   data: one subdir per task instance (challenge.json + compose.yaml + assets)
                (bundled in the wheel; override location with AGENTSEC_CHALLENGES_DIR)
tests/          pytest: scorer unit tests + judge calibration

Interpreting results

Scores are per-axis means in [0, 1] (partial credit on the hard/stretch tiers). Browse full transcripts with inspect view, and render a comparison table across runs with:

python -m agentsec.cli report ./logs   # axis x harness x model x difficulty x score

Compare models by fixing the harness and varying --model; compare harnesses by fixing --model and varying the bridged harness (see the differentiator section). For a stable mean on stochastic challenges, use --epochs N.

Contributing

New challenges need no code — drop a directory under src/agentsec/challenges/<axis>/<name>/ with a challenge.json (+ compose.yaml and assets) and it's picked up automatically. See CHALLENGES.md for the schema, the subtask/partial-credit format, and the no-egress rules. Run pytest -q and ruff check src tests before opening a PR. Contributions of harder challenges (especially stretch-tier) and additional harness definitions are welcome.

Safety & responsible use

• All offensive and harness_safety challenges are self-contained, intentionally-vulnerable, network-isolated sandbox targets — internal: true networks with no host or internet egress, enforced by tests/test_no_egress.py.
• sandbox_type=local executes agent-generated commands on your machine; it is gated to the benign coding axes and intended only for pipeline smoke-testing. Use Docker for real runs.
• AgentSec-Bench is intended for authorized evaluation of AI agents' security capabilities and safety — measuring and improving model/harness behavior. It is not a tool for attacking systems you don't own. Use it accordingly.

Acknowledgements

Built on Inspect AI by the UK AI Safety Institute. The offensive axis and challenge layout are inspired by Cybench and the inspect_evals cyber suite.

Citation

@software{agentsec_bench,
  title  = {AgentSec-Bench: A Security Benchmark for LLM Agents and Harnesses},
  author = {Santhosraj},
  year   = {2026},
  url    = {https://github.com/Santhosraj/AgentSec-Bench}
}

License

MIT © 2026 Santhosraj. AgentSec-Bench is provided for research and authorized security evaluation; you are responsible for using it lawfully.