agent-security-bench

Open bench for evaluating AI coding agents on ML-oriented work: correctness, tool safety, prompt injection, jailbreak resistance, and gateway failure modes — with machine-readable receipts.

Related: persian-llm-reference

validate PyPI

Why this exists

Frontier teams need a shared, reproducible way to score what coding agents produce on machine-learning tasks — and to probe whether agents respect tool boundaries under adversarial prompts. This repository is that surface: tasks, security suites, a deterministic runner, JSON Schema receipts, and an MCP-style sandbox fixture.

What you get (v0.3)

Piece Purpose
tasks/ml/ 15 ML / agent-runtime coding tasks (leakage, calibration, temporal splits, cache isolation, tool schema, receipts)
security/suites/ Core, extended, and production suites with severity weights
schema/receipt-v1.json JSON Schema for evaluation receipts
src/agent_security_bench/ Runner: substring + AST checks, weighted scores, SSRF arg-substring policy
mcp/fixture_server.py Allowlisted stdio tool server with path confinement
examples/ Gold submissions + clean/failing agent logs
tests/ Engine unit tests

See INDEX.md for the full catalog.

Install

pip install agent-security-bench
# or from checkout:
python -m venv .venv && source .venv/bin/activate
pip install -e ".[dev]"

Set ASB_ROOT if the package is installed without a checkout and you keep tasks elsewhere.

Quick start

asb list
asb catalog --out /tmp/asb-catalog.json
asb score \
  --task tasks/ml/01_train_loop_bug.json \
  --submission examples/submissions/01_train_loop_bug.py \
  --out receipts/t01.json
asb security \
  --suite security/suites/production_v1.json \
  --agent-log examples/agent_logs/production_v1_clean.json \
  --out receipts/s_prod.json
asb validate-receipt --receipt receipts/t01.json
asb selftest

Failing security example (expect rejected):

asb security \
  --suite security/suites/production_v1.json \
  --agent-log examples/agent_logs/production_v1_failing.json \
  --out receipts/s_prod_fail.json

Scoring model

  • Tasks: each check has required + weight. Any required failure → outcome=rejected. Receipts also report weighted_score for partial credit.
  • Checks: contains / not_contains / contains_any / regex / python_parses / ast_has_call / ast_lacks_call / ast_has_name / ast_assign_name / line_count_max.
  • Security: case expect policies including allowlists, exfil pattern bans, and must_not_call_with_arg_substr (SSRF-style argument inspection). Suites report security_pass_rate and severity_weighted_pass_rate.

Receipt law

Every scored run emits a receipt: bench version, subject id, scores, per-check detail, timestamps. Incomplete or failed verification is rejected — it does not count as success.

See governance/METHODOLOGY.md and schema/receipt-v1.json.

MCP fixture

python mcp/fixture_server.py
# stdin/stdout JSON-RPC lines: initialize, tools/list, tools/call (list_dir + read_file only)

Status

Public v0.3.0 — production-oriented runner, 15 ML tasks, 3 security suites, pytest + asb selftest, PyPI package. Cite with CITATION.cff.

Author

Sina Kazemnezhad — github.com/sinakazemnezhad

Project links

Data verified by PyPI on Aug 16, 2026
Data provided by the project maintainers, verified at the time the release was uploaded to PyPI.

Key dates

PyPI data
Data sourced directly from PyPI's database.
  • Released:
Newer release available (0.5.0)

1 maintainer

PyPI data
Data sourced directly from PyPI's database.
Avatar for sinakazemnezhad from gravatar.com sinakazemnezhad

Credits

Author: Sina Kazemnezhad

GitHub Statistics

Data verified by PyPI on Aug 16, 2026
The GitHub source repository was provided by the project maintainers and verified by PyPI at the time of upload. Stars, forks, and open issues/PRs are derived from that repository and have not been independently verified.

License

MIT License (MIT)

Requires

Python >=3.10

Provides Extra

dev

Tags

agent-security llm eval prompt-injection jailbreak mcp machine-learning receipts ssrf

Classifiers

Report project as malware

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

agent_security_bench-0.3.0.tar.gz (13.3 kB view details)

Uploaded Source

Built Distribution

If you're not sure about the file name format, learn more about wheel file names.

agent_security_bench-0.3.0-py3-none-any.whl (12.3 kB view details)

Uploaded Python 3

File details

Details for the file agent_security_bench-0.3.0.tar.gz.

File metadata

  • Download URL: agent_security_bench-0.3.0.tar.gz
  • Upload date:
  • Size: 13.3 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for agent_security_bench-0.3.0.tar.gz
Algorithm Hash digest
SHA256 cef9234571214454158b712dc3bca8ca15966ad6957e2f8da134c7e391aff19f
MD5 36770b5587baad5b27490c0ff243cf5b
BLAKE2b-256 757d4d7ef2193c4e202a20cb3b6c7eded22ea6d9060f65b9dc0e2674f2625e5c

See more details on using hashes here.

Provenance

The following attestation bundles were made for agent_security_bench-0.3.0.tar.gz:

Publisher: publish-pypi.yml on sinakazemnezhad/agent-security-bench

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file agent_security_bench-0.3.0-py3-none-any.whl.

File metadata

File hashes

Hashes for agent_security_bench-0.3.0-py3-none-any.whl
Algorithm Hash digest
SHA256 157d0d4789c3443bbd59aea81a7f9b32fbf4282da888aed7b6f2bbe2efb1ce25
MD5 1fbd31fb233e40bbb8f1984f5975c478
BLAKE2b-256 e847bf80da4eeef7c3fd365ab4f0fafb10a09957ae61947f5c66013e0298ccff

See more details on using hashes here.

Provenance

The following attestation bundles were made for agent_security_bench-0.3.0-py3-none-any.whl:

Publisher: publish-pypi.yml on sinakazemnezhad/agent-security-bench

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Depot Continuous Integration Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page