agent-safety-bench-envs

Tool environments for the Agent Safety Bench (ASB) benchmark. Each environment exposes a small set of mock tools (Python callables described by JSON schemas) that an LLM agent can call. ASB pairs these environments with prompts to evaluate agent safety under prompt-injection and other adversarial conditions.

The companion prompt dataset lives on the Hugging Face Hub: aradhye/asb.

Install

pip install agent-safety-bench-envs

Quickstart

from asb_envs import EnvManager

mgr = EnvManager()
env = mgr.init_env("AccessControl_1", env_params=None)

print(env.tool_list)
print(env.get_tool_descs(["list_access_attempts"]))

result = env.call_tool("list_access_attempts", {})
print(result)

init_env(name, env_params) dynamically imports the env class and instantiates it. Each env is a subclass of BaseEnv whose tools are auto-validated against the {ClassName}.json schema shipped with the package.

Schema

Every env class has a sibling JSON file declaring its tools:

[
  {
    "name": "tool_name",
    "description": "...",
    "parameters": {
      "type": "object",
      "properties": { "...": { "type": "string" } },
      "required": ["..."]
    }
  }
]

BaseEnv.call_tool(name, args) validates required params, coerces basic types, and dispatches to the matching method on the env class.

Citation

If you use these environments, please cite the ASB paper (ICML 2026):

@inproceedings{asb2026,
  title={...},
  author={...},
  booktitle={ICML},
  year={2026}
}

License

Apache 2.0. The vendored functions_runtime.py is from agentdojo and is MIT-licensed — see NOTICE.