clawsbench 0.2.0

ClawsBench: Evaluating Capability and Safety of LLM Productivity Agents in Simulated Workspaces

ClawsBench

Evaluating Capability and Safety of LLM Productivity Agents in Simulated Workspaces

ClawsBench evaluates LLM agents on realistic productivity tasks across 5 high-fidelity mock services (Gmail, Calendar, Docs, Drive, Slack), measuring both capability (task success) and safety (harmful action prevention).

• 44 tasks: 30 single-service + 14 cross-service, including 24 safety-critical scenarios
• 6 models: Claude Opus 4.6, Claude Sonnet 4.6, GPT-5.4, Gemini 3.1 Pro, Gemini 3.1 Flash-Lite, GLM-5
• 4 harnesses: OpenClaw, Claude Code, Codex, Gemini CLI
• 33 conditions, 7,224 trials

Links

• GitHub: https://github.com/benchflow-ai/ClawsBench
• Paper: https://arxiv.org/abs/2604.05172
• Dataset: https://huggingface.co/datasets/benchflow/ClawsBench
• Website: https://clawsbench.benchflow.ai
• Discord: https://discord.gg/mZ9Rc8q8W3

Citation

@misc{li2026clawsbenchevaluatingcapabilitysafety,
      title={ClawsBench: Evaluating Capability and Safety of LLM Productivity Agents in Simulated Workspaces}, 
      author={Xiangyi Li and Kyoung Whan Choe and Yimin Liu and Xiaokun Chen and Chujun Tao and Bingran You and Wenbo Chen and Zonglin Di and Jiankai Sun and Shenghan Zheng and Jiajun Bao and Yuanli Wang and Weixiang Yan and Yiyuan Li and Han-chung Lee},
      year={2026},
      eprint={2604.05172},
      archivePrefix={arXiv},
      primaryClass={cs.AI},
      url={https://arxiv.org/abs/2604.05172}, 
}