SecureVibeBench

✨ News | 🛠️ Quick Start | 🔭 Overview | 📚 Citation | 🙏 Acknowledgments

✨ News

• [2026-10-29] We updated to v0.2, introducing a PyPI package, Harbor-format tasks, and unified model generation and evaluation. The original version is now referred to as v0.1; results may differ between versions.

• [2026-04-07] 🎉 Our paper has been accepted to ACL 2026.

🛠️ Quick Start

Requirements: Python 3.12+, Linux x86_64, Docker Engine with Compose, and a running Docker daemon.

Install

We recommend uv for managing Python environments. pip also works; uv is not installed as a package dependency.

# Run without installing into your current environment
uvx securevibebench --help

# Or install into an active virtual environment
uv pip install securevibebench

# Or with pip
pip install securevibebench

Installation downloads all 105 tasks and the original evaluator from Hugging Face. Task containers are prepared when evaluation starts.

For development

git clone https://github.com/iCSawyer/SecureVibeBench.git
cd SecureVibeBench
uv sync  # editable install + development tools

Use uv run securevibebench eval ... for the development checkout, or activate .venv/bin/activate.

Without uv, use python tools/install.py --dev --installer pip in an active virtual environment.

Run evaluation

export OPENAI_API_KEY='<your-api-key>'
export MODEL='openai/<model-name>'

# With uv, without a persistent installation
uvx securevibebench eval --model "$MODEL" --task 992

# Or after installing the package
securevibebench eval --model "$MODEL" --task 992

Omit --task 992 to run all 105 tasks. Static analysis is disabled by default. Enable it with --run-sast TRUE and configure Semgrep credentials such as SEMGREP_APP_TOKEN. Results are saved to runs/<job>/summary.json under the current working directory.

🔭 Overview

SecureVibeBench is the first SWE-bench-level benchmark for secure vibe coding of agents, consisting of 105 C/C++ coding tasks sourced from real vulnerabilities (OSS-Fuzz/ARVO) covering various projects.

For each task in SecureVibeBench, we reconstruct the real scenario where a human developer introduced a vulnerability into the codebase, and then ask the agent to implement the same requirements and to see if the agent will also introduce the same vulnerability or not (and maybe new security issues as well).

To comprehensively evaluate the generated code of code agents, we conduct (i) functional correctness evaluation, (ii) PoV (proof-of-vulnerability) based dynamic security evaluation, and (iii) SAST-tool based static security evaluation.

📚 Citation

If you find SecureVibeBench useful, please cite:

@inproceedings{chen2026securevibebench,
  title     = "{S}ecure{V}ibe{B}ench: Benchmarking Secure Vibe Coding of {AI} Agents via Reconstructing Vulnerability-Introducing Scenarios",
  author    = "Chen, Junkai and Huang, Huihui and Lyu, Yunbo and An, Junwen and
               Shi, Jieke and Yang, Chengran and Zhang, Ting and Tian, Haoye and
               Li, Yikun and Li, Zhenhao and Zhou, Xin and Hu, Xing and Lo, David",
  editor    = "Liakata, Maria and Moreira, Viviane P. and Zhang, Jiajun and Jurgens, David",
  booktitle = "Proceedings of the 64th Annual Meeting of the {A}ssociation for {C}omputational {L}inguistics (Volume 1: Long Papers)",
  year      = "2026", month = jul, pages = "24144--24168",
  address   = "San Diego, California, United States",
  publisher = "Association for Computational Linguistics",
  url       = "https://aclanthology.org/2026.acl-long.1107/",
  ISBN      = "979-8-89176-390-6"
}

🙏 Acknowledgments

Our work builds on OSS-Fuzz and ARVO.

@misc{mei2024arvoatlasreproduciblevulnerabilities,
  title         = {ARVO: Atlas of Reproducible Vulnerabilities for Open Source Software},
  author        = {Xiang Mei and Pulkit Singh Singaria and Jordi Del Castillo and Haoran Xi and
                   Abdelouahab and Benchikh and Tiffany Bao and Ruoyu Wang and Yan Shoshitaishvili and
                   Adam Doupé and Hammond Pearce and Brendan Dolan-Gavitt},
  year          = {2024}, eprint = {2408.02153},
  archivePrefix = {arXiv}, primaryClass = {cs.CR},
  url           = {https://arxiv.org/abs/2408.02153}
}

@article{serebryany2017oss,
  title  = {$\{$OSS-Fuzz$\}$-Google's continuous fuzzing service for open source software},
  author = {Serebryany, Kostya},
  year   = {2017}
}