CooperBench

Can AI agents work together as teammates? CooperBench is the first benchmark designed to measure how well AI agents can cooperate when handling individual tasks with potential conflicts.

We find that coordinating agents perform much worse than a single agent given the same total workload. This coordination deficit presents a fundamental barrier to deploying AI systems that can work alongside humans or other agents.

Installation

pip install cooperbench

For development:

git clone https://github.com/cooperbench/CooperBench.git
cd CooperBench
pip install -e ".[dev]"

Requirements

• Python 3.12+
• Execution Backend (choose one):
  • Modal (default, cloud-based)
  • GCP (Google Cloud Platform)
  • Docker (local execution)
• Redis (for inter-agent communication in coop mode)

Setup

Option 1: Modal (Default)

1. Modal: Sign up at modal.com and run modal setup
2. Redis: Run locally with docker run -p 6379:6379 redis:7 or use a cloud provider
3. LLM API keys: Set in .env file:

ANTHROPIC_API_KEY=your_key
OPENAI_API_KEY=your_key
GEMINI_API_KEY=your_key

Option 2: GCP (Recommended for Scale)

Prerequisites: Install gcloud CLI first

• macOS: brew install google-cloud-sdk
• Linux: curl https://sdk.cloud.google.com | bash

Setup:

# 1. Install GCP dependencies
pip install 'cooperbench[gcp]'

# 2. Run configuration wizard (handles authentication, project setup, validation)
cooperbench config gcp

# 3. You're ready to run experiments!
cooperbench run --backend gcp -s lite

Also needed: Redis and LLM API keys (same as Option 1)

See GCP Setup Guide for detailed instructions.

Dataset

Download the benchmark dataset from HuggingFace into ./dataset:

cooperbench prepare

Quick Start

CLI

Run agents on a task:

# Run cooperative agents (N peers, shared Redis messaging)
cooperbench run -n my-experiment -r llama_index_task -m gpt-4o

# Run solo agent (1 agent handling both features)
cooperbench run -n my-experiment -r llama_index_task -m gpt-4o --setting solo

# Run team mode (lead + members, shared task list + scratchpad)
cooperbench run -n my-experiment -r llama_index_task -m gpt-4o --setting team

# Evaluate results
cooperbench eval -n my-experiment

Python API

from cooperbench import run, evaluate

# Run agents
run(
    run_name="my-experiment",
    repo="llama_index_task",
    model_name="gpt-4o",
    setting="coop",  # or "solo", "team"
)

# Evaluate patches
evaluate(run_name="my-experiment")

Settings: solo / coop / team

CooperBench supports three settings, selected via --setting:

• solo — one agent implements every feature in the task. The agent works alone in a single container; no Redis, no git server.
• coop — N peer agents, each assigned one feature. Containers talk via Redis (coop-send / coop-recv shell commands inside the container, auto-injected as user messages in the Python-loop adapters). Optional --git enables a shared team git remote so agents can fetch/merge each other's branches.
• team — N agents organized as one lead + N-1 members, with a Redis-backed shared task list (atomic claim via coop-task-claim), a shared scratchpad volume mounted at /workspace/shared in every container, and role-specific system prompts. The lead's prompt directs them to organize work via coop-task-create / coop-task-list before coding; members claim open tasks and report progress via coop-task-update.

Team-mode result.json includes a metrics block with coordination indicators computed from the task-list audit log: tasks_total, tasks_done, unowned_at_end, time_to_first_claim_seconds, claims_per_agent, updates_per_agent.

All three settings are evaluated the same way (cooperbench eval): per-feature tests against either the single agent's patch (solo) or each agent's individual patch (coop / team).

Running with Harbor

CooperBench is also available as a Harbor adapter, which provides parallelized cloud execution on Modal via Docker-in-Docker sandboxes, built-in oracle validation, and standardized result collection.

Requires Modal authentication (run modal setup) and an LLM API key.

Install and Prepare Tasks

# Install Harbor
uv tool install harbor

# Clone Harbor and prepare the adapter
git clone https://github.com/harbor-framework/harbor.git
cd harbor/adapters/cooperbench
uv sync

# Generate flash subset (50 pairs) with openhands-sdk harness
uv run python -m cooperbench.main \
  --subset flash \
  --agent-harness openhands-sdk \
  --output-dir ../../datasets/cooperbench

Run on Modal

cd ../..  # back to harbor root

# Set up .env with your API key
echo "GEMINI_API_KEY=your_key" > .env

# Run the flash subset (50 tasks, concurrency 10)
uv run harbor run -p datasets/cooperbench --agent nop -e modal \
  --env-file .env --n-concurrent 10 \
  --ae COOPERBENCH_MODEL=gemini/gemini-3-flash-preview

# Run oracle (validates infrastructure, expects 100% pass)
uv run harbor run -p datasets/cooperbench --agent oracle -e modal \
  --env-file .env --n-concurrent 28

See the Harbor CooperBench adapter for full documentation.

CLI Reference

cooperbench config

Configure execution backends (GCP, Modal, etc.).

# Configure GCP backend
cooperbench config gcp

# Skip validation tests for faster setup
cooperbench config gcp --skip-tests

See GCP Setup Guide for details.

cooperbench run

Run agents on benchmark tasks.

cooperbench run -n NAME [OPTIONS]

Agent Configuration: Pass agent-specific parameters via a config file. CooperBench forwards the file path to your agent without parsing it.

cooperbench eval

Evaluate completed runs.

cooperbench eval -n NAME [OPTIONS]

Experiment Settings

Dataset Structure

dataset/
  <repo_name>/
    task<id>/
      setup.sh          # Repository setup script
      run_tests.sh      # Test runner script
      feature1/
        feature.md      # Feature description
        feature.patch   # Golden implementation
        tests.patch     # Test cases
      feature2/
        ...

Output Structure

Results are saved to logs/:

logs/<run_name>/<repo>/task<id>/features_<i>_<j>/
  agent1/
    trajectory.json     # Full agent trajectory
    patch.diff          # Generated patch
  agent2/
    ...
  eval.json             # Evaluation results

Benchmark Statistics

Key Findings

1. Agents perform worse together than alone — GPT-5 and Claude Sonnet 4.5 achieve only 25% success with two-agent cooperation, roughly 50% lower than when a single agent handles both tasks.

2. Communication reduces conflicts but not failures — Agents spend up to 20% of their budget on communication, reducing merge conflicts but not improving overall success.

3. Three capability gaps underlie coordination failures:

   • Expectation failures (42%) — agents fail to integrate partner state information
   • Communication failures (26%) — questions go unanswered, breaking decision loops
   • Commitment failures (32%) — agents break promises or make unverifiable claims

Development

# Install dev dependencies
pip install -e ".[dev]"

# Run tests
pytest tests/ -v

# Run integration tests (requires Modal)
pytest tests/ -v --run-modal

# Lint
ruff check src/
ruff format src/

# Type check
mypy src/cooperbench/

Citation

@article{cooperbench2026,
  title={CooperBench: Why Coding Agents Cannot be Your Teammates Yet},
  author={Khatua*, Arpandeep and Zhu*, Hao and Tran†, Peter and Prabhudesai†, Arya
          and Sadrieh†, Frederic and Lieberwirth†, Johann K. and Yu, Xinkai
          and Fu, Yicheng and Ryan, Michael J. and Pei, Jiaxin and Yang, Diyi},
  journal={arXiv preprint},
  year={2026},
  url={https://arxiv.org/abs/2601.13295},
  note={*Equal contribution (Stanford) · †Equal contribution (SAP Labs)}
}

License

MIT