benchspan 0.1.0

Run agentic benchmarks end-to-end

Benchspan

Run agentic benchmarks end-to-end. One CLI to evaluate any AI agent against 40+ benchmarks with parallel cloud execution, reproducible environments, and a team dashboard.

Install

pip install benchspan

Requires Python 3.11+.

Quick start

# Login to your Benchspan account
benchspan login

# Scaffold a new agent
benchspan init ./my-agent

# Edit my-agent/runner.sh to invoke your agent, then run:
benchspan run --benchmark swebench --agent ./my-agent --instances 10

Your agent runs in parallel Docker containers on Benchspan's infrastructure. Results stream to your terminal and appear on the dashboard.

Built-in agents

Benchspan ships with pre-configured agents you can use immediately:

# List available agents
benchspan agents

# Run with Claude Code
benchspan run --benchmark swebench --agent claude-code --instances 10

# Run with OpenHands
benchspan run --benchmark swebench --agent openhands --instances 10

Set the required API keys on your dashboard under Profile > Secrets.

Writing an agent

An agent is a directory with a runner.sh. Benchspan injects it into a benchmark container and runs it:

#!/bin/bash
# Available environment variables:
#   $PROBLEM_STATEMENT  — what to solve (natural language)
#   $WORKING_DIR        — where to do your work
#   $OUTPUT_DIR         — where to write outputs
#   $INSTANCE_ID        — unique instance identifier

cd "$WORKING_DIR"

# Invoke your agent here

If your agent can read a problem statement and write code, it works on Benchspan. See the full guide for patterns and best practices.

Benchmarks

40+ benchmarks available out of the box:

# List all benchmarks
benchspan benchmarks list

# List instances for a benchmark
benchspan benchmarks instances swebench

# Run with a subset
benchspan run --benchmark swebench.django --agent ./my-agent

Includes SWE-bench, HumanEval, Terminal-Bench, BFCL, SimpleQA, and many more.

Managing runs

# List your runs
benchspan runs list

# List all team runs
benchspan runs list --all

# Show run details
benchspan runs show <run_id>

# Compare two runs side-by-side
benchspan runs compare <run_a> <run_b>

# Watch a running benchmark
benchspan runs watch <run_id>

# Cancel a run
benchspan runs cancel <run_id>

# Retry failed instances
benchspan retry <run_id> --agent ./my-agent

Key features

• Parallel execution — hundreds of instances run simultaneously, not sequentially
• Resume on failure — retry only failed instances, don't re-run the whole suite
• Reproducible — every instance runs in an identical Docker container
• Team dashboard — all results in one place at benchspan.com
• Any agent — if it runs via bash, it works on Benchspan
• Private benchmarks — bring your own proprietary benchmarks

Documentation

Full documentation at docs.benchspan.com, including:

• Quickstart — up and running in 5 minutes
• Writing runner.sh — agent integration patterns
• Healthcheck testing — validate your agent before running real benchmarks
• CLI reference — all commands and options
• Troubleshooting — common issues and fixes

Links

• Website
• Documentation
• Book a demo
• Email