gsobench 0.1.1

Add your description here

---

GSO: Challenging Software Optimization Tasks for Evaluating SWE-Agents

GSO (Global Software Optimization) is a benchmark for evaluating language models' capabilities in developing high-performance software. We present 100+ challenging optimization tasks across 10 codebases spanning diverse domains and programming languages. Each task provides a codebase and performance test as a precise specification, with agents required to optmize the codebase and measured against expert developer commits.

📰 News

• [May 30, 2025]: 🤗 GSO dataset is now available on HuggingFace! Access it at gso-bench/gso.
• [May 30, 2025]: Prebuilt docker images for GSO tasks are now available on Docker Hub.
• [May 30, 2025]: Initial release of the GSO benchmark: gso-bench.github.io

👋 Overview

GSO evaluates language models on software performance optimization. Each task provides:

• A codebase with a specific performance bottleneck
• A performance test as a precise specification
• An agent must generate a patch that improves runtime efficiency
• Success is measured against expert developer optimizations

To access GSO, copy and run the following code:

from datasets import load_dataset
gso = load_dataset('gso-bench/gso', split='test')

🚀 Setup

curl -LsSf https://astral.sh/uv/install.sh | sh
source $HOME/.local/bin/env

git clone --recursive https://github.com/gso-bench/gso.git
cd gso && uv venv && source .venv/bin/activate
uv sync

(Additional) Setup HuggingFace token:

export HF_TOKEN="huggingface_token"

💽 Usage

Evaluation Harness

1. Building Dockers for GSO tasks:

docker login

uv run src/gso/harness/prepare_images.py \
    --push_to_registry True \
    --dockerhub_username <dockerhub_username> \
    --dockerhub_repo <dockerhub_repo>

2. Running Evaluations:

uv run src/gso/harness/opt_at_k.py \
    --prediction_paths <prediction_paths> \
    --timeout 3600 \
    --run_id <run_id> \
    --k 10 \
    --model <modelname>

For detailed instructions and options, see the Harness documentation.

GSO Collection Framework

The collection framework enables you to create your own GSO tasks through a four-step pipeline:

1. Commit Extraction & Filtering: Extract performance-related commits using LLMs
2. API Identification: Identify affected high-level APIs for each commit
3. Performance Test Generation: Generate tests for API-Commit pairs
4. Test Execution: Execute tests to identify performance improvements

For detailed instructions and usage, see the Collection Framework documentation.

⬇️ Artifacts

Datasets	Tools	Dockers
💿 GSO	🔧 Evaluation Harness	🐳 Docker Hub
	🔧 Collection Framework

💫 Contributions

We welcome contributions from the broader NLP, Machine Learning, and Software Engineering research communities! Please file a new pull request or issue and fill in the corresponding templates accordingly.

✍️ Citation & license

MIT license. Check LICENSE file.