Skip to main content

The official SWE-bench package - a benchmark for evaluating LMs on software engineering

Project description

Kawi the SWE-Llama


Code and data for our ICLR 2024 paper SWE-bench: Can Language Models Resolve Real-World GitHub Issues?

Build License

Please refer our website for the public leaderboard and the change log for information on the latest updates to the SWE-bench benchmark.

📰 News

  • [Aug. 13, 2024]: Introducing SWE-bench Verified! Part 2 of our collaboration with OpenAI Preparedness. A subset of 500 problems that real software engineers have confirmed are solvable. Check out more in the report!
  • [Jun. 27, 2024]: We have an exciting update for SWE-bench - with support from OpenAI's Preparedness team: We're moving to a fully containerized evaluation harness using Docker for more reproducible evaluations! Read more in our report.
  • [Apr. 15, 2024]: SWE-bench has gone through major improvements to resolve issues with the evaluation harness. Read more in our report.
  • [Apr. 2, 2024]: We have released SWE-agent, which sets the state-of-the-art on the full SWE-bench test set! (Tweet 🔗)
  • [Jan. 16, 2024]: SWE-bench has been accepted to ICLR 2024 as an oral presentation! (OpenReview 🔗)

👋 Overview

SWE-bench is a benchmark for evaluating large language models on real world software issues collected from GitHub. Given a codebase and an issue, a language model is tasked with generating a patch that resolves the described problem.

To access SWE-bench, copy and run the following code:

from datasets import load_dataset
swebench = load_dataset('princeton-nlp/SWE-bench', split='test')

🚀 Set Up

SWE-bench uses Docker for reproducible evaluations. Follow the instructions in the Docker setup guide to install Docker on your machine. If you're setting up on Linux, we recommend seeing the post-installation steps as well.

Finally, to build SWE-bench from source, follow these steps:

git clone git@github.com:princeton-nlp/SWE-bench.git
cd SWE-bench
pip install -e .

Test your installation by running:

python -m swebench.harness.run_evaluation \
    --predictions_path gold \
    --max_workers 1 \
    --instance_ids sympy__sympy-20590 \
    --run_id validate-gold

💽 Usage

[!WARNING] Running fast evaluations on SWE-bench can be resource intensive We recommend running the evaluation harness on an x86_64 machine with at least 120GB of free storage, 16GB of RAM, and 8 CPU cores. You may need to experiment with the --max_workers argument to find the optimal number of workers for your machine, but we recommend using fewer than min(0.75 * os.cpu_count(), 24).

If running with docker desktop, make sure to increase your virtual disk space to have ~120 free GB available, and set max_workers to be consistent with the above for the CPUs available to docker.

Support for arm64 machines is experimental.

Evaluate model predictions on SWE-bench Lite using the evaluation harness with the following command:

python -m swebench.harness.run_evaluation \
    --dataset_name princeton-nlp/SWE-bench_Lite \
    --predictions_path <path_to_predictions> \
    --max_workers <num_workers> \
    --run_id <run_id>
    # use --predictions_path 'gold' to verify the gold patches
    # use --run_id to name the evaluation run

This command will generate docker build logs (logs/build_images) and evaluation logs (logs/run_evaluation) in the current directory.

The final evaluation results will be stored in the evaluation_results directory.

To see the full list of arguments for the evaluation harness, run:

python -m swebench.harness.run_evaluation --help

Additionally, the SWE-Bench repo can help you:

  • Train your own models on our pre-processed datasets
  • Run inference on existing models (either models you have on-disk like LLaMA, or models you have access to through an API like GPT-4). The inference step is where you get a repo and an issue and have the model try to generate a fix for it.
  • Run SWE-bench's data collection procedure on your own repositories, to make new SWE-Bench tasks.

⬇️ Downloads

Datasets Models
🤗 SWE-bench 🦙 SWE-Llama 13b
🤗 "Oracle" Retrieval 🦙 SWE-Llama 13b (PEFT)
🤗 BM25 Retrieval 13K 🦙 SWE-Llama 7b
🤗 BM25 Retrieval 27K 🦙 SWE-Llama 7b (PEFT)
🤗 BM25 Retrieval 40K
🤗 BM25 Retrieval 50K (Llama tokens)

🍎 Tutorials

We've also written the following blog posts on how to use different parts of SWE-bench. If you'd like to see a post about a particular topic, please let us know via an issue.

  • [Nov 1. 2023] Collecting Evaluation Tasks for SWE-Bench (🔗)
  • [Nov 6. 2023] Evaluating on SWE-bench (🔗)

💫 Contributions

We would love to hear from the broader NLP, Machine Learning, and Software Engineering research communities, and we welcome any contributions, pull requests, or issues! To do so, please either file a new pull request or issue and fill in the corresponding templates accordingly. We'll be sure to follow up shortly!

Contact person: Carlos E. Jimenez and John Yang (Email: carlosej@princeton.edu, johnby@stanford.edu).

✍️ Citation

If you find our work helpful, please use the following citations.

@inproceedings{
    jimenez2024swebench,
    title={{SWE}-bench: Can Language Models Resolve Real-world Github Issues?},
    author={Carlos E Jimenez and John Yang and Alexander Wettig and Shunyu Yao and Kexin Pei and Ofir Press and Karthik R Narasimhan},
    booktitle={The Twelfth International Conference on Learning Representations},
    year={2024},
    url={https://openreview.net/forum?id=VTF8yNQM66}
}

🪪 License

MIT. Check LICENSE.md.

Project details


Release history Release notifications | RSS feed

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

swebench-2.1.3.tar.gz (91.7 kB view details)

Uploaded Source

Built Distribution

swebench-2.1.3-py3-none-any.whl (102.8 kB view details)

Uploaded Python 3

File details

Details for the file swebench-2.1.3.tar.gz.

File metadata

  • Download URL: swebench-2.1.3.tar.gz
  • Upload date:
  • Size: 91.7 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/5.0.0 CPython/3.12.1

File hashes

Hashes for swebench-2.1.3.tar.gz
Algorithm Hash digest
SHA256 e4e7bb9169714929ce2b78e2a4c4f07e938f3b98db59847b5b23b75e5a539fbc
MD5 69d39100f2661bde7e525c1361c54a8a
BLAKE2b-256 049182ffe8db72768fd6380f534e8accde63b504c739b6a28b616cd23ca0243e

See more details on using hashes here.

File details

Details for the file swebench-2.1.3-py3-none-any.whl.

File metadata

  • Download URL: swebench-2.1.3-py3-none-any.whl
  • Upload date:
  • Size: 102.8 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/5.0.0 CPython/3.12.1

File hashes

Hashes for swebench-2.1.3-py3-none-any.whl
Algorithm Hash digest
SHA256 d0f262127d05a2db47dae94ac583ad344384d32fdc4b823a5ea2e560c26d19bc
MD5 7c363e84fc2ab97851bd01e619d2f986
BLAKE2b-256 b1a5e65a1101d26baa23884756c79aec6aec1dd61456be9d311612ff67794a46

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page