SWE-bench CLI
A command-line interface for interacting with the SWE-bench API. Use this tool to submit predictions, manage runs, and retrieve evaluation reports.
Read the full documentation here. For submission guidelines, see here.
Installation
pip install sb-cli
Authentication
Before using the CLI, you'll need to get an API key:
- Generate an API key:
sb-cli gen-api-key your.email@example.com
- Set your API key as an environment variable - and store it somewhere safe!
export SWEBENCH_API_KEY=your_api_key
# or add export SWEBENCH_API_KEY=your_api_key to your .*rc file
- You'll receive an email with a verification code. Verify your API key:
sb-cli verify-api-key YOUR_VERIFICATION_CODE
Subsets and Splits
SWE-bench has different subsets and splits available:
Subsets
swe-bench-m: The main datasetswe-bench_lite: A smaller subset for testing and developmentswe-bench_verified: 500 verified problems from SWE-bench Learn more
Splits
dev: Development/validation splittest: Test split (currently only available forswe-bench_liteandswe-bench_verified)
You'll need to specify both a subset and split for most commands.
Usage
Submit Predictions
Submit your model's predictions to SWE-bench:
sb-cli submit swe-bench-m test \
--predictions_path predictions.json \
--run_id my_run_id
Options:
--run_id: ID of the run to submit predictions for (optional, defaults to the name of the parent directory of the predictions file)--instance_ids: Comma-separated list of specific instance IDs to submit (optional)--output_dir: Directory to save report files (default: sb-cli-reports)--overwrite: Overwrite existing report (default: 0)--gen_report: Generate a report after evaluation is complete (default: 1)
Get Report
Retrieve evaluation results for a specific run:
sb-cli get-report swe-bench-m dev my_run_id -o ./reports
List Runs
View all your existing run IDs for a specific subset and split:
sb-cli list-runs swe-bench-m dev
Predictions File Format
Your predictions file should be a JSON file in one of these formats:
{
"instance_id_1": {
"model_patch": "...",
"model_name_or_path": "..."
},
"instance_id_2": {
"model_patch": "...",
"model_name_or_path": "..."
}
}
Or as a list:
[
{
"instance_id": "instance_id_1",
"model_patch": "...",
"model_name_or_path": "..."
},
{
"instance_id": "instance_id_2",
"model_patch": "...",
"model_name_or_path": "..."
}
]
Submitting to the Multimodal Leaderboard
To submit your system to the SWE-bench Multimodal leaderboard:
- Submit your predictions for the
swe-bench-m/testsplit using the CLI - Fork the experiments repository
- Add your submission files under
experiments/multimodal/YOUR_MODEL_NAME/ - Create a PR with your submission
See the detailed guide in our submission documentation.
Note: Check your test split quota using sb-cli quota swe-bench-m test before submitting.
Related projects
Metadata
Release files for sb-cli 0.1.5
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| sb_cli-0.1.5.tar.gz | 590.4 kB | Without provenance | Details for sb_cli-0.1.5.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| sb_cli-0.1.5-py3-none-any.whl | Python 3 | none | any | Details for sb_cli-0.1.5-py3-none-any.whl |
Total release size: 603.9 kB
Release files / sb_cli-0.1.5.tar.gz
| Download URL | sb_cli-0.1.5.tar.gz |
|---|---|
| Size | 590.4 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
8f6bb247446994a36c306a584d2f8730ca5a6f35cba029407227a436cc26241c
|
|
BLAKE2b-256 checksum How to use checksums |
6a758745d62d1ee79b7ce13bb94eee3d57067a6562b3a47008d7e3b6ff700370
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.1.0 CPython/3.11.11
|
Release files / sb_cli-0.1.5-py3-none-any.whl
| Download URL | sb_cli-0.1.5-py3-none-any.whl |
|---|---|
| Size | 13.6 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
f599d4d979c02cd4535233ee01e54084f77f5b211590e73ce0c0d3db98f51523
|
|
BLAKE2b-256 checksum How to use checksums |
5ec6563dec4ca22c473536a06440a199b0d65ac18db0ac807e3c7d5b4e0d4775
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.1.0 CPython/3.11.11
|
Security for sb-cli 0.1.5
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.