SWE Lab
Tooling to build, run, enrich, audit and fix SWE (a.k.a. coding agent) evaluation data.
What's in it
-
The run engine (
src/swe_lab/sandbox/,workflow/) — one sandboxed run of a coding agent over a benchmark instance, composed from three plug-in axes: a harness (claude_code,codex,grok_build), a dataset (SWE-Bench Pro, DeepSWE 1.1) and an eval method (unit-test grading). A workflow chains those steps — solve, grade, or both — and is run by name:python -m swe_lab run --list # what can be run python -m swe_lab run rollout_and_unit_test <instance_id> # solve, then grade
-
Benchmark integrity (
src/swe_lab/git/,integrity/) — the task repo's future is stripped out of the container before the agent starts (and the strip is proved), and each run is swept for the ways an agent can reach the answer anyway. Detection, never a gate — ADR-0010. -
Related-files annotation (
src/swe_lab/pipelines/related_files/) — for each task instance, a ground-truth list of the code snippets a model needs to read to solve it. Shipped: 731/731 SWE-Bench Pro instances annotated & QA'd. -
Quality auditing (planned) — flag "skewed" eval examples that no longer measure real capability (ambiguous specs vs. overly-specific tests, broken environments, contamination, brittle graders), in the spirit of OpenAI's Separating signal from noise in coding evaluations. Not started; the nearest shipped work is the benchmark-integrity detection above.
The overall roadmap and status live in docs/README.md; the
codebase map, commands and hazards in
docs/conventions.md.
Setup
Prerequisites
- uv for environment and dependency management
- direnv for auto-activating the environment
- Python 3.13 (uv will install it automatically if missing)
1. Clone
git clone https://github.com/Luolc/swe-lab.git
cd swe-lab
Optional — the --capture proxy mode compiles the standalone
cc-reverse-proxy Go project. It is
not a submodule: by default it is looked up as a sibling checkout next to
this repo (../cc-reverse-proxy/reverse_proxy.go); clone it there, or point
CC_REVERSE_PROXY_SRC at its reverse_proxy.go. The default stream capture
needs none of this.
2. Set up the environment
uv sync # create .venv and install all (incl. dev) dependencies
direnv allow # auto-activate the venv on cd (uses .envrc)
If you don't use direnv, activate manually with source .venv/bin/activate.
Install the pre-commit hooks (the full set is listed in
docs/conventions.md):
uv run pre-commit install
3. Download the datasets
Dataset data files are gitignored and must be downloaded locally. See
datasets/README.md for the list of available datasets
and per-dataset download instructions.
Disclaimer
This is a personal project and is not affiliated with any company. The content does not reflect any specific company's projects, products or internal work.
1 maintainer
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Filter files by name, interpreter, ABI, and platform.
If you're not sure about the file name format, learn more about wheel file names.
Copy a direct link to the current filters
File details
Details for the file swe_lab-0.3.1.tar.gz.
File metadata
- Download URL: swe_lab-0.3.1.tar.gz
- Upload date:
- Size: 1.2 MB
- Tags: Source
- Uploaded using Trusted Publishing? Yes
- Uploaded via:
twine/7.0.0 CPython/3.13.14
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
a86d758e8484a6000383df4f5434b6fdc864b299f7dd8fe8945f08a3b1c6b8f7
|
|
| MD5 |
412c143b5cf02a159c6794c551d46da0
|
|
| BLAKE2b-256 |
00b77957b369d48574ade565f6ad473d350dada71193cd817720dca7e0ef7b87
|
Provenance
The following attestation bundles were made for swe_lab-0.3.1.tar.gz:
Publisher:
publish.yml on Luolc/swe-lab
-
Statement:
-
Statement type:
https://in-toto.io/Statement/v1 -
Predicate type:
https://docs.pypi.org/attestations/publish/v1 -
Subject name:
swe_lab-0.3.1.tar.gz -
Subject digest:
a86d758e8484a6000383df4f5434b6fdc864b299f7dd8fe8945f08a3b1c6b8f7 - Sigstore transparency entry: 2703208789
- Sigstore integration time:
-
Permalink:
Luolc/swe-lab@47e1a33ccd26993fd10cece932109be177b234ca -
Branch / Tag:
refs/tags/v0.3.1 - Owner: https://github.com/Luolc
-
Access:
public
-
Token Issuer:
https://token.actions.githubusercontent.com -
Runner Environment:
github-hosted -
Publication workflow:
publish.yml@47e1a33ccd26993fd10cece932109be177b234ca -
Trigger Event:
release
-
Statement type:
File details
Details for the file swe_lab-0.3.1-py3-none-any.whl.
File metadata
- Download URL: swe_lab-0.3.1-py3-none-any.whl
- Upload date:
- Size: 1.3 MB
- Tags: Python 3
- Uploaded using Trusted Publishing? Yes
- Uploaded via:
twine/7.0.0 CPython/3.13.14
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
9d94c7144afa17ef577ade65a22823f0051471eb4191bf50971f320d508ad889
|
|
| MD5 |
fd45df4d34749cb33f268324081216a6
|
|
| BLAKE2b-256 |
6daf29dcb5da8163f336a29e7e86e522900ce55e9a0fcee11372662a8feaf527
|
Provenance
The following attestation bundles were made for swe_lab-0.3.1-py3-none-any.whl:
Publisher:
publish.yml on Luolc/swe-lab
-
Statement:
-
Statement type:
https://in-toto.io/Statement/v1 -
Predicate type:
https://docs.pypi.org/attestations/publish/v1 -
Subject name:
swe_lab-0.3.1-py3-none-any.whl -
Subject digest:
9d94c7144afa17ef577ade65a22823f0051471eb4191bf50971f320d508ad889 - Sigstore transparency entry: 2703208935
- Sigstore integration time:
-
Permalink:
Luolc/swe-lab@47e1a33ccd26993fd10cece932109be177b234ca -
Branch / Tag:
refs/tags/v0.3.1 - Owner: https://github.com/Luolc
-
Access:
public
-
Token Issuer:
https://token.actions.githubusercontent.com -
Runner Environment:
github-hosted -
Publication workflow:
publish.yml@47e1a33ccd26993fd10cece932109be177b234ca -
Trigger Event:
release
-
Statement type:
