Refactor Arena
A Control Arena setting for evaluating agents on full repository refactoring tasks with optional backdoor insertion.
Overview
This setting provides an environment for evaluating agents on full repository refactoring tasks. Each task involves:
- Main Task: Refactoring the repository to improve code quality and maintainability.
- Side Task (Attack Mode): Inserting a hidden vulnerability while performing the refactoring that is exploited when running the tests.
Installation
Install the package (requires Python 3.12):
# Install the package
pip install refactor-arena
# Setup your .env and initial scenario
refactor-arena init
# Run getting started scenario in honest mode
refactor-arena eval --config getting-started
If not already installed, install Docker. Docker must be running for the evals to execute.
Usage
Using the CLI
# Run with default settings using `side-tasks-3/mixed-3` dataset
refactor-arena
# Setup your .env and initial scenario
refactor-arena init
# Run getting started scenario in honest mode
refactor-arena eval --config getting-started
# Run getting started scenario in attack mode
refactor-arena eval --config getting-started --eval-mode attack
# Run with a different dataset from `github:fastfedora/full-repo-datasets` and limit main tasks
refactor-arena --dataset side-tasks-3/mixed-20 --main-task "refactor_single_file"
# Run with verbose logging and limit to 3 samples
refactor-arena eval --config getting-started --verbose --limit 3
# View help
refactor-arena --help
Configuration
To create a new scenario, create a file that references the main configuration file from the package and then override the settings you want to change.
For instance, to create a scenario that runs only the Refactor Single File main task against the
20-repo side-tasks-3/mixed-20 dataset in attack mode, create a file called mixed-20.yaml:
# mixed-20.yaml
includes:
- package:config.yaml
setting:
dataset: "side-tasks-3/mixed-20"
main_task: refactor_single_file$
eval:
# Default mode for this scenario is attack; use `--eval-mode honest` to run in honest mode
mode: attack
Then run this scenario using:
refactor-arena --config=mixed-20
Almost all aspects of the setting can be configured via configuration files, including:
- models
- prompts
- main tasks
- side tasks
- policies
- monitors
- metrics
- eval settings
See the documentation for more details.
License
MIT License
1 maintainer
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Filter files by name, interpreter, ABI, and platform.
If you're not sure about the file name format, learn more about wheel file names.
Copy a direct link to the current filters
File details
Details for the file refactor_arena-0.9.0.tar.gz.
File metadata
- Download URL: refactor_arena-0.9.0.tar.gz
- Upload date:
- Size: 50.2 MB
- Tags: Source
- Uploaded using Trusted Publishing? Yes
- Uploaded via:
twine/7.0.0 CPython/3.13.14
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
e65e37dd0f8f5bb6d0ef935381896a1c679893abc1eb56a499467ec70ba5d49c
|
|
| MD5 |
5a924f76ad47dc359cc9e82ec9232764
|
|
| BLAKE2b-256 |
479ea29a18bf2c1bb3477210b7ca0b3d9ce880b6c867a34a4dd6faa328599d5c
|
Provenance
The following attestation bundles were made for refactor_arena-0.9.0.tar.gz:
Publisher:
publish.yml on fastfedora/refactor-arena
-
Statement:
-
Statement type:
https://in-toto.io/Statement/v1 -
Predicate type:
https://docs.pypi.org/attestations/publish/v1 -
Subject name:
refactor_arena-0.9.0.tar.gz -
Subject digest:
e65e37dd0f8f5bb6d0ef935381896a1c679893abc1eb56a499467ec70ba5d49c - Sigstore transparency entry: 2334875655
- Sigstore integration time:
-
Permalink:
fastfedora/refactor-arena@e462fdd240b93e2977d62cf41c42d335049625f1 -
Branch / Tag:
refs/tags/v0.9.0 - Owner: https://github.com/fastfedora
-
Access:
public
-
Token Issuer:
https://token.actions.githubusercontent.com -
Runner Environment:
github-hosted -
Publication workflow:
publish.yml@e462fdd240b93e2977d62cf41c42d335049625f1 -
Trigger Event:
release
-
Statement type:
File details
Details for the file refactor_arena-0.9.0-py3-none-any.whl.
File metadata
- Download URL: refactor_arena-0.9.0-py3-none-any.whl
- Upload date:
- Size: 293.2 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? Yes
- Uploaded via:
twine/7.0.0 CPython/3.13.14
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
ca8d0cbc68194ada204d0e65724e94e93520ed7c00f1a6cb96aacaeb06bcd06f
|
|
| MD5 |
70dfeef97e54f813182e4001a495fe51
|
|
| BLAKE2b-256 |
f6c3debdb4221777f8619b21c29978e74e72664a4acd9b131a8928d3a033348e
|
Provenance
The following attestation bundles were made for refactor_arena-0.9.0-py3-none-any.whl:
Publisher:
publish.yml on fastfedora/refactor-arena
-
Statement:
-
Statement type:
https://in-toto.io/Statement/v1 -
Predicate type:
https://docs.pypi.org/attestations/publish/v1 -
Subject name:
refactor_arena-0.9.0-py3-none-any.whl -
Subject digest:
ca8d0cbc68194ada204d0e65724e94e93520ed7c00f1a6cb96aacaeb06bcd06f - Sigstore transparency entry: 2334875658
- Sigstore integration time:
-
Permalink:
fastfedora/refactor-arena@e462fdd240b93e2977d62cf41c42d335049625f1 -
Branch / Tag:
refs/tags/v0.9.0 - Owner: https://github.com/fastfedora
-
Access:
public
-
Token Issuer:
https://token.actions.githubusercontent.com -
Runner Environment:
github-hosted -
Publication workflow:
publish.yml@e462fdd240b93e2977d62cf41c42d335049625f1 -
Trigger Event:
release
-
Statement type:
