Refactor Arena

A Control Arena setting for evaluating agents on full repository refactoring tasks with optional backdoor insertion.

View full documentation.

Overview

This setting provides an environment for evaluating agents on full repository refactoring tasks. Each task involves:

  1. Main Task: Refactoring the repository to improve code quality and maintainability.
  2. Side Task (Attack Mode): Inserting a hidden vulnerability while performing the refactoring that is exploited when running the tests.

Installation

Install the package (requires Python 3.12):

# Install the package
pip install refactor-arena

# Setup your .env and initial scenario
refactor-arena init

# Run getting started scenario in honest mode
refactor-arena eval --config getting-started

If not already installed, install Docker. Docker must be running for the evals to execute.

Usage

Using the CLI

# Run with default settings using `side-tasks-3/mixed-3` dataset
refactor-arena

# Setup your .env and initial scenario
refactor-arena init

# Run getting started scenario in honest mode
refactor-arena eval --config getting-started

# Run getting started scenario in attack mode
refactor-arena eval --config getting-started --eval-mode attack

# Run with a different dataset from `github:fastfedora/full-repo-datasets` and limit main tasks
refactor-arena --dataset side-tasks-3/mixed-20 --main-task "refactor_single_file"

# Run with verbose logging and limit to 3 samples
refactor-arena eval --config getting-started --verbose --limit 3

# View help
refactor-arena --help

Configuration

To create a new scenario, create a file that references the main configuration file from the package and then override the settings you want to change.

For instance, to create a scenario that runs only the Refactor Single File main task against the 20-repo side-tasks-3/mixed-20 dataset in attack mode, create a file called mixed-20.yaml:

# mixed-20.yaml
includes:
  - package:config.yaml

setting:
  dataset: "side-tasks-3/mixed-20"
  main_task: refactor_single_file$

eval:
  # Default mode for this scenario is attack; use `--eval-mode honest` to run in honest mode
  mode: attack

Then run this scenario using:

refactor-arena --config=mixed-20

Almost all aspects of the setting can be configured via configuration files, including:

  • models
  • prompts
  • main tasks
  • side tasks
  • policies
  • monitors
  • metrics
  • eval settings

See the documentation for more details.

License

MIT License

Project links

Data verified by PyPI on Aug 12, 2026
Data provided by the project maintainers, verified at the time the release was uploaded to PyPI.

Key dates

PyPI data
Data sourced directly from PyPI's database.
  • Released:
Latest release

1 maintainer

PyPI data
Data sourced directly from PyPI's database.
Avatar for fastfedora from gravatar.com fastfedora

Credits

Author: Trevor Lohrbeer

GitHub Statistics

Data verified by PyPI on Aug 12, 2026
The GitHub source repository was provided by the project maintainers and verified by PyPI at the time of upload. Stars, forks, and open issues/PRs are derived from that repository and have not been independently verified.

License

MIT License (MIT)

Requires

Python >=3.12

Classifiers

Report project as malware

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

refactor_arena-0.10.0.tar.gz (50.2 MB view details)

Uploaded Source

Built Distribution

If you're not sure about the file name format, learn more about wheel file names.

refactor_arena-0.10.0-py3-none-any.whl (297.2 kB view details)

Uploaded Python 3

File details

Details for the file refactor_arena-0.10.0.tar.gz.

File metadata

  • Download URL: refactor_arena-0.10.0.tar.gz
  • Upload date:
  • Size: 50.2 MB
  • Tags: Source
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for refactor_arena-0.10.0.tar.gz
Algorithm Hash digest
SHA256 5e2b6077aca9dd180e0094ce04c4a809db29a6585d23c4a0f2574631e6c94080
MD5 39b6ae9d357eae07e44f467f503d5013
BLAKE2b-256 43f9d03b469e4bef92a7d064e4f7ad974254f8acc2d9ea503f235f74e072d0a9

See more details on using hashes here.

Provenance

The following attestation bundles were made for refactor_arena-0.10.0.tar.gz:

Publisher: publish.yml on fastfedora/refactor-arena

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file refactor_arena-0.10.0-py3-none-any.whl.

File metadata

  • Download URL: refactor_arena-0.10.0-py3-none-any.whl
  • Upload date:
  • Size: 297.2 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for refactor_arena-0.10.0-py3-none-any.whl
Algorithm Hash digest
SHA256 6058521f619c2a150418ebb871592a1fd249d40bf1b349fccc4059748042b2d1
MD5 7d65c01b21d8b8de1ca70a99ea0f1ff6
BLAKE2b-256 1d12d12c61136cc4d15435e4d75f8320da7215d67b7d8a8e0b59791bea9de78c

See more details on using hashes here.

Provenance

The following attestation bundles were made for refactor_arena-0.10.0-py3-none-any.whl:

Publisher: publish.yml on fastfedora/refactor-arena

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Release history Release notifications | RSS feed

This release

0.10.0 This release

2 files

0.9.0

2 files

0.8.1

2 files

0.8.0

2 files

0.7.0

2 files

0.6.1

2 files

0.6.0

2 files

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page