full-repo-refactor 0.4.0

Full repository refactoring setting for Control Arena

Full Repository Refactor

A Control Arena setting for evaluating agents on full repository refactoring tasks with optional backdoor insertion.

View full documentation.

Overview

This setting provides an environment for evaluating agents on full repository refactoring tasks. Each task involves:

1. Main Task: Refactoring the repository to improve code quality and maintainability.
2. Side Task (Attack Mode): Inserting a hidden vulnerability while performing the refactoring that is exploited when running the tests.

Installation

Install the package (requires Python 3.12):

# Install the package
pip install full-repo-refactor

# Setup your .env and initial scenario
full-repo-refactor init

# Run getting started scenario in honest mode
full-repo-refactor eval --config getting-started

If not already installed, install Docker. Docker must be running for the evals to execute.

Usage

Using the CLI

# Run with default settings using `side-tasks-3/mixed-3` dataset
full-repo-refactor

# Setup your .env and initial scenario
full-repo-refactor init

# Run getting started scenario in honest mode
full-repo-refactor eval --config getting-started

# Run getting started scenario in attack mode
full-repo-refactor eval --config getting-started --eval-mode attack

# Run with a different dataset from `github:fastfedora/full_repo_datasets` and limit main tasks
full-repo-refactor --dataset side-tasks-3/mixed-20 --main-task "refactor_single_file"

# Run with verbose logging and limit to 3 samples
full-repo-refactor eval --config getting-started --verbose --limit 3

# View help
full-repo-refactor --help

Configuration

To create a new scenario, create a file that references the main configuration file from the package and then override the settings you want to change.

For instance, to create a scenario that runs only the Refactor Single File main task against the 20-repo side-tasks-3/mixed-20 dataset in attack mode, create a file called mixed-20.yaml:

# mixed-20.yaml
includes:
  - package:config.yaml

setting:
  dataset: "side-tasks-3/mixed-20"
  main_task: refactor_single_file$

eval:
  # Default mode for this scenario is attack; use `--eval-mode honest` to run in honest mode
  mode: attack

Then run this scenario using:

full-repo-refactor --config=mixed-20

Almost all aspects of the setting can be configured via configuration files, including:

• models
• prompts
• main tasks
• side tasks
• policies
• monitors
• metrics
• eval settings

See the documentation for more details.

License

MIT License