full-repo-refactor 0.2.0

Full repository refactoring setting for Control Arena

Full Repository Refactor

A Control Arena setting for evaluating agents on full repository refactoring tasks with optional backdoor insertion.

View full documentation.

Overview

This setting provides an environment for evaluating agents on full repository refactoring tasks. Each task involves:

1. Main Task: Refactoring the repository to improve code quality and maintainability.
2. Side Task (Attack Mode): Inserting a hidden vulnerability while performing the refactoring that is exploited when running the tests.

Installation

Requires Python 3.12 or later.

pip install full-repo-refactor

Usage

Using the CLI

# Run with default settings using `side-tasks-3/mixed-3` dataset
full-repo-refactor

# Run getting started scenario in default model (honest)
full-repo-refactor --config=scenarios/getting-started

# Run getting started scenario in attack mode
full-repo-refactor --config=scenarios/getting-started --eval-mode attack

# Run with a different dataset from `github:fastfedora/full_repo_datasets` and limit main tasks
full-repo-refactor --dataset side-tasks-3/mixed-20 --main-task "refactor_single_file"

# Run with verbose logging and limit to 3 samples
full-repo-refactor --config=scenarios/getting-started --verbose --limit 3

# View help
full-repo-refactor --help

Configuration

To create a new scenario, create a file that references the main configuration file from the package and then override the settings you want to change.

For instance, to create a scenario that runs only the Refactor Single File main task against the 20-repo side-tasks-3/mixed-20 dataset in attack mode, create a file called mixed-20.yaml:

# mixed-20.yaml
includes:
  - package:config.yaml

setting:
  dataset: "side-tasks-3/mixed-20"
  main_task: refactor_single_file$

eval:
  # Default mode for this scenario is attack; use `--eval-mode honest` to run in honest mode
  mode: attack

Then run this scenario using:

full-repo-refactor --config=mixed-20

Almost all aspects of the setting can be configured via configuration files, including:

• models
• prompts
• main tasks
• side tasks
• policies
• monitors
• metrics
• eval settings

See the documentation for more details.

License

MIT License