AutoGPT

AutoGPT is an auto-evolving agent harness.

Coding agents are only as good as what they know about your codebase. AutoGPT measures how well any agent works in your repository, and automatically evolves the skills, conventions, and policies that make it better — keeping only the changes that are proven to help.

pip install autogpt
cd your-repo
autogpt init
autogpt bench mine
autogpt bench run --agent claude

Auto-evaluate: your git history is the benchmark

Every commit that changed both code and tests is a task with a known answer. AutoGPT replays them:

1. Mine. Pick commits that change source and runnable tests. The commit message becomes the task; PR numbers and trailers are removed so the agent cannot look the answer up.
2. Snapshot. Export the code as it was before the commit into a fresh repository with no history, then add the commit's tests. The agent cannot run git log to find the solution.
3. Verify first. Run the tests before the agent starts. If they already pass, the task is discarded as invalid.
4. Run the agent. Claude Code, Codex, or any command line agent.
5. Verify after. Put the original tests back, in case the agent edited them, and run them. Pass or fail is decided by the tests, not by the agent's own report.

Every run keeps its patch, duration, and (for Claude Code) cost, so runs can be inspected and compared later.

Auto-compare: prove what helps

Run the same tasks with different skills, instructions, or agents, and compare:

autogpt bench run --agent claude --variant baseline
autogpt bench run --agent claude --variant with-skills --overlay ./my-skills
autogpt bench report

Example output (illustrative numbers, not a published result):

variant               valid  passed  pass rate  avg time      cost
------------------------------------------------------------------
baseline                 24       11       46%      212s    $18.40
with-skills              24       16       67%      174s    $15.10

--overlay copies a directory into every snapshot, so it works with whatever your agent reads: .claude/skills/, AGENTS.md, CLAUDE.md, and so on. Variants are scored only on tasks that are valid in every variant.

Auto-evolve: the roadmap

Measurement comes first, because evolution without measurement is just drift. Next:

• v0.2 Auto-learn: mine candidate skills and conventions from successful and failed runs.
• v0.3 Auto-validate: A/B test each candidate on held-out tasks. Promote it only if it measurably raises the pass rate or lowers cost.
• v0.4 Auto-evolve: a continuous loop. Skills move through candidate → validated → production → deprecated, each with its measured effect, and can be rolled back at any time.

The model never changes. The harness around it gets better:

Harness(t+1) = Harness(t) + validated skills + policies + conventions

Configuration

autogpt init writes .autogpt/config.toml:

[bench]
agent = "claude"
test_cmd = "python -m pytest -q {tests}"
setup_cmd = ""
timeout = 1800

• test_cmd: {tests} is replaced with the task's test files.
• setup_cmd: runs in each snapshot before the tests, e.g. to install dependencies.
• agent: claude, codex, or any command containing {prompt} or {prompt_file}.

Safety

• The claude preset lets the agent edit files only. It cannot run shell commands.
• The codex preset runs inside Codex's own sandbox.
• A custom agent command runs with your user's permissions on this machine. Snapshots are temporary directories, not containers.

Requirements

Python 3.11.4 or newer, git, and the agent CLI you want to measure. Linux and macOS.

AutoGPT (this project) is an independent project. It is not affiliated with Significant Gravitas Ltd or its AutoGPT Platform.