AgentSelfEdit

An agent that rewrites its own system prompt from execution feedback — proposing edits, A/B testing them against a held-out task set, and promoting only statistically-proven winners under deterministic guardrails.

The agentic ecosystem raced ahead on orchestration and retrieval, but skipped the capability that would make agents feel alive: the ability to get measurably better at their own job over time. AgentSelfEdit turns prompt optimization into a self-sustaining, evidence-driven loop with provenance, rollback, and guardrails.

Why

Most production agents are prompt-tuned once by hand — usually by the human who wrote them, usually once, then never again. The prompt freezes the moment it ships. Every recurring failure is silently absorbed until a human manually tunes again. And the agent that sees the most failure data is the least able to use it.

Two common answers are not enough:

• "Reflection" is not learning. Appending a paragraph of prose to context makes prompts longer, not better. The prompt itself never changes, so the same failure repeats tomorrow.
• Sharing the raw prompt optimizer with an LLM breaks everything. Unmanaged, LLM-judged edits poison the baseline within a few iterations.

AgentSelfEdit turns prompt optimization into a self-sustaining, evidence-driven loop with provenance, rollback, and guardrails — and it is designed as a sidecar. It does not modify the agent's runtime. It observes execution traces and proposes prompt edits.

Quick Start

# Install
pip install agent-self-edit

# Scaffold a project
agent-self-edit init

# Run the self-improvement loop
agent-self-edit run --once

# Or in Docker (requires local OMLX endpoint)
docker build -t agent-self-edit .
docker run --rm --network=host agent-self-edit run --once

How It Works

Agent executes task ──▶ Execution trace stored (SQLite)
                                │
                                ▼
                      Feedback Analyzer (LLM)
                      reviews traces, proposes concrete edits,
                      each with a written hypothesis
                                │
                                ▼
─────────────────────  A/B Test Engine  ─────────────────────
  candidate edit vs current prompt on a held-out task set:
  win rate, bootstrap confidence interval, effect size,
  permutation p-value, per-task breakdown
────────────────────────────────────────────────────────────
                                │
                                ▼
                Promotion Gate (deterministic checks)
                1. Sample floor     4. Frozen sections
                2. Effect size      5. Edit-distance limit
                3. Confidence p-val 6. Drift detection
                                │
              ┌─────────────────┼─────────────────┐
              ▼                 ▼                 ▼
          Promoted         Near-miss        Rejected
     prompt updated in     logged for       archived with
     versioned Registry    human review     full reasoning

1. Analyze — After each task, an LLM reviews the execution trace and identifies what went wrong: the specific instruction that was missing, ambiguous, or wrong.
2. Propose — It proposes one or more concrete, minimal prompt edits, each with a stated hypothesis about why it should improve outcomes.
3. Test — Each candidate edit is A/B tested against the current prompt on a held-out task set, with confidence intervals, effect-size thresholds, and a minimum sample floor before any promotion decision.
4. Promote or Archive — The statistically-proven winner becomes the new baseline. The loser is archived with its full reasoning.
5. Guard — Frozen core sections, edit-distance limits, and drift detection keep the agent from rewriting itself into a worse version over hundreds of iterations.

Core Components

The Promotion Gate

The gate is the real product. It is deterministic — never LLM-judged — and runs six fail-fast checks before any edit is promoted:

1. Sample floor — minimum number of A/B trials completed
2. Effect size — improvement exceeds a configurable minimum threshold
3. Confidence — p-value below the significance threshold (default: p < 0.05)
4. Frozen core sections — user-annotated sections the analyzer cannot modify
5. Edit-distance limit — maximum lines changed per cycle (configurable)
6. Drift detection — semantic similarity to the original prompt, alerts on divergence

Field test result

The gate was validated in a 15-iteration field test against a real LLM (Qwen3.5-4B-4bit, local). Every edit was correctly rejected because the improvement was real but statistically underpowered (p=0.23 >= 0.05).

The gate rejecting is the success condition, not the failure condition. The system is mechanically sound. The current analyzer strategy does not yet produce statistically significant improvement — a valid finding, not a bug.

Trigger Modes

• Batch — analyze after N tasks (default: 50)
• Time-based — analyze every N hours
• Manual — analyze on demand

Where It Helps

Any agent that repeats a similar task type and sees execution feedback beats its prompt frozen on day one:

• Customer support ticket classification — a change that helps one category can't degrade another.
• Code review / PR analysis — false positives on docs-only PRs are learned and fixed.
• Data extraction / entity parsing — PDF-with-tables extraction doesn't regress plain-text extraction.
• Content moderation — false-positive / false-negative tradeoffs are measured, not guessed.
• Sales outreach personalization — winning variants are proven, not copy-pasted.
• Documentation generation — output length and tone adapt from feedback.

Status

✅ v0.1.0 released on PyPI. All 11 milestones complete.

Field Test Results

The full 15-iteration improvement loop was run against Qwen3.5-4B-4bit on Apple Silicon. Every iteration produced inspectable A/B artifacts (prompt-a/b, results-a/b, ab-comparison, analysis, accuracy). The gate correctly rejected every edit.

See final field test report for the complete analysis.

Roadmap

License

MIT — see LICENSE