AgentSelfEdit
A self-improving agent prompt optimizer. Proposes edits to its own system prompt from execution feedback, A/B tests them against a held-out task set, and promotes only statistically-proven winners under deterministic guardrails.
Changelog · v0.2.0 Release Notes · v0.1.0 Field Test Report · v0.2.0 Field Test Report
Why
Most production agents are prompt-tuned once by hand, then frozen at ship time. Every recurring failure is silently absorbed. Two common approaches fall short:
- Reflection appends prose to context but never changes the prompt itself. The same failure repeats.
- Unconstrained LLM-judged edits poison the baseline within a few iterations.
AgentSelfEdit is a sidecar that observes execution traces and proposes prompt edits through a deterministic, evidence-driven loop.
Quick Start
pip install agent-self-edit
agent-self-edit init
agent-self-edit run --once
How It Works
Agent executes task → Execution trace stored (SQLite)
↓
Feedback Analyzer (LLM)
reviews traces, proposes concrete edits,
each with a written hypothesis
↓
──── A/B Test Engine ────
candidate vs current prompt
held-out task set, bootstrap CI,
effect size, permutation p-value
───────────────────────────
↓
Promotion Gate (deterministic)
1. Sample floor 4. Frozen sections
2. Effect size 5. Edit-distance limit
3. Confidence p-val 6. Drift detection
↓
┌──────────┼──────────┐
▼ ▼ ▼
Promoted Near-miss Rejected
- Analyze — An LLM reviews execution traces and identifies failures.
- Propose — Concrete, minimal prompt edits with stated hypotheses.
- Test — Each candidate is A/B tested with confidence intervals and effect-size thresholds.
- Promote or Archive — Winners become the new baseline; losers are archived with reasoning.
- Guard — Frozen sections, edit-distance limits, and drift detection prevent degradation.
Core Components
| Component | What it does |
|---|---|
| Feedback Analyzer | LLM that reviews traces and produces edit proposals. v0.2.0: staged 4-pipeline (summarize → select → synthesize → score) with rejection-context feedback. |
| A/B Test Engine | Compares candidate vs current prompt on held-out tasks. v0.2.0: 40-task promotion corpus. |
| Promotion Gate | Six deterministic checks: sample floor, effect size, confidence, frozen sections, edit-distance, drift. Code, not LLM-judged. |
| Prompt Registry | File-based versioned store with full lineage, diff, rollback, SHA-256 integrity. |
| Scorers | Runtime-selectable: ExactMatch, StructuredExtraction, LLMJudge. Label-set-aware, multi-domain. |
| CLI | init, run, status, diff, rollback, guardrails, lineage, propose, ingest, validate. |
The Promotion Gate
Six fail-fast checks before any edit is promoted:
- Sample floor — minimum A/B trials completed
- Effect size — minimum improvement threshold
- Confidence — p < 0.05 (default)
- Frozen core sections — user-annotated protected regions
- Edit-distance limit — max lines changed per cycle
- Drift detection — semantic similarity to original prompt
Field Test Results
v0.1.0
15-iteration loop against Qwen3.5-4B-4bit (local). Gate correctly rejected every edit.
| Metric | Result |
|---|---|
| FP rate (bad edits promoted) | 0% |
| FN rate (good edits rejected) | 0% |
| LLM calls | 4,150 |
| Wall time | 37 min |
| Cost | $0.00 |
v0.2.0
Validation across 4 domains, adversarial edits, real-trace ingestion, Docker. Strongest analyzer: Mistral Small 24B (OpenRouter). Directional gains but no promotable edit — framework is sound, analyzer quality is the bottleneck.
| Metric | Result |
|---|---|
| FP rate | 0% |
| FN rate | not observed |
| Multi-domain suites | 4/4 |
| Adversarial edits caught | 5/5 (0 FN) |
| Strongest analyzer | mistralai/mistral-small-3.2-24b-instruct |
| Coverage | 81% (target 92%) |
Trigger Modes
- Batch — analyze after N tasks (default: 50)
- Time-based — analyze every N hours
- Manual — analyze on demand
Where It Helps
Agents that repeat a task type and see execution feedback: classification, code review, data extraction, content moderation, sales outreach, documentation generation.
Status
| Area | Status |
|---|---|
| Core loop | ✅ |
| Model role separation | ✅ |
| Staged analyzer | ✅ |
| Multi-domain support | ✅ |
| Runtime scorer selection | ✅ |
| Adversarial rejection (5/5) | ✅ |
| Real-trace ingestion | ✅ |
| Tests (489/489 pass) | ✅ |
| Ruff + mypy strict | ✅ |
| Security (bandit, gitleaks, trufflehog) | ✅ clean |
| Coverage | ⬜ 81% |
Roadmap
| Version | Focus |
|---|---|
| v0.1.0 | ✅ Released — core loop, gate, CLI, Docker |
| v0.2.0 | ✅ Released — staged analyzer, model roles, multi-domain, field test |
| v0.3.0 | Framework adapters, multi-failure clustering, adaptive sample floors |
| v0.4.0 | Fleet-wide learning, cost-aware improvement |
| v1.0.0 | Stable API, production deployment guide |
License
MIT — see LICENSE
Metadata
Release files for agent-self-edit 0.3.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| agent_self_edit-0.3.0.tar.gz | 132.6 kB | Without provenance | Details for agent_self_edit-0.3.0.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| agent_self_edit-0.3.0-py3-none-any.whl | Python 3 | none | any | Details for agent_self_edit-0.3.0-py3-none-any.whl |
Total release size: 199.6 kB
Release files / agent_self_edit-0.3.0.tar.gz
| Download URL | agent_self_edit-0.3.0.tar.gz |
|---|---|
| Size | 132.6 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
7cda3a1574c36887eac610934ae38aff5f1861a8c34b70c5a093f11e09dd9e25
|
|
BLAKE2b-256 checksum How to use checksums |
3df874f07bc4a98d5d96695f58ea930575e7b1032a9799b7b874bb2e4f83719f
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.14.5
|
Release files / agent_self_edit-0.3.0-py3-none-any.whl
| Download URL | agent_self_edit-0.3.0-py3-none-any.whl |
|---|---|
| Size | 67.0 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
ec4ac62b34c66a9563de3c7544723e518f3a0c7f278a10221e063ec484501d13
|
|
BLAKE2b-256 checksum How to use checksums |
1e1aab9ff22719f3a4c1b51e733d8f14b8e524241559f49ca29348ecb7b4ab68
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.14.5
|
Security for agent-self-edit 0.3.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.