PromptPotter: LLM-Driven Evolution of Prompts and Pipelines

PromptPotter evolves better prompts. Most prompt engineering is manual. PromptPotter automates the generate → score → critique cycle. It tries multiple prompt and pipeline variations together, keeps memory across runs, and recovers automatically when a generated prompt produces broken output. Weak candidates get eliminated early on statistical confidence (Posterior-of-Being-Best — PoBB) so you don't burn LLM budget on losers. Built for RAG pipelines, LLM agents, and multi-step LLM workflows — drop in via CLI, Python SDK, or the /potter-run Claude Code skill.

How to Optimize LLM Prompts in 3 Steps

Describe your 1️⃣ task, drop in a labeled 2️⃣ dataset, and 3️⃣ run the loop. The task is the goal you want the AI to hit; the dataset is examples of hitting it. Each round, PromptPotter generates variations 🧪, scores them ⚖️, and keeps the winners 🏆. It stops when results plateau. ✨ Prompt optimized.

[!IMPORTANT] New here? Start with docs/manual/ — six chapters covering install → first run → reading output → troubleshooting.

⭐ Features

Every measurement costs money, so the whole design is most fitness per dollar. The capabilities PromptPotter shares with the rest of the field are in the comparison table below; these are the ones it doesn't:

• 💬 Chat-first — talk to the Potter and watch it work inline, Perplexity-style: the searches, the tool calls, each round as it lands, and a button whenever a decision is yours. Ships as a reusable chat-app template — keep the chat core, delete the optimizer panes. spec
• Climbs the hill, escapes dead ends — each round steps uphill on your metric; when a branch is spent the search rewinds to a better ancestor and climbs a different ridge instead of stalling. Most evolutionary search is one-armed: it only ever expands the latest winner.
• Hard-sample leaderboard — score preferentially on the samples that actually separate variants; the ones everyone aces or fails are noise.
• Guards against self-validation — the loop can't grade itself into a false win: scores are ability-based and subset-invariant (Rasch θ), constant-answer and other degenerate candidates are caught before they count, and the layer that validates a fix is never the one that proposed it.
• Optimizes itself — point the optimizer at its own optimizer prompts. L4
• Pick your block library mode — proven personas, thinking styles and answer formats (from PromptWizard and the Self-Discover modules it draws on, plus what our own runs turned up). Let the optimizer suggest from the library, restrict it to the library, or switch it off.

Five ways to run it

1. /potter-run Claude Code skill — drive a full campaign from your editor.
2. CLI — python -m promptpotter new <name> / resume.
3. Python / Jupyter notebook.
4. REST API.
5. WebApp — read-only dashboard at http://localhost:8001/.

Direction — the sixth way: a tool another agent calls. The aim is parity as a first-class agent-callable tool (MCP), so an operating agent — yours, or an ML-research agent like NVIDIA's AutoResearch — can invoke PromptPotter as its try-harness-first move before reaching for fine-tuning. Why + a same-dataset head-to-head: related-work § PromptPotter × NVIDIA AutoResearch; tracked as roadmap § Agent-tool parity (C5, MCP server mode).

Common questions

• What does L1 actually mutate? The prompt template's fields (persona, task instruction, …) plus whatever your pipeline.yaml declares as tunable. See state-record.md.
• Where do I get a starting prompt? Bring one with your dataset (datasets/{name}/prompts/{node}.yaml). Walkthrough: manual ch. 03.
• How do I watch a run? Open dashboard.json in an auto-reload editor + watch the CLI terminal. Full guide: Watching a run.
• My scoring formula was wrong — did I lose results? No. Traces are facts; scores are policy. The optimizer rescores on load and replays decisions; on divergence, fork. See scoring-and-memory.md.
• What if it stalls? Stall and failure are different triggers. Failures route back to the proposing layer (self-healing); stalls escalate L1 → L2 → L3 (the-loop). Stuck for other reasons: troubleshooting.

Limitations

• Parameter-based optimization only. PromptPotter optimizes any pipeline that exposes tunable parameters (prompts, thresholds, model settings). It cannot optimize internal model weights, neural architectures, or modality-specific representations (e.g. image embeddings, DNA sequences).
• Requires a labeled dataset. Input/output pairs are mandatory.

Roadmap

Status and the full forward plan live in one place: docs/specs/roadmap.md. Documentation index: docs/README.md.

Peer systems

PromptPotter belongs to the LLM-driven evolution family: an LLM proposes variants, an evaluator scores them, and the winners breed. The peer systems — full comparison + benchmark notes in docs/research/related-work.md:

• Code evolution — AlphaEvolve · OpenEvolve · AlgoTuner · AutoResearch
• Prompt evolution — PromptWizard · MIPROv2 · GEPA · PromptPotter

The code-evolution systems mutate source; the prompt-evolution systems mutate a prompt. PromptPotter evolves both the prompt and the pipeline parameters around it, jointly.

Background: Asankhaya Sharma on OpenEvolve (talk).

Scientific framing

PromptPotter is a tree search over prompt programs — precisely, AlphaZero-shaped MCTS over the lineage: PoBB prunes losers within a round, each round's ability backpropagates to its ancestors, and a UCB rule picks the ancestor to re-expand once a branch is spent. Rollouts give way to deterministic evaluation on your dataset, as in AlphaZero. Comparison.

The loop

Each round is generate → score → critique: L1 proposes candidates, they're scored against your dataset, and the critique steers the next round. When the inner layer stalls, an outer layer (L2, then L3) redirects it; when the branch itself is spent, the search rewinds. Full mechanics: the-loop.md.

AlphaEvolve is the closest published peer — same loop, same memory across runs; read row one as scope (a different target, not a missing feature), the rest as capability.

The one place AlphaEvolve is unambiguously stronger is code optimization — its search reaches into source, which is not what PromptPotter is pointed at. The 🟡s are partial versions of the same capability. Full grading, including the prompt-tooling neighbours: related-work.md.

Citation

@software{promptpotter,
  title  = {PromptPotter: LLM-Driven Evolution of Prompts and Pipelines},
  author = {Streuli, David},
  year   = {2026},
  url    = {https://github.com/PromptPotter/prompt-potter-optimizer}
}

Benchmarks

Head-to-head comparison on the BIG-Bench Extra Hard (BBEH) benchmark against DSPy optimizers (GEPA, MIPROv2, BootstrapFewShot) and CAPO. Same model (gpt-oss-120b), same dataset splits, same scoring, no cross-paper number mixing. See docs/research/benchmarks.md for results and docs/research/bbeh-comparison/ for reproducible Colab notebooks.

Documentation

Developer internals (Python symbols, data contracts, wiring) live under docs/developer/. Statistical foundations under docs/methods/.

Watching a run

While python -m promptpotter resume is running, the cleanest setup is campaigns/{campaign_id}/cycles/{cycle_id}/dashboard.json open in an auto-reloading editor + the CLI terminal visible. dashboard.json is the live scalar state (phase, round, candidate, accuracy, in-flight query); the CLI prints HIT/MISS lines + per-candidate + per-round banners as they happen. Drill-down peers in the same directory: rounds/, log.md, index.json. Internal resume + audit state lives under .runtime/ (hidden by convention). Alternatives: /potter-run Claude Code skill, the notebook, or the read-only webapp at http://localhost:8001/. Full guide in docs/manual/04-reading-the-output.md.

[!TIP]

TODO: What a round actually looks like (click to expand)

Every cycle writes a per-round digest to log.md. This is an excerpt from a real JustLogic run — the round that took it from 65.0% to 75.0%:

### Round 2 — Added systematic deduction step to thinking_style to force
exhaustive derivation before defaulting to Uncertain, targeting the premature
'no direct info' pattern seen in #82, #37, #0. (75.0%)

- improved: **yes**
- samples: 28
- composite_fitness: `0.7500`

> Fix: thinking_style: Adopt a formal logical deduction method: translate each
> premise into symbolic formulas, then derive the claim's truth value using
> entailment rules. This counters the pattern of defaulting to Uncertain when
> logical entailment exists, as seen in the 'When X is true' misinterpretation.
> Axes: thinking_style, instruction, problem_description, answer_format
> Failures:
>   Logical misinterpretation (~5/14 misses): model fails to recognize that a
>   premise phrased 'When X is true, it follows Y' asserts X as true, leading
>   to over-hedging with Uncertain. Predicted: Uncertain, GT: TRUE. (Query #82)

P(best) trajectory:
  5536f04bc9 ▆▇▇▇▇▇▇████████▇▇▇█████   87.6% [winner]
  c2ac162dea ▇▆▆▆▆▆▆▅▄▅▅▅▄▄▅▅▄▅▄▅▅▅▄   50.0%

Those trajectory bars are PoBB at work — one tick per scored sample, and the trailing arm's posterior never recovers. What every stream a run produces means: reading the output.