promptpotter.com — 10 free optimization runs on your own data, up to 10 rounds each, in the browser and on my key. Offer stands until 15 Sept 2026; bring-your-own-key is in the works.

PromptPotter: LLM-Driven Evolution of Prompts and Pipelines

PromptPotter evolves better prompts. Most prompt engineering is manual. PromptPotter automates the generate → score → critique cycle. It tries multiple prompt and pipeline variations together, keeps memory across runs, and recovers automatically when a generated prompt produces broken output. Weak candidates get eliminated early on statistical confidence (Posterior-of-Being-Best — PoBB) so you don't burn LLM budget on losers. Built for RAG pipelines, LLM agents, and multi-step LLM workflows — drop in via CLI, Python SDK, the /potter-run Claude Code skill, or as a DSPy optimizer where you would reach for GEPA.

How to Optimize LLM Prompts in 3 Steps

Describe your 1️⃣ task, drop in a labeled 2️⃣ dataset, and 3️⃣ run the loop. The task is the goal you want the AI to hit; the dataset is examples of hitting it. Each round, PromptPotter generates variations 🧪, scores them ⚖️, and keeps the winners 🏆. It stops when results plateau. ✨ Prompt optimized.

⭐ Features

Every measurement costs money, so the whole design is most fitness per dollar. What that buys in practice: a real run on a Swiss invoice account-coding set moved exact-match accuracy from 5% to 55% — and stopped because it hit its own spend cap, not because it ran out of ideas. (5 rounds, n=20; a measured lift, not a significance claim.)

The capabilities PromptPotter shares with the rest of the field are in the comparison table below; these are the ones it doesn't:

• 💬 Chat-first — talk to the Potter and watch it work inline, Perplexity-style: the searches, the tool calls, each round as it lands, and a button whenever a decision is yours. Ships as a reusable chat-app template — keep the chat core, delete the optimizer panes. spec
• Searches a tree, not a trail — every round's result flows back to each ancestor it descends from, so a spent branch rewinds to whichever ancestor the evidence favours and climbs a different hill instead of stalling. Branches are kept and compared rather than discarded, and an ancestor's score reflects what re-expanding from it actually yielded — including in branches it never ran itself.
• Hard-sample leaderboard — score preferentially on the samples that actually separate variants; the ones everyone aces or fails are noise.
• 🛡️ Guards against self-validation — an optimizer that picks its own winner is the easiest thing in this field to fake, and the loop is built so it can't grade itself into a false win. Three independent guards: scores are subset-invariant ability (Rasch θ), so a candidate that drifted onto easier samples cannot out-rank an honest one; degenerate candidates — a constant answer, a shape that games the scorer — are caught before they count; and the layer that validates a fix is never the layer that proposed it. What a winner's number is allowed to claim: verdict-resolution.md.
• Optimizes itself — point the optimizer at its own optimizer prompts. L4
• Pick your block library mode — proven personas, thinking styles and answer formats (from PromptWizard and the Self-Discover modules it draws on, plus what our own runs turned up). Let the optimizer suggest from the library, restrict it to the library, or switch it off.

Six ways to run it

1. WebApp — Full UI setup
2. /potter-run Claude Code skill — drive a full campaign from your editor.
3. CLI — python -m promptpotter new <name> / resume.
4. REST API
5. Python / Jupyter notebook — Promptpotter is written in Python. The DSPy setting is only recommended for simple cases, due to DSPy not permitting the mechanics of promptpotter, i.e. ["", ""] .

Direction — the sixth way: a tool another agent calls. Parity as a first-class agent-callable tool (MCP), so an operating agent — yours, or an ML-research agent like NVIDIA's AutoResearch — reaches for PromptPotter as its try-harness-first move before spending on fine-tuning. Why, plus a same-dataset head-to-head · roadmap § Agent-tool parity.

Common questions

• What does L1 actually mutate? The prompt template's fields (persona, task instruction, …) plus whatever your pipeline.yaml declares as tunable. See the-loop.md.
• Where do I get a starting prompt? Bring one with your dataset (datasets/{name}/prompts/{node}.yaml). Walkthrough: manual ch. 03.
• How do I watch a run? Open dashboard.json in an auto-reload editor + watch the CLI terminal. Full guide: Watching a run.
• My scoring formula was wrong — did I lose results? No. Traces are facts; scores are policy. The optimizer rescores on load and replays decisions; on divergence, fork. See scoring-and-memory.md.
• What if it stalls? Stall and failure are different triggers. Failures route back to the proposing layer (self-healing); stalls escalate L1 → L2 → L3 (the-loop). Stuck for other reasons: troubleshooting.

Limitations

• Parameter-based optimization only. PromptPotter optimizes any pipeline that exposes tunable parameters (prompts, thresholds, model settings). It cannot optimize internal model weights, neural architectures, or modality-specific representations (e.g. image embeddings, DNA sequences).
• Requires a labeled dataset. Input/output pairs are mandatory.

Roadmap

Status and the full forward plan live in one place: docs/specs/roadmap.md. Documentation index: docs/README.md.

Peer systems

PromptPotter belongs to the LLM-driven evolution family: an LLM proposes variants, an evaluator scores them, and the winners breed. The peer systems — full comparison + benchmark notes in docs/research/related-work.md:

• Prompt evolution — GEPA · MIPROv2 · PromptWizard · PromptPotter
• Code evolution — AlphaEvolve · OpenEvolve · AlgoTuner · AutoResearch

The Python line behind them: APE and OPRO (2022–23) established that a language model can author its own instruction; DSPy (2023–) made the prompt a compiled artifact, MIPROv2 searching its instructions and demonstrations; PromptWizard (2024) added critique-guided refinement; CAPO (2025) added racing and an explicit token budget. GEPA (ICLR 2026 Oral) is the one most people now compare against — though CAPO outscored it on both tasks of the promptolution paper's own head-to-head, so the reference point is not automatically the best result.

The code-evolution systems mutate source; every prompt-evolution system above mutates one artifact, the prompt. PromptPotter evolves both the prompt and the pipeline parameters around it, jointly.

Background: Asankhaya Sharma on OpenEvolve (talk).

Scientific framing

PromptPotter is a tree search over prompt programs — precisely, AlphaZero-shaped MCTS over the lineage: PoBB prunes losers within a round, each round's ability backpropagates to its ancestors, and a UCB rule picks the ancestor to re-expand once a branch is spent. Rollouts give way to deterministic evaluation on your dataset, as in AlphaZero. Comparison.

Test-time compute, moved out of the request. The popular way to spend compute at inference is to search inside the request — sample many answers, verify, pick (Snell et al. · survey). PromptPotter is that same search run as tuning infrastructure instead: you run it, it finds a configuration, and from then on that configuration is simply used on demand — an ordinary call, with no search in the path and nothing learning at request time. The test-time budget is itself one of the things it can tune (reasoning_effort, max_tokens, the model), so the inference bill is something it optimizes rather than something it grows.

If what you actually want is "pick the best currently available model for whatever I ask next" — one user, an arbitrary next question, live selection across whatever keys/subscriptions/local models you have — that's a gateway, not PromptPotter: OpenRouter (hosted) or OmniRoute (self-hosted, stitches your own providers into one local endpoint) are built for exactly that. PromptPotter runs the opposite axis — one fixed task, hit repeatedly, tuned once offline against your real eval data until a prompt+model+params combo wins, then deployed and reused — so it's what tunes the thing a gateway ends up calling, not a replacement for the gateway itself. The two compose rather than compete.

The loop

Each round is generate → score → critique: L1 proposes candidates, they're scored against your dataset, and the critique steers the next round. When the inner layer stalls, an outer layer (L2, then L3) redirects it; when the branch itself is spent, the search rewinds. Full mechanics: the-loop.md.

Compared to GEPA, and to AlphaEvolve

GEPA (arXiv:2507.19457, ICLR 2026 Oral) is the reference point for prompt optimization today, and ships inside DSPy as dspy.GEPA; AlphaEvolve is the closest published peer on the code-evolution side. Same family and the same reflective core — a critique of what failed drives the next proposal. GEPA evolves the instruction text of a DSPy program, AlphaEvolve evolves source code, PromptPotter evolves prompts and the pipeline parameters around them, jointly. That is scope — a different target, not a missing feature; the table below is capability.

🟡 is a partial version of the same capability — GEPA gates on a minibatch where PromptPotter runs a sequential test, and its checkpointing lives in the standalone gepa package rather than on the DSPy path; AlphaEvolve's are that capability expressed at the code level. A — is a closed hosted service whose behaviour is not publicly documented. Where each is ahead of us: GEPA on reach — it sits inside DSPy, with that ecosystem's adapters, tracing and audience, which is why PromptPotter ships as a DSPy optimizer rather than asking anyone to leave it; AlphaEvolve on code optimization, its search reaching into source, which is not what PromptPotter is pointed at. Full grading: related-work.md — and DSPy's own engine, read from its source, in § What a DSPy source study settles.

Use it as a DSPy optimizer

pip install promptpotter[dspy], then — where you would write dspy.GEPA(…):

from promptpotter.presentation.teleprompter import PromptPotterOpt

optimizer = PromptPotterOpt(metric=my_metric, dataset_name="my-task")
compiled = optimizer.compile(my_program, trainset=trainset)

Your metric stays the scorer, and compile() returns a copy of your program with the winning prompt applied. It mints a real campaign on disk, so PoBB pruning, Rasch scoring, the block library, the measurement cache, the spend ceiling and the CLI verbs (pause, resume --from N, fork) all come along — what you trade away is the operator surfaces, not the search. Full guide.

Citation

@software{promptpotter,
  title  = {PromptPotter: LLM-Driven Evolution of Prompts and Pipelines},
  author = {Streuli, David},
  year   = {2026},
  url    = {https://github.com/PromptPotter/prompt-potter-optimizer}
}

Benchmarks

A head-to-head against DSPy's optimizers (GEPA, MIPROv2, BootstrapFewShot) and CAPO on BIG-Bench Extra Hard (BBEH), held to a standard this literature mostly does not hold itself to: every method scored on the same held-out rows, one published split seed, one export schema for every method, and no cross-paper number mixing. Split, seed, metric and export schema are pinned in docs/research/bbeh-comparison/ — Colab notebooks for the peers, local for PromptPotter — and numbers publish once the target model and the optimization budget are held constant across every method too. What we measure on, what we refuse to measure on, and why: docs/research/benchmarks.md.

Documentation

Developer internals (Python symbols, data contracts, wiring) live under docs/developer/. Statistical foundations under docs/methods/.

Watching a run

While python -m promptpotter resume is running, the cleanest setup is campaigns/{campaign_id}/cycles/{cycle_id}/dashboard.json open in an auto-reloading editor + the CLI terminal visible. dashboard.json is the live scalar state (phase, round, candidate, accuracy, in-flight query); the CLI prints HIT/MISS lines + per-candidate + per-round banners as they happen. Drill-down peers in the same directory: rounds/, log.md, index.json. Internal resume + audit state lives under .runtime/ (hidden by convention). Alternatives: /potter-run Claude Code skill, the notebook, or the read-only webapp at http://localhost:8001/. Full guide in docs/manual/04-reading-the-output.md.