LLM-driven evolution of prompts and pipelines

Hand it a labeled dataset and a pipeline. It evolves the prompt and the pipeline's parameters together,
cuts a losing candidate the moment the evidence says it will not win, and shows you every measurement behind the winner.

Try it without installing: promptpotter.com — 10 free optimization runs on your own data, up to 10 rounds each, in the browser and on my key. Offer stands until 3 Dec 2026; bring-your-own-key is in the works.

What it does differently:   Losers cut early · Pipeline, not just prompt · Budget · No self-grading · Campaign on disk
Try it, then read on:   Quickstart · App tour · How the search works · vs GEPA / AlphaEvolve · The bench · Five ways in · Hack on it · Going deeper · Docs

_{The control plane on a real, finished campaign — seven rounds on JustLogic. Nothing here is staged: the thread itself reports that round 6 resolved nothing.
Every figure on this page is an in-campaign reading from a small run, not a benchmark result.}

One round, replayed

Prompt optimization that stops paying for losers. Every measurement costs money, so the whole design is most fitness per dollar.

_{One round of a real campaign, replayed line for line from its readout.log — the gutter is the source line number. Candidate 3 is cut after 7 of 34 samples; the other 27 are never bought.}

Each generation, an LLM mutates the parent into a small population. Every individual walks the same samples, and after each one a posterior asks a single question: what is the probability this candidate ends up the round's best? Below the threshold, measurement stops — Posterior-of-Being-Best. The survivor is elected and breeds the next generation. The drawing is the replay above, read out of the same log.

Four more claims follow, each with the screen that backs it.

_{↑ map}

It moves the pipeline, not just the words

Model, temperature, node thresholds, the output schema — whatever your pipeline.yaml declares tunable — are part of the individual, so they mutate and are selected jointly with the prompt fields. In the round replayed above, two of the three mutations were to a schema field description, not to the prompt. The run card says which prompt fields and which node parameters the best candidate changed, what it cost, and — on the cells both were measured on — how many samples it now gets right, and wrong, against the origin.

The budget goes where candidates separate

Samples everyone aces or everyone fails are noise, and paying for them buys nothing. The hard-sample leaderboard orders the dataset by how much each sample tells you about which candidate is better, with a fitted difficulty δ beside it, and scoring goes preferentially to those. Measurements are content-addressed: a configuration already scored on a sample is replayed, never re-bought — and the replay still shows its full cost on the ledger.

Every number opens down to the call. One cell = one candidate on one sample: prediction, ground truth, the exact input each node received, tokens, seconds — and whether it was paid for or replayed from the archive.

Every candidate, with its interval. Accuracy, ability θ and the shared-cell overlap for each individual across all generations, with the lineage beneath. The caveat above the chart is served by the backend, not decided in the browser.

It does not grade its own homework

An optimizer that picks its own winner is the easiest thing in this field to fake: with three arms and no real difference between them, some arm scores above its parent most of the time. PromptPotter still promotes the best of what it saw — and tells you, per round, on screen and on disk, that the lift interval spans zero. The terminal replay does the same in its own words: its pick is printed with p=0.42 (ns) beside it.

Three guards sit behind that. Scores are a subset-invariant ability (Rasch θ), so a candidate that drifted onto easier samples cannot out-rank an honest one. Degenerate candidates — a constant answer, a shape that games the scorer — are caught before they count. And the layer that validates a fix is never the layer that proposed it. What a winner's number may claim.

A campaign is an object on disk

Not a script that has to finish. The file tree is the dashboard: each round is a JSON file you can open in an editor, and the app renders the same file as a scoreboard. Pause a campaign, resume it, rewind to any round, or fork a sibling and compare:

python -m promptpotter resume --from 3              # rewind in place
python -m promptpotter resume --fork-on-divergence  # sibling cycle at the divergence point

Changing the scoring formula loses no results — traces are facts, scores are policy — and every artifact stamps the ledger position it was written at, so any past moment is reconstructible (how).

The whole app in one walk

_{The thread, the run card, the optimizer's own loop, the sample leaderboard, one measured cell opened. 📺 Longer: Live #2 — a Swiss-invoice campaign, run raw.}

_{↑ map}

Quickstart

Python 3.13+ and an OpenRouter key — the optimizer's default provider.

pip install "promptpotter[all]"
python -m promptpotter new tickets.csv --set task_description="classify support tickets by urgency"
python -m promptpotter resume

Bring a labeled dataset — input/output pairs as CSV, TSV, JSON, JSONL or .xlsx — and say what the job is. No key set yet? new offers to write one on first run. resume picks up wherever the last run stopped: Ctrl+C pauses a campaign rather than losing it.

To watch it in the app shown above:

python -X utf8 -m uvicorn promptpotter.main:app --port 8001    # then open http://localhost:8001/

Already in DSPy? It installs where you would reach for GEPA — what you keep and what you trade away that way is dspy-optimizer.md:

from promptpotter.presentation.teleprompter import PromptPotterOpt

optimizer = PromptPotterOpt(metric=my_metric, dataset_name="my-task")
compiled = optimizer.compile(my_program, trainset=trainset)

Six chapters from install to troubleshooting: the manual.

_{↑ map}

How the search works

One round is the unit; a campaign is rounds under three layers, and a higher layer constrains the lower one rather than replacing it:

The loop exits on a stopping rule — goal reached, budget spent, or a plateau — not on a fixed trial count. The critique-and-refine pattern is inspired by PromptWizard. Full mechanics: the-loop.md.

It searches a tree, not a trail. Each round's result flows back to every ancestor it descends from, and when a branch is spent a UCB rule picks the ancestor to re-expand — so the search climbs a different hill instead of stalling. An ancestor's score reflects what re-expanding from it actually yielded, including in branches it never ran itself. This is AlphaZero-shaped MCTS over the lineage, with deterministic evaluation on your dataset in place of rollouts: the four-phase mapping.

_{↑ map}

Where it sits

PromptPotter belongs to the LLM-driven evolution family — an LLM proposes variants, a scorer ranks them, the winners breed. On the prompt side that family is GEPA, MIPROv2, PromptWizard and CAPO; on the code side AlphaEvolve and OpenEvolve. GEPA evolves the instruction text of a DSPy program, AlphaEvolve evolves source code, PromptPotter evolves a whole declared pipeline jointly with the prompt — a different target, not a missing feature.

The table is capability, not a benchmark result:

_{🟡 is a partial version of the same capability; AlphaEvolve's are that capability at the code level; — is a closed service whose behaviour is not documented. GEPA keeps a candidate pool within a run — its 🟡 on knowledge is about what survives it. One documented exception to the pricing row: a DSPy call replayed from DSPy's own cache is not counted.}

Where the others are ahead. GEPA has reach — it lives inside DSPy, with that ecosystem's adapters, tracing and audience, which is why PromptPotter also installs as a DSPy optimizer. AlphaEvolve optimizes code, which PromptPotter is not pointed at. Opik, the closest product rival, ships six optimizers behind one API today and tunes sampling parameters too. Langfuse has production telemetry.

The bench

Peers ship an algorithm. PromptPotter ships an algorithm and the interface around it — measurement, the content-addressed archive, one spend ledger, persistence, the control plane — and the direction of travel is to run other people's optimizers inside that interface, as manifests. A menu of optimizers makes them reachable; it does not make their numbers comparable. What has to be held equal before two methods' numbers may sit in one table:

Put campaigns side by side and the first thing served is that list — what was held equal, and what stays confounded. The numbers come after.

Status, exactly: manifests for potter, capo, levi and gepa are in the tree (promptpotter/assets/optimizers/). The milestone closes when one peer — CAPO first — has run end to end in a real campaign, in a head-to-head whose only difference is the optimizer. That has not happened yet, and no comparison number is published before it does. Design and status: roadmap.md § The optimizer plug point · the argument: a menu is not a bench.

_{↑ map}

Five ways in

A sixth — PromptPotter as a tool another agent calls (MCP) — is open for discussion in #27. Opinions welcome.

Hack on it

The optimizer is itself a pipeline of nodes — the same graph the dashboard draws — so it can be pointed at its own optimizer prompts (self-optimization, L4).

git clone https://github.com/PromptPotter/prompt-potter-optimizer && cd prompt-potter-optimizer
pip install -e ".[all,dev]"
python scripts/gate.py          # every check CI runs

The engine is four seams — prompt structure, dispatch, the scoring node, cross-run memory — and the developer guide draws them on one page. Every optimizer runs end to end at zero spend (offline-run.md), so a change can be proven before it costs anything. The shape every PR is measured against: architecture.md §0. A recipe per expansion point: adding-a-surface.md. House rules: conventions.md.

_{↑ map}

Going deeper

- improved: **yes**
- samples: 34
- composite_fitness: `0.4237`
- ability θ: `+0.308` (ruler 5cd26f22e62708a4, 68 cells, 1PL)
- overlap: 34 shared cells, +16 measured: C0 41.2%  →  C1.2 44.1%  →  C4.1 52.9%
- verdict: C4.1 won on θ lift +0.216 = +0.072 over the parent + 0.144 parent selection bias (θ +0.293 vs parent +0.221, se 0.196); runner-up C4.2 at +0.149
- cost: **$0.0311** (optimizer $0.0048 c23% ·w4384 · backend $0.0137 c17% · bench $0.0127 c37%)

P(best) trajectory:
  f8356ad77a ▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅   50.0% [winner]
  6f7d643a1e ▇▆▅▅▅▅▅▅▅▄▅▄▅▄▅▅▅▅▅▅▅▅▅▅▅▅▅▅▄   49.7%
  a2169d009a ▂▂   15.8%

Benchmarks

No head-to-head number is published yet, on purpose. The comparison is against DSPy's optimizers (GEPA, MIPROv2, BootstrapFewShot) and CAPO on BIG-Bench Extra Hard (BBEH), held to a standard this literature mostly does not hold itself to: every method scored on the same held-out rows, one published split seed, one export schema, no cross-paper number mixing. Numbers publish once the target model and the optimization budget are held constant across every method too — the budget in calls, reported beside tokens, dollars (priced at run date) and wall clock. Harness: bbeh-comparison/ · what we measure on, what we refuse to, and why: benchmarks.md.

Limitations

• Parameter-based optimization only. It optimizes any pipeline that exposes tunable parameters — prompts, thresholds, model settings — but not model weights, neural architectures, or modality-specific representations.
• Requires a labeled dataset. Input/output pairs are mandatory.
• In-campaign figures are not held-out figures. The winner's own number is read off the rows that selected it, so it overstates deployment performance. A selection-clean partition is on the roadmap; published head-to-head numbers use a reserved split instead.

Documentation

Roadmap · docs index · stable API

Citation

@software{promptpotter,
  title  = {PromptPotter: LLM-Driven Evolution of Prompts and Pipelines},
  author = {Streuli, David},
  year   = {2026},
  url    = {https://github.com/PromptPotter/prompt-potter-optimizer}
}