promptpotter.com — 10 free optimization runs on your own data, up to 10 rounds each, in the browser and on my key. Offer stands until 3 Dec 2026; bring-your-own-key is in the works.

PromptPotter: LLM-Driven Evolution of Prompts and Pipelines

PromptPotter evolves better prompts. Most prompt engineering is manual. PromptPotter automates the generate → score → critique cycle: it tries many prompt and pipeline variations together, keeps memory across runs, and recovers on its own when a generated prompt produces broken output. Weak candidates are eliminated early on statistical confidence (Posterior-of-Being-Best — PoBB), so the budget stops going to losers. Built for RAG pipelines, LLM agents and multi-step LLM workflows — drop in via CLI, Python, the /potter-run Claude Code skill, or as a DSPy optimizer where you would reach for GEPA.

Quickstart

Python 3.13+ and an OpenRouter key — the optimizer's default provider.

pip install "promptpotter[all]"

Bring a labeled dataset: input/output pairs as CSV, TSV, JSON, JSONL, or .xlsx. Point the optimizer at it and describe the job:

python -m promptpotter new tickets.csv --set task_description="classify support tickets by urgency"
python -m promptpotter resume

No key set yet? new offers to write one on first run. resume picks up wherever the last run stopped — Ctrl+C pauses a campaign rather than losing it.

new also needs a pipeline backend to score against — the default is TermNorm on :8000, and a campaign refuses to start without one. Setting it up by hand is 03-first-campaign.md § No backend?.

Working in Claude Code? Type /potter-run instead — it downloads and starts that backend for you, then audits your setup, picks a dataset, runs init, and reads the rounds back to you as they land.

📺 Watch it work

• Live #2 — a Swiss-invoice campaign, run raw — small, unedited, cut with an opencode-go friction test.

How to Optimize LLM Prompts in 3 Steps

Describe your 1️⃣ task, drop in a labeled 2️⃣ dataset, and 3️⃣ run the loop. The task is the goal you want the AI to hit; the dataset is examples of hitting it. Each round, PromptPotter generates variations 🧪, scores them ⚖️, and keeps the winners 🏆. It stops when results plateau. ✨ Prompt optimized.

⭐ Features

Every measurement costs money, so the whole design is most fitness per dollar: the budget goes to the comparisons that actually separate candidates, and nothing is measured twice.

The capabilities PromptPotter shares with the rest of the field are in the comparison table; these are the ones it doesn't:

• 💬 Chat-first — talk to the Potter and watch it work inline, Perplexity-style: the searches, the tool calls, each round as it lands, and a button whenever a decision is yours. Ships as a reusable chat-app template — keep the chat core, delete the optimizer panes. spec
• Searches a tree, not a trail — every round's result flows back to each ancestor it descends from, so a spent branch rewinds to whichever ancestor the evidence favours and climbs a different hill instead of stalling. An ancestor's score reflects what re-expanding from it actually yielded, including in branches it never ran itself.
• Hard-sample leaderboard — score preferentially on the samples that actually separate variants; the ones everyone aces or fails are noise.
• 🛡️ Guards against self-validation — an optimizer that picks its own winner is the easiest thing in this field to fake. Three independent guards: scores are subset-invariant ability (Rasch θ), so a candidate that drifted onto easier samples cannot out-rank an honest one; degenerate candidates — a constant answer, a shape that games the scorer — are caught before they count; and the layer that validates a fix is never the layer that proposed it. What a winner's number may claim: verdict-resolution.md.
• Optimizes itself — point the optimizer at its own optimizer prompts. L4
• Pick your block library mode — proven personas, thinking styles and answer formats (from PromptWizard and the Self-Discover modules it draws on, plus what our own runs turned up). Suggest from the library, restrict to it, or switch it off.
• A campaign is an object on disk, not a script that has to finish — pause it, resume it, rewind to any round, or fork a sibling and compare. Every artifact stamps the exact ledger position it is of, so any past moment is reconstructible. How

Five ways to run it

1. WebApp — the browser control plane, at the domain root.
2. /potter-run Claude Code skill — drive a full campaign from your editor.
3. CLI — python -m promptpotter new <name> / resume.
4. REST API
5. Embedded in Python — a host program drives one campaign inside its own event loop. The DSPy optimizer is the simple-case entry into this one; it installs like any other DSPy optimizer and hands you the search, but not the operator surfaces.

A sixth — PromptPotter as a tool another agent calls (MCP) — is open for discussion in #27. Opinions welcome.

Reading a run

While a campaign runs, the cleanest setup is cycles/{cycle_id}/dashboard.json open in an auto-reloading editor + the CLI terminal visible: the file is live scalar state (phase, round, candidate, accuracy, in-flight query), and the terminal prints HIT/MISS and per-round banners as they happen. Drill-down peers sit beside it — rounds/, log.md, index.json. Alternatives: the /potter-run skill, a notebook, or the webapp at http://localhost:8001/. Full guide: reading the output.

### Round 2 — Added systematic deduction step to thinking_style to force
exhaustive derivation before defaulting to Uncertain, targeting the premature
'no direct info' pattern seen in #82, #37, #0. (75.0%)

- improved: **yes**
- samples: 28
- composite_fitness: `0.7500`

> Fix: thinking_style: Adopt a formal logical deduction method: translate each
> premise into symbolic formulas, then derive the claim's truth value using
> entailment rules. This counters the pattern of defaulting to Uncertain when
> logical entailment exists, as seen in the 'When X is true' misinterpretation.
> Axes: thinking_style, instruction, problem_description, answer_format
> Failures:
>   Logical misinterpretation (~5/14 misses): model fails to recognize that a
>   premise phrased 'When X is true, it follows Y' asserts X as true, leading
>   to over-hedging with Uncertain. Predicted: Uncertain, GT: TRUE. (Query #82)

P(best) trajectory:
  5536f04bc9 ▆▇▇▇▇▇▇████████▇▇▇█████   87.6% [winner]
  c2ac162dea ▇▆▆▆▆▆▆▅▄▅▅▅▄▄▅▅▄▅▄▅▅▅▄   50.0%

Common questions

• What does L1 actually mutate? The prompt template's fields (persona, task instruction, …) plus whatever your pipeline.yaml declares as tunable. See the-loop.md.
• My scoring formula was wrong — did I lose results? No. Traces are facts; scores are policy. The optimizer rescores on load and replays decisions; on divergence, fork. See scoring-and-memory.md.
• What if it stalls? Stall and failure are different triggers. Failures route back to the proposing layer (self-healing); stalls escalate L1 → L2 → L3 (the-loop). Stuck for other reasons: troubleshooting.

Limitations

• Parameter-based optimization only. It optimizes any pipeline that exposes tunable parameters — prompts, thresholds, model settings — but not model weights, neural architectures, or modality-specific representations.
• Requires a labeled dataset. Input/output pairs are mandatory.
• In-campaign figures are not held-out figures. The winner's own number is read off the rows that selected it, so it overstates deployment performance. A selection-clean partition is on the roadmap; published head-to-head numbers use a reserved split instead.

Where it sits

PromptPotter belongs to the LLM-driven evolution family — an LLM proposes variants, a scorer ranks them, the winners breed. Every prompt system in that family mutates one artifact, the prompt; the code-side systems mutate source. PromptPotter evolves the prompt and the pipeline parameters around it, jointly.

Who else is in the family, the line behind it, why test-time compute belongs in tuning rather than in the request, and a capability table against GEPA and AlphaEvolve: docs/research/related-work.md.

Benchmarks

A head-to-head against DSPy's optimizers (GEPA, MIPROv2, BootstrapFewShot) and CAPO on BIG-Bench Extra Hard (BBEH), held to a standard this literature mostly does not hold itself to: every method scored on the same held-out rows, one published split seed, one export schema for every method, and no cross-paper number mixing. Split, seed, metric and export schema are pinned in docs/research/bbeh-comparison/ — Colab notebooks for the peers, local for PromptPotter — and numbers publish once the target model and the optimization budget are held constant across every method too. What we measure on, what we refuse to measure on, and why: docs/research/benchmarks.md.

Documentation

Status and the full forward plan: docs/specs/roadmap.md. Documentation index: docs/README.md. The stable programmatic surface is stable-api.md; contributor rules are conventions.md.

Citation

@software{promptpotter,
  title  = {PromptPotter: LLM-Driven Evolution of Prompts and Pipelines},
  author = {Streuli, David},
  year   = {2026},
  url    = {https://github.com/PromptPotter/prompt-potter-optimizer}
}