Prompt Playoff — Prompt Optimization and Benchmarking for Local LLMs

Deterministic prompt-technique selection, compilation and benchmarking for local LLMs: it picks the technique your task needs, compiles the prompt that technique implies, measures it on your own Ollama or OpenAI-compatible model, and searches for a better one — chain-of-thought, self-consistency, ReAct, schema-first extraction and 25 more, ranked on your data rather than on a blog post's opinion.

# macOS
git clone https://github.com/KazKozDev/prompt-playoff.git && cd prompt-playoff && ./start.command

# Linux (Python 3.11+, after cloning)
python3 -m venv .venv && .venv/bin/pip install -e '.[all]'
.venv/bin/python -m prompt_playoff serve

Double-click start.command on macOS. Linux runs the same server from the checkout.

Quick start

1. Run the commands above. On macOS, start.command finds a Python 3.11+ interpreter, builds .venv, installs every optional extra, offers to install Ollama and pull llama3.2:3b, picks a free port in 8000–8020, and opens the browser. It rebuilds the environment when the checkout moved, because an editable install pinned to an old path imports fine and runs the wrong code.

2. Keep Ollama running with at least one local model. Selection and compilation work without one; only measurement needs it.

3. Describe the task, read the ranking, then measure the winner instead of trusting it:

   describe → ranked techniques with reasons → compiled prompt → measured result → optimized prompt
   

   The same four steps are recommend, compile, benchmark and optimize on the command line.

Which prompting technique to use for your task

Selection runs in two passes. Hard constraints come first — declared capabilities, tool access, call budget, model class, and whether the evidence is supplied or still has to be fetched. Whatever survives is ranked on what the request looks like: dependent steps, a checkable answer, a fixed output shape, a long input, real cost of error. Not on its task type alone.

Every recommendation and every rejection carries its reasons, and says whether the number behind it was declared or measured:

Schema-first output (structured.schema-first)
  • Strong declared fit for structured_extraction.
  • Built for this request being exact format, verifiable.
  • Unmeasured prior 0.91 from: task:structured_extraction, provider:ollama, default.
  • Executes as single (1 call minimum).

The registry ships 29 techniques across 7 execution strategies — 13 single, 11 multi_stage, and one each of self_consistency, map_reduce, tool_loop, program_of_thought and tree_search. 22 of them carry the paper they come from, and the catalog links it.

Compilation then turns the task into the prompt that technique implies — its own blocks, stages and call count. Schema-first and map-reduce do not produce the same prompt with a different label on it:

prompt-playoff compile --task structured_extraction \
  --input-file examples/book_excerpt.txt --schema-file examples/entity_schema.json \
  --technique structured.schema-first --capabilities structured_output,system_messages

Benchmark and compare prompt techniques on your own model

benchmark runs the compiled prompt on a dataset and prints what the model actually did next to what the registry claimed:

prompt-playoff benchmark --model llama3.2:3b --model-class small --dataset entity-extraction --repeats 3

                Measured: Schema-first output on llama3.2:3b
  Metric                      Measured   Declared
  quality                        0.867      0.840
  reliability                    1.000      0.960
  contract pass rate             1.000          —
  stability across repeats       1.000          —
  mean latency (s)               0.936          —
  mean tokens                    204.3          —

quality is the headline grader for that data — field_f1 here, so 3 of 4 entities scores 0.86 rather than 0. reliability is contract pass rate × stability, because a technique that emits valid JSON every time but a different answer each time is not reliable. stability needs --repeats > 1 and reports the share that produced the modal answer. Latency, tokens and calls are summed across every call the technique makes, so a three-sample technique reports three calls' worth of cost.

compare ranks several techniques on one dataset, weighted by your priorities:

prompt-playoff compare --model llama3.2:3b --model-class small --dataset entity-extraction \
  --techniques structured.schema-first,structured.few-shot-repair,reasoning.self-consistency,direct.explicit-constraints

  Technique                       Weighted  Quality  Reliability  Latency s  Tokens  Calls
  direct.explicit-constraints        0.962    0.891        1.000       0.64     127    1.0
  structured.schema-first            0.833    0.867        1.000       1.11     204    1.0
  structured.few-shot-repair         0.711    0.700        1.000       1.51     366    2.0
  reasoning.self-consistency         0.706    0.775        1.000       2.08     499    3.0

That contradicts the registry, which priors structured.schema-first at 0.95 for this task: on a 3B model the plainer technique wins. Results are recorded to benchmark-results/measurements.json and reused for later ranking, labelled measured instead of prior only.

Eleven datasets ship with the package, from a 6-example smoke set to entity-extraction-hard, multiconer-en and few-nerd at 200 each. prompt-playoff list-datasets prints their sizes and how many carry gold answers.

Automatic prompt optimization, natively or with DSPy

prompt-playoff optimize --model llama3.2:3b --model-class small \
  --dataset entity-extraction --technique structured.schema-first \
  --rounds 3 --token-cost 0.3 --export my-technique.yaml

The loop: seed candidates (baseline, plus few-shot demos bootstrapped from the train examples the baseline already gets right) → benchmark each on the train split → score with your priorities over measured quality, reliability, latency and tokens → feed the worst failures back to the model and ask for better instructions → repeat → verify the winner on data it never optimized against.

  Metric (held-out)   Baseline   Optimized     Delta
  quality                1.000       1.000    +0.000
  reliability            1.000       1.000    +0.000
  mean tokens          211.500     201.000   -10.500
  mean latency s         0.826       0.741    -0.086

Only instruction blocks are mutable — a candidate cannot win by dropping the output contract. The Pareto front over quality, reliability, tokens and latency is reported next to the scalarized winner, so a cheaper-but-slightly-worse prompt stays visible instead of being averaged away.

--backend swaps the search algorithm and nothing else: the prompt is still built by this project's compiler, executed by the technique's own strategy and graded by its graders. native reflects on measured failures, dspy:mipro searches instructions and demonstrations jointly, dspy:gepa is reflective and Pareto-selected, dspy:bootstrap does demonstrations only and needs no proposer model.

prompt-playoff optimize --model llama3.2:3b --model-class small \
  --dataset entity-extraction --backend dspy:gepa --max-metric-calls 60

On entity-extraction-hard (40 examples, 26 train / 14 held out, llama3.2:3b), MIPROv2 beat the native loop by +0.064 F1 over three repeats, using about half the model calls. Neither recovered the dataset's annotation rules, because the proposer was the same 3B model — full write-up with the failure modes.

That last point is why the proposer is a separate profile. PROMPT_PLAYOFF_ENGINE_MODEL puts a stronger model on the job of writing candidate prompts while the numbers still describe your target model, and a run where the two are the same says so in its notes:

Candidate prompts were written by llama3.2:3b, the same model the numbers describe. Part of the gain may be that model's own phrasing rather than a better prompt.

Selection, compilation and grading stay LLM-free either way: the engine never chooses the technique and never scores anything.

Prompt regression testing in CI

Commit prompt-playoff.yaml with the model and the thresholds your build promises:

version: 1
model:
  provider: ollama
  model_id: llama3.2:3b
  model_class: small
  capabilities: [structured_output, system_messages]
checks:
  - name: entities-schema-first
    technique: structured.schema-first
    task: structured_extraction
    dataset: entity-extraction
    repeats: 3
    require:
      quality_min: 0.85
      reliability_min: 0.95
      mean_total_tokens_max: 300
      p95_latency_seconds_max: 2.0

prompt-playoff check

Exit code 0 means every bound passed, 1 means at least one regression, 2 means invalid configuration such as an unknown dataset or an unreachable provider. Requirement names are explicit fields ending in _min or _max, and an empty require block is an error — a check can never pass while enforcing nothing. --json prints machine-readable output; --update rewrites the committed bounds to the current measurements, preserving YAML comments and key order.

For wider matrices, hand the work to promptfoo:

prompt-playoff export-promptfoo --techniques structured.schema-first,direct.explicit-constraints \
  --models llama3.2:3b,qwen3.5:4b --model-class small --dataset entity-extraction --output promptfoo
cd promptfoo && promptfoo eval && promptfoo view

The export writes the compiled prompts with {{input}} templated, a config covering techniques × providers, and a Python assertion bridge that calls this project's graders — so promptfoo reports the same field_f1, not a different metric wearing the same name. Multi-call techniques export their first stage only, and the command says so instead of silently truncating.

Build datasets from traces and public corpora

prompt-playoff import-traces --output datasets/from-prod.jsonl --limit 200
prompt-playoff import-hf multiconer-en --output datasets/multiconer.jsonl --limit 200

With PROMPT_PLAYOFF_TRACING set to langfuse or phoenix, every call of every technique becomes its own span, and import-traces turns observed traffic into a dataset. Rows arrive with expected: null and tagged unreviewed — a trace has no gold answer, and pretending otherwise would benchmark a model against its own past mistakes.

Four presets convert Hugging Face corpora instead: MultiCoNER v2 and Few-NERD for extraction, GSM8K for reasoning graded on the number, and MBPP for code graded by running its own tests. Licence and citation are printed on every import.

How it differs from DSPy, promptfoo and PromptWizard

DSPy optimizes the prompt inside a module you have already chosen: you write dspy.ChainOfThought or dspy.ReAct yourself, and the optimizer tunes that module's instructions and demonstrations. Prompt Playoff makes the choice you would otherwise make by hand, deterministically and with its reasons printed, and then hands the winner to DSPy if you want its search — --backend dspy:mipro and dspy:gepa run against this project's compiler and this project's graders.

promptfoo measures prompts you have already written. It is a test harness, not a designer, and it does not tell you which technique the task needs. Prompt Playoff produces the prompt to be measured; export-promptfoo then writes a promptfoo project whose assertions call this project's graders, so both tools report the same field_f1 rather than two metrics sharing a name.

PromptWizard and other agent-driven optimizers ask an LLM to critique and rewrite instructions. Prompt Playoff does that too, but only during optimization: selection is scored constraints, grading is deterministic code, and no model is ever asked how good its own answer was.

When not to use it

Do not use it for one prompt on one task that already works — selection needs something to rank against, and every number here comes from examples with expected answers. It pays for itself when several techniques are plausible, when you have a dataset with gold answers, when picking wrong is expensive, or when the choice has to be defended to somebody else.

It is also the wrong tool for open-ended prose. There is no LLM judge in this project, so quality is measured as field overlap, grounding overlap, contract compliance and constraint coverage — not as whether the writing is good.

How it works

One Python package with a Typer CLI, a FastAPI service and a YAML registry.
The normalizer turns a description into a TaskProfile.
The selector filters on hard constraints, then ranks on the shape of the request, and prints a reason for every accept and reject.
The compiler builds the blocks and stages that technique implies, and a strategy executor issues its 1..n provider calls.
Deterministic graders score the result, and the measurement is stored where ranking, the optimizer and the CI gate all read it.

Task → TaskProfile → Selector → Compiled prompt → Model calls → Graders → Measurement

Adding a technique is one YAML file and no Python — see docs/extending.md.

prompt-playoff new-technique structured.my-technique
prompt-playoff validate-registry     # placeholders, strategies, graders, render probe

GET   /v1/capabilities  /v1/techniques  /v1/datasets  /v1/lint  /v1/integrations
POST  /v1/recommend  /v1/select  /v1/compile  /v1/author  /v1/run
POST  /v1/benchmark  /v1/compare  /v1/optimize  /v1/export/promptfoo
GET   /v1/jobs  /v1/jobs/{id}  /v1/measurements

docker build -t prompt-playoff .
docker run --rm -p 8000:8000 prompt-playoff

python3 -m venv .venv && source .venv/bin/activate
pip install -e '.[dev]'
make test          # pytest
make lint          # ruff
make validate      # registry lint

308 passed in 12.81s

License

Prompt Playoff is free and open-source software licensed under the MIT License.

Issues · Contributing · Extending · Integrations · Papers · LICENSE · LinkedIn