whetstone-ai

Generic toolkit for evaluating and optimizing LLM prompts and programs.

Whetstone sits above the dr-* libraries (graphs, providers, store, serialize, exec, platform) and below domain-specific environments. It owns the reusable experiment contract, batched evaluation engine, optimizer harness, and evidence/analysis plumbing — not task datasets, domain scoring rules, or application UI.

In scope here: evaluation at scale, a shared optimization harness, and stepping through runs to inspect behavior. Optimizers are not co-equal:

Out of scope here: particular benchmarks or envs (those live in separate packages or repos), one-off experiment scripts, and product-facing runners.

Core capabilities

1. Evaluation — batched, efficient sweeps over candidates and tasks; configurable splits, graph rollouts, concurrency, and durable evidence. Bundled reference drivers: GraphRolloutEvalDriver (eval/drivers/graph_rollout.py) — the default, parallel in-process graph rollouts with injected EvalProcedureRunner — and SubprocessGraphRolloutEvalDriver (eval/drivers/subprocess_graph_rollout.py), which runs the same rows on a dr-exec worker pool with per-row and per-batch wall-time budgets.
2. Evaluation analysis — bootstrap confidence intervals, power analysis, and anchor calibration over persisted evaluation evidence (eval/analysis/).
3. Optimization — shared harness and adapters that propose candidates and drive evaluation intents in a loop. COPRO is the platform-wired optimizer; GEPA and MIPROv2 exist as adapters (GEPA also has a step engine) but are not registered in the default runtime.
4. Sandbox & interpretation — dry-run previews and toy-graph helpers to step through optimizer behavior before spending full eval budget (whetstone-sandbox).
5. Codex MCP eval — whetstone-mcp-eval serves the Codex evaluate-candidate tool over stdio.

Evaluation  →  Evaluation analysis
     ↓
Optimization  →  Sandbox / interpretation

dr-* libraries

Stable seams

• Experiment — generation graph, initial/ceiling candidates, eval configs, reward policy
• EvaluationEngine — validates and evaluates a candidate; returns typed evidence refs
• OptimizerAdapter — COPRO plugs into the shared harness on the default runtime; GEPA and MIPROv2 adapters exist but are not platform-wired
• StepContractProvider — each optimizer declares its first-step and continuation contracts and parses its own launch control, registered by adapter key; StepRequestBuilder and HarnessRunController dispatch through it
• Step evidence — a step reports evaluations it asked the harness to run in resolved_intents, and evaluations its own search drove in search_evidence, each bound to its run and step index and verified by the harness; a terminal step whose contract sets terminal_proposal_count and that accepted no improvement over the run's own initial candidate sets seed_retained
• Graph rollouts — experiment/graph/ builds standard two-node graphs; drivers execute them per row

Platform pipeline

The optim pipeline (whetstone.optim.v1) has stages optim_step → eval_row → eval_fanin, plus run_completion. EvalDispatchMode.INLINE evaluates inside the step. EvalDispatchMode.PLATFORM persists eval intents, fans out row jobs, fans results back in, then resumes the step. Submit a run with submit_optim_run.

whetstone-optim run is a stub until step 5; it echoes the control ref and exits 2.

Sandbox

uv run whetstone-sandbox copro --task-prompt "Say hello"
uv run whetstone-sandbox graph --run

Requires Python 3.13+. Optional extras: dbos, postgres, platform.

Platform integration tests

Tier 2 tests exercise the dr-platform harness against Postgres + DBOS:

uv sync --extra platform
createdb whetstone_platform_test   # once, if needed
uv run pytest -m integration tests/integration/

Set WHETSTONE_TEST_DATABASE_URL when not using the default postgresql+psycopg:///whetstone_platform_test. Locally, tests skip when Postgres is unavailable; in CI they fail hard. Default uv run pytest excludes integration tests via the pytest marker.