routeval

Evals for AI pipelines that route.

For pipelines with more than one path — a model cascade, an abstain, a human handoff, a cache — routeval replays your golden set through your real pipeline and tells you whether the routing earned its money: cost per correct decision, deflection quality, and the lucky hits your accuracy number is hiding.

Why not promptfoo or DeepEval? They score outputs. They can't tell you that 40 of your 184 wins took the wrong path to get there.

Status

Pre-v0.1 — planning complete, build in progress. The launch essay is the spec; the run.json schema is the API.

The contract (all of it)

You write one function. It returns an answer, the route it took, and what it cost (or model + tokens, and routeval prices it). You point the harness at a JSONL golden set of {input, expected} — optionally with expected_route. You get a terminal report, a versioned run.json, and CI exit codes.

Metrics are tiered by what you've labelled:

Errors are never swallowed: accuracy is computed over attempted items, and failures are listed. (If your pipeline crashed on 10% of the golden set, that's a result, not an inconvenience.)

Hello world (zero API keys)

pip install routeval
routeval init      # scaffolds a toy two-route pipeline + 20-item golden set
routeval run       # full routing-economics report, no keys required

What this is not

Not a framework. Not a judge. Not a dashboard. Not hosted. It composes with pytest, promptfoo, and whatever judge you already trust.

MIT. Extracted from Quorum, where these metrics killed a threshold-tuning project in one afternoon (22% lucky-hit rate, a flat cost-quality frontier, and a 0.44-accurate escalation stage that no threshold could fix).