evalroute

Route a task to the right (model, reasoning effort) arm before you start. evalroute is a Python package built around the evalroute procedure — a harness that measures cost per verified success per task lane (evalroute/harness/ tier_a.py: measure, then serve the results) — and turns that output into a route table with provenance on every row: pick the lane, pick the model, pick the effort, as data.

The Hermes plugin is a thin adapter over this package, at keppy/hermes-plugin-evalroute (/route, the evalroute_route tool, the first-turn sniff, and the bundled skill live there; the routing core lives here).

Install

pip install evalroute          # runtime: pyyaml only
pip install "evalroute[hub]"   # + huggingface_hub, for `sync`

Standalone CLI

The same argparse tree the plugin registers, standalone and byte-identical in output to hermes evalroute ...:

evalroute route --lane routine-coding "fix the failing test"   # route card
evalroute route --json "read this 80-page spec and summarize"  # JSON envelope
evalroute dispatch brief.md          # route a brief, spawn hermes chat on that arm,
                                     # print the rate line (--dry-run prints the argv)
evalroute sync --status              # which route table is active
evalroute rate pass --note "why"     # label the last routed task
evalroute install-routes --dry-run   # route table -> agent.reasoning_overrides

Works with no Hermes installed: the Hermes home falls back to HERMES_HOME or ~/.hermes, and the ledger, dataset pins, and effort-override reads all honor it.

The nine-name contract

The plugin adapter relies on exactly these names; everything else in the package is private to the library (see evalroute/contract.py, CONTRACT_VERSION = 1):

The route table

evalroute/data/routes.yaml — one row per lane: id, keywords (the rule layer), model, effort, escalation, provenance, notes. Lane taxonomy is the union of the two source tables (9 lanes); where they disagreed (long-doc merged into web-research in one, orchestration only in the other) both are kept as distinct lanes. Model ids must match /model spelling exactly.

The table began with a 2026-09-26 priors snapshot (public benchmarks, many vendor-run). Three lanes now have small measured batches; the others remain marked priors. Replace those rows only after your own controlled data, and keep each row's provenance visible.

Route table: bundled or synced

The table you route against is either the bundled one or a pinned dataset revision; the card says which. The dataset is fetched only when you run sync (never on install, never while routing), it is pinned to a resolved revision, and the library routes fully offline without it:

evalroute sync --revision <sha>   # pin the published table (default: main)
evalroute sync --status           # bundled, or dataset @ <sha>
evalroute sync --clear            # back to the bundled table

sync downloads only the routes/ config of keppy/evalroute-flywheel into <hermes home>/evalroute/dataset/<sha>/ — never the measured evidence (grows over time; leave it on the Hub). It needs pip install huggingface_hub (the hub extra). Routing data lands only under the Hermes home, like the ledger.

Regenerating from measured data

# in the harness venv (openai + anthropic; it stays out of the runtime venv)
python -m evalroute.harness.tier_a run -m models.json -t tasks.jsonl -k 3
python -m evalroute.harness.tier_a report -o runs.jsonl -t tasks.jsonl -k 3 --csv report.csv
python -m evalroute.routes_from_report --csv report.csv --runs runs.jsonl \
    --models models.json --k 3 --out routes.generated.yaml

report needs the matching --tasks file: it checks each run's prompt/checker contract and treats missing declared tasks as incomplete. Without that file it prints diagnostics but selects no route. The report.csv files under examples/artifacts/ were regenerated from their runs.jsonl and tasks.jsonl with this harness and carry the complete column. A legacy CSV (no complete column) is refused even with --runs, since the old winner selection may have ignored pending cells; regenerate it the same way.

The harness is packaged so the loop is complete inside one repo: write tasksets (deterministic python checkers where possible — validate every checker against a reference solution before paid runs), run k samples per arm, report, then flip the lane's row. routes_from_report applies the report's own routing rule (coverage-gated lowest all-in $/success), stamps provenance: measured ... with a gonogo McNemar stamp when the winner and runner-up shared cases, preserves each lane's keywords/match_hint/ escalation/notes, and carries unmeasured lanes over verbatim — regeneration never silently deletes a route. The tier-a tasksets and runs that produced the current measured lanes are under examples/artifacts/ (sets: tasks.jsonl; raw run records: runs.jsonl).

Inspect the generated YAML before replacing the bundled table. --models maps harness arm names such as glm-5.3@high to /model IDs; omission is only safe if the CSV already contains routable IDs. --k defaults to 3; graded sample counts must divide evenly by k, or the generator refuses to invent a task count. The harness v2 resume key includes the full task and checker spec, model price/config and effective max tokens, plus the judge configuration when used. It keeps legacy JSONL reportable, but reporting mixed legacy/v2 or multiple prompt/checker versions together fails explicitly.

The Hermes shim (api: "cmd" arms)

To run Hermes itself as an evalroute arm (agentic cells):

{"name": "hermes-glm@high", "api": "cmd", "effort": "high",
 "cmd": "python -m evalroute.runners.herbes_shim --model {model} --effort {effort} --prompt {prompt_file}",
 "model": "z-ai/glm-5.3", "in": 0.91, "out": 2.86, "timeout": 3600}

The shim wraps hermes -z (one-shot; tools, memory, AGENTS.md loaded as normal; approvals auto-bypassed), reads the usage report (--usage-file), and emits the contract evalroute expects: the answer on stdout, then one JSON last line {"text": ..., "usage": {"inp", "out", "cache_read"}}. A non-zero hermes exit becomes an error field in that line (the harness records an error row and retries on the next run); the shim itself always exits 0. --system is prepended to the prompt. --hermes PATH overrides the executable (tests use this to point at a fake — no real runs).

Classification: rules first, LLM when weak

The classifier's first layer is deterministic keyword rules over evalroute/data/routes.yaml — free, no API calls — but rules alone misroute paraphrase ("manage life, writing, and researchy tasks" has zero keyword signal) and negation ("not usually hard math though" used to count as a math hit; the rules now guard negated keywords). So routing is two-layer:

1. Strong rules (2+ distinct keyword hits on the winning lane) — trusted outright, no LLM call.
2. Weak signal (0-1 hits) — one structured call via the facade set with routing.set_llm_facade (the host's own model and auth; the plugin sets it at register time, but it consumes tokens and may incur provider charges). None (the default, and what a bare evalroute CLI sees) disables the fallback. The LLM judges what the work IS — a description of an assistant's duties routes to orchestration, not to whatever nouns appear.

The card always prints which layer decided: rules match, LLM fallback, or no keyword hit - defaulted. If the LLM call fails (offline, no facade), the weak rules result stands and the card says so. Pin manually with --lane <id> when you know better.

Facets: labels with dimensions

A lane is the routing decision; facets are the label. Every route also captures the task's shape along three axes, defined in evalroute/data/facets.yaml:

• input-shape: long-doc | interactive
• domain: domain-dlml | domain-alignment | domain-math | domain-prose | domain-research
• demand-tier: tier-routine | tier-hard | tier-orchestration

So "audit my RL training plan files" is recorded as long-doc + domain-dlml + tier-hard — three facts about one task — instead of one collapsed lane. The rules layer derives facets from keyword evidence (conservative: only lanes that drew hits claim facets); the LLM fallback names them semantically in the same structured call.

When a task claims facets on multiple axes, the card describes the conjunction. Facets do not alter the chosen arm: the lane classifier selects the arm; no domain/tier precedence is implemented:

facets: long-doc + domain-dlml (descriptive conjunction; lane chooses arm)

Facet conjunctions aggregate in routes_from_labels, so the high-dimensional nodes — "how do long-doc x dl-ml tasks fare on arm X?" — fill in from daily use without controlled-batch spend.

Flywheel: labels from daily workflow

The controlled harness is not the only source of data. As you route in daily sessions, the library quietly builds an observational dataset:

• route logs the assignment (lane, recommended arm, method, confidence, facets) to <hermes home>/evalroute/labels.jsonl — the task text you typed is the label.
• /model or /reasoning after a route logs a process-global switch observation. Without a session join it is not a verified route rejection and cannot supply the actual arm for a flip.
• rate pass|fail [--route-id <id>] [--lane <id>] [--model <id> --effort <level>] [--note ...] labels the outcome when you finish. Use both arm flags to self-report what actually ran; otherwise the arm stays unknown. --lane corrects a lane; skip discards that route. Prefer --route-id in overlapping sessions.
• Nothing else is recorded: no response bodies or turn telemetry. Task text, switch arguments and optional notes are recorded. The last-seen model is process-global memory only and is not assigned to a route as fact.

The ledger therefore holds two row qualities: an observed arm (a /model or /reasoning switch after a route — a process-global candidate, not proof) and a caller-stated arm (evalroute dispatch <brief.md> records the spawn arguments as arm_attribution: explicit_user on the outcome row). dispatch is the one-line form of the flywheel: route the brief, spawn hermes chat on exactly that arm, print the rate it: line — it still never auto-rates pass.

The ledger is profile-wide, not session-scoped: command hooks do not supply a reliable session ID for route and rate. The card prints a route ID; when tasks overlap, select it with --route-id. Without it, rate consumes the latest pending route in that profile, which may be another session's. /model and /reasoning observations are process-global candidates, not proof of which model served a given route; only an explicit rate --model ... --effort ... confirms an observational arm. To replace a pinned card, use route --lane <lane> --replace-route-id <old-id> <same task>; identical task text alone never consumes another pending route. Inspect the route ID, task and lane before trusting a label.

Continuing across sessions (turn caps). A task that outlives its session — the turn limit hits, the terminal closes mid-task — is a continuation, not a new route. The row being labeled is (task, arm), not (task, session):

• Prefer staying in the session: continue: <what remains> gets a fresh iteration budget, and the arm is session-scoped and persists.
• Otherwise hermes -c continues the same conversation, or paste the capped session's final turn as the new session's opener.
• Never re-route the continuation. If the new session is a different task, rate skip --route-id <id> clears that specific pending row first if it belongs to you; don't consume another user's pending row.
• Steer as much as you like. The observed layer is defined as daily-workflow-with-a-human-in-the-loop; a directive continuation is normal operation, and it matches a detailed original prompt better than a bare "continue" — which quietly tests prompt-luck instead of the arm. Measured rows are untouched: they come from fixed-prompt, fresh-context harness cells.
• Put the methodology in the note: --note "completed across two sessions (turn cap), directive continuation". The label records neither cost nor session boundaries, and a session-spanning pass re-reads the accumulated context at full input price — the note is where that lives.

Turning labels into route data: python -m evalroute.routes_from_labels prints per-lane, per-arm pass rates, lane corrections, and facet conjunction outcomes; --apply writes routes.observed.yaml. Observed rows carry honest, weaker provenance:

observed 23 outcomes, same-maintainer observational single-arm, pass 78%, 2026-10-30;
not independent trials or a controlled comparison

and only where the lane has no measured row — observational data can contest a priors row, never overwrite a measured one. When gonogo is installed, each observed row also carries its decide() verdict, so a row with 3 outcomes reads as INSUFFICIENT_EVIDENCE rather than a pass rate someone will trust. The one provisional flip threshold is 3 user-confirmed arm failures on the recommended arm and 2 user-confirmed wins on another observed arm. This is still same-maintainer, non-randomized evidence — review it and verify with paired controlled cases before treating it as a quality comparison. The two same-arm outcomes in the old snapshot never justify a route flip.

Publishing the labels. The live ledger stays local and append-only; do not check raw labels into a public repo. Task text and notes can expose paths and private project details even without response bodies. Do not claim retroactive erasure for any label data once shared.

What the harness measures

Three lanes measured with the evalroute harness via the Nous inference API — 10 tasks x 3 samples per arm (four routine-coding, five DL/ML, five alignment arms, of which only four alignment arms have graded samples: 119 graded, 30 judge-pending, and one missing cell). Raw API and judge cost in the vendored rows totals $3.0590051, excluding verification time across routine coding, DL/ML research engineering, and alignment reasoning. Every winner was statistically indistinguishable from its runner-up at n=10 (McNemar, via gonogo) — the empirical paired gap is zero, but the conservative interval spans [-33.4%, +33.4%]; $p=1$ is not a population equivalence test. All-coverage lanes have a ceiling on this taskset; the routes choose cost among observed ties, not quality parity. The alignment judge has no blind checker audit; its 30 pending Qwen outputs are excluded from the route comparison. No quality claim spans that arm. The historical v1 run records omit model IDs; the arm-name-to-ID mapping is the vendored models.json, not an ID echoed by those records. Routine coding overturned the priors' vendor pick: glm-5.3-flash at medium effort covered every task at $0.00005/success, 2.4–3.9x cheaper than the V4.1 Flash arms at equal coverage.

Every row is priors, observed, or measured. Nothing hypothesis-shaped masquerades as a result — the card prints the row's provenance verbatim, statistical stamp included.

Known limitations

• The rule layer is keywords. Deterministic, free, and misses paraphrase — which is what the LLM fallback is for. The fallback's quality tracks whatever model the host is on; it costs one small structured call (temp 0, 256 tokens) only when rules are weak.
• The library cannot switch the model for you. route prints the card; you run /model <id>. Run it before turn 1 — mid-session switches re-read the whole context at full input price.
• One effort slot per model id (agent.reasoning_overrides); when a model serves two lanes, install-routes keeps the higher effort. A lower-effort lane's card explicitly prints /reasoning <lane-effort> after /model, because the installed override alone would run the wrong arm.
• Provenance is priors until you measure. Several rows are explicitly untested/unmeasured/contested; the card prints the provenance verbatim so nobody mistakes a hypothesis for a result.
• The harness stays out of the runtime venv. Run it in its own venv with openai/anthropic; EVALROUTE_PYTHON points the runners at that interpreter. The package itself requires nothing beyond pyyaml (huggingface_hub only for sync).
• The sniff hook is advisory only (plugin side): it speaks when the classifier is confident and the session's model disagrees with the lane's route; it never rewrites, blocks, or switches.
• install-routes needs the Hermes config module to write. Standalone, writing agent.reasoning_overrides works inside the hermes process; --dry-run works anywhere.