jev-eval

Stop guessing your confidence threshold. Measure it, and get the cut-point the SDKs can load.

jev-eval measures and calibrates TypeSafe AI's Jev on your own labeled data. Give it a labeled dataset and your Jev questions. It reports per-question accuracy, whether Jev's confidence can be trusted (ECE, a reliability table), and a risk-coverage table. Then it recommends the confidence cut-points for the family's core pattern: run Jev on everything, auto-decide the confident cases, escalate the rest.

Every confidence-gated workflow (jev-sort --escalate, Leash repairAt/noteAt, a jev-guard hold threshold, an SDK consumer) needs a cut-point. Jev is ~68% accurate raw, so that number decides whether gating works at all. Set it too low and you auto-accept wrong answers. Set it too high and everything escalates, which loses the cost win that justified Jev.

Zero runtime dependencies (stdlib only), Python >= 3.10.

Quick start

$ export JEV_API_KEY=...
$ jev-eval run --data labeled.jsonl --config jev-eval.json --cache run.jsonl
$ jev-eval report --cache run.jsonl
$ jev-eval thresholds --cache run.jsonl --target-accuracy 0.95
team (choice, n=412): gate at 0.82
  auto-handle 58.7% (~242 rows) at 94.6% accuracy out-of-bag, escalate 41.3%
  95% range: gate 0.76-0.88, accuracy 91.2%-97.9%
  scored on all rows instead: 95.3% (optimistic)
wrote thresholds.json (3 questions)

Only run calls Jev (and costs money). Every other command is a pure function over the local cache, so you can re-slice with different bins, targets or guards for free.

Inputs

Labeled data (--data, JSONL): a state (string or object, sent to Jev verbatim), a labels map from question id to gold, and an optional meta object for per-slice reports. meta is never sent to Jev. Gold is a bool for Noul, a criterion key for Choice and a criterion label for Score. A row without a label for a question is skipped for that question only, so partial labels are fine. A gold value that does not fit its question (for example a typo in a Choice key) fails the run up front.

{"state": {"subject": "Charged twice", "text": "billed two times, refund please"}, "labels": {"team": "billing", "urgent": true, "sentiment": "negative"}, "meta": {"source": "email"}}

Questions, as --config (JSON) and/or inline -q flags in jev-sort's syntax:

-q 'team:choice(billing,tech,sales)'  -q 'urgent:noul'  -q 'sentiment:score(negative,neutral,positive)'

{
  "model": "jev-latest",
  "questions": {
    "team": {"kind": "choice", "instructions": "Which team should handle this ticket?",
             "options": {"billing": "payment or charge issues", "tech": "bugs, errors, outages", "sales": "pre-purchase questions"}},
    "urgent": {"kind": "noul", "instructions": "Does the message convey urgency?"},
    "sentiment": {"kind": "score", "instructions": "Overall customer sentiment", "levels": ["negative", "neutral", "positive"]}
  }
}

The config uses jev-sort's question schema (kind/options/levels). The Jev wire shape (type/criteria) is accepted too. Config files are JSON, not YAML: YAML would need a runtime dependency. jev-sort reads YAML, and JSON is valid YAML, so write a shared config as JSON and both tools read the same file. That also keeps the questions identical, which definitions in thresholds.json checks.

What it measures

One batched Jev call per row asks only that row's labeled questions. Each answer is compared to its gold label:

• Risk-coverage (the headline): for each candidate gate t, the coverage (rows auto-decided at >= t), the selective accuracy on those rows and the escalation rate. Noul also reports precision and recall on "yes".
• Calibration (the trust check): ECE, MCE and the reliability table. When ECE is above 0.1 the report warns that thresholds built on it are shaky.
• Raw accuracy: reported, but on purpose not the headline. Score also reports within-1 accuracy and mean level distance, because levels are ordered and one level off is not as wrong as the opposite end.
• Row-level gate: jev-sort escalates a row on the minimum of its Choice/Score confidences. When more than one question is gated, jev-eval scores that composite too (correct = every gated answer correct), so the recommended cut-point matches what the tool actually does.
• Per-slice (report --slice length or --slice meta.source): accuracy and ECE per slice, to catch Jev doing worse on one source or on long states. length buckets the state at 500, 2k and 8k characters.
• Chart (report --svg reliability.svg): one reliability panel per question. Bars under the diagonal mean overconfidence.
• Usage and the context budget: run and report print the input tokens Jev metered and an estimated cost at TypeSafe's list price ($0.042 per million input tokens). Jev takes 32k tokens for the state plus the longest question, and its docs do not say what happens past that. So run warns before sending a row estimated (at ~4 characters per token) within 10% of the budget, and report counts those rows. A row Jev rejects (HTTP 400, 413 or 422) is recorded with its error and counted as unanswered; the run carries on, and a rerun asks it again.

A Score answer's score is read as a level index (0..N-1, like jev-rerank). If the answer has a numeric legend ({label: position}), the nearest legend position wins instead.

Thresholds

jev-eval thresholds --cache run.jsonl [--target-accuracy 0.95 | --max-escalation 0.30] [--yes-precision 0.9,0.7]
                    [--bootstrap N | --holdout FRACTION | --no-guard] [--tolerance N] [--seed S] [--out thresholds.json]

• --target-accuracy A: the lowest gate whose selective accuracy reaches A (the most coverage at that accuracy).
• --max-escalation E: the strictest gate that still escalates at most E of rows (the best accuracy within that budget).
• --tolerance N: for Score, count within-N levels of gold as correct. Recorded in thresholds.json.
• --yes-precision P[,P...]: for each Noul, the lowest raw P(yes) whose "yes" is right at least P of the time, plus the share of real yeses it catches. This is the cut-point for tools that act when P(yes) is high, such as Leash's repairAt/noteAt (for example 0.9,0.7) or jev-guard's Noul cut-offs. Measured under the same guard as the gates. It can be the only goal.

Candidate gates are the confidences actually observed, so every recommended t can be reached. If no gate meets the goal, jev-eval says so instead of picking the closest.

Honesty guard. If you pick a cut-point using the same rows you score it on, the reported accuracy comes out too high, because you picked the cut that happened to look best on those rows. The default guard is bootstrap with out-of-bag scoring (--bootstrap 1000). It resamples your rows 1000 times. Each resample picks a gate on the rows it drew and scores that gate on the rows it left out. You get:

• the gate picked on all rows,
• an honest accuracy and coverage: the out-of-bag mean, measured on rows each pick never saw (slightly pessimistic, the safe direction),
• a 95% range for the gate and its accuracy,
• the optimistic same-rows number, shown next to it so you can see the gap.

On small datasets that gap can be large: a gate that looks like 60% in-sample on uninformative confidences scores under 50% out-of-bag. Use --holdout 0.3 on large datasets (pick on 70%, report on the unseen 30%). Use --no-guard only when you accept the optimism. Resampling is seeded (--seed), so output is reproducible.

Minimum N. Below 200 labeled rows per question the recommendation is marked provisional. Below 50 it is refused with "gather more labels" and left out of thresholds.json.

thresholds.json (contract version 1)

A small, versioned file the rest of the family can load:

{
  "version": 1,
  "model": "jev-latest",
  "generatedAt": "2026-10-04T12:00:00Z",
  "guard": {"method": "bootstrap", "resamples": 1000, "seed": 0},
  "tolerance": 0,
  "questions": {
    "team": {"type": "choice", "threshold": 0.82, "accuracy": 0.946, "coverage": 0.587, "n": 412},
    "urgent": {"type": "noul", "threshold": 0.7, "accuracy": 0.951, "coverage": 0.55, "n": 412}
  },
  "composite": {"threshold": 0.8, "accuracy": 0.92, "coverage": 0.48, "n": 412, "questions": ["sentiment", "team"]},
  "yesAt": {
    "urgent": [{"target": 0.9, "threshold": 0.86, "precision": 0.912, "recall": 0.61, "flagged": 0.31, "n": 412}]
  },
  "definitions": {
    "team": {"type": "choice", "instructions": "Which team should handle this ticket?", "criteria": {"billing": "payment or charge issues", "tech": "bugs, errors, outages", "sales": "pre-purchase questions"}}
  }
}

yesAt and definitions arrived in 1.2.0. Both are additive, so the version stays 1; consumers ignore keys they do not know. Readers: jev-sort --thresholds (1.2.0), with jev-guard and the jev-dotnet and jev-java SDKs following.

Questions that were refused, unstable under the guard, or had no reachable gate are left out, never written as null. Thresholds are on Jev's raw confidences.

Check

jev-eval check --cache this-month.jsonl --thresholds thresholds.json

Applies every gate in a published thresholds.json (questions, composite, yes cut-points) as is to a newly labeled run, and compares its accuracy there with what the file recorded. A gate has drifted when the new accuracy is lower by more than sampling noise explains: two standard errors at the recorded accuracy over the rows the gate covers now. Fewer than 30 covered rows reads as "too few rows" rather than drift. It also warns when the model changed or a question was reworded since the file was made. The exit code is 1 when any gate drifted, so a scheduled job can alert; it never rewrites the file.

Compare

jev-eval compare baseline.jsonl candidate.jsonl

Pairs the two caches on (state, question id), so the model or the question wording may differ, and that difference is the point. Per question it shows accuracy and ECE for each run, how often they agree, and what B fixes (A wrong, B right) and breaks (A right, B wrong). Use it before moving a model pin, or to A/B two wordings of a question. Rows only one run answered are counted as unpaired, never dropped silently.

Recalibration (measured, not applied)

report and calibrate answer one question: if you remapped Jev's raw confidence with an isotonic (monotone) fit learned from your labels, how calibrated would it be on rows the fit never saw (5-fold cross-validation)?

• For gating it changes nothing for Choice and Score. The remap keeps the ordering, so "gate at raw 0.82" and "gate at calibrated 0.95" auto-handle exactly the same rows. thresholds.json already captures that, on raw values.
• It matters when the confidence is used as a probability: showing "% sure" to people, expected-cost decisions, or comparing confidences across questions that are calibrated differently.
• Noul is the exception. A remap of P(yes) can move the 0.5 crossing, so the report says where calibrated 0.5 sits in raw terms ("Jev's 0.5 is really 0.42").

jev-eval calibrate --cache run.jsonl also writes an optional calibration.json: per question, input (confidence or noul), eceRaw, eceRecalibrated (cross-validated), and steps as [x_lo, x_hi, calibrated]. A raw value maps to the first step whose x_hi is >= it, or to the last step above that. Nothing in the family is required to load it: jev-eval measures Jev, it does not wrap it.

Library

from jev_eval import load_rows, provider_from_env, recommend_threshold, report, run, score_records

records = run(load_rows("labeled.jsonl"), questions, provider_from_env(), cache_path="run.jsonl")
summary = report(records, bins=10)                  # accuracy, ece, reliability, risk-coverage, recalibration
cut = recommend_threshold(score_records(records).items["team"], target_accuracy=0.95)

Also exported: compare, slice_report, recommend_all, thresholds_document, recalibration, calibration_document, isotonic, and the metric primitives (bin_by_confidence, ece, mce, selective_accuracy, risk_coverage).

Provider is a protocol (evaluate(state, questions) -> dict). Pass any object that implements it, for example a fake in tests. provider_from_env reads JEV_API_KEY and honors TYPESAFE_AI_BASE_URL for a self-host, proxy or mock. The default model is jev-latest.

The cache is JSONL, one line per (row, question): model, state and question hashes, the question definition, the raw answer, the gold label, meta, the state size, and the call that answered it with its metered input_tokens (or the error Jev refused it with). The state itself is not stored. A rerun reuses cached answers (keyed by model, state hash and question hash). Progress streams to <cache>.partial, so an interrupted run resumes, and a repeated state is paid for once.

Honest limitations

• No labels, no signal. Label quality caps everything here. ECE measured on noisy gold is an upper bound on the true error, not the true error.
• It measures Jev. It does not improve Jev. ~68% raw accuracy is a property of the model. jev-eval tells you the honest number and the best gate given it.
• Score correctness is a modeling choice (nearest label). The report states this and shows within-1 and mean distance next to it.
• Small samples lie, which is why the min-N guard and out-of-bag scoring exist. A gate that covers a handful of rows can clear any target by luck; the out-of-bag numbers and the range expose it.
• Not a gate. It recommends a cut-point for a human to review. It never changes a production threshold itself.
• Text only, 32k tokens for the state plus the longest question, like the rest of the family. Long states need summarizing first; jev-eval warns about them but cannot shorten them.

Status

1.2: check for drift, --yes-precision yes cut-points and question definitions in the contract. 1.1: usage and cost, near-limit warnings and per-row rejections on top of 1.0: run, report (with slices and an SVG chart), thresholds with the out-of-bag bootstrap guard, compare, measure-only calibrate, and the thresholds.json v1 contract, for all three question types. Tested against a fake provider at 99% coverage. Gated by the Foundry Python stack (ruff, mypy strict, pytest, pip-audit, structural smells).

MIT (c) Christoffer Maintz