Oloproof

Oloproof tells you whether your AI system is good enough, before it ships and once it is doing real work. It measures what the system does, puts an interval behind every decision, and decides: PASS, FAIL, INSUFFICIENT_EVIDENCE or MANUAL_REVIEW.

It works in three layers. What happened: every case, score, latency and cost, traced to the run, the judge and the data that produced it. How sure you can be: an interval with the right denominator wherever one is admitted, and judges checked against people before they are allowed to count. What happens next: pass, fail, or not enough evidence yet, with how many more cases would settle it and which part broke.

There are two ways in. Evaluate a system: hold one system to what you require of it, with no baseline needed. Compare a change: compare a change with what is live, on the same cases, and know whether the difference is real. Both run locally with no account, from the terminal or in CI, and a hosted workspace shows the same records in the browser once you push them. Once the system is live, people review a sample a hosted workspace draws from real traffic, in the browser, while the content stays on your side. The engine, CLI and SDK are free, Apache-2.0 licensed.

Quick Start

Your first evaluation, with no baseline: one version of one application, held to a bar you write down. Five steps take you from nothing to a release decision on your own machine, with no account and no model calls (installing needs the network once), then a sixth points it at your own application. Every output below is from running these exact commands in an empty directory. The terms (case, evaluator, metric, interval, gate) are defined in the core concepts guide.

1. Install

python3 -m venv .venv
. .venv/bin/activate
pip install oloproof

Python 3.11 or later. The command-line tool, the engine and the SDK all run locally. The optional extra oloproof[collect] receives OpenTelemetry traces with oloproof collect.

2. Scaffold a project

oloproof init oloproof-demo
cd oloproof-demo

init writes a small support bot to evaluate (app.py, declared with @system), 30 cases in data/example.jsonl, the suite in oloproof.yaml (the dataset, the system and an evaluator that checks each answer's label) and the release policy in release.yaml (the confidence level and a rule that the label is right at least 80% of the time).

3. Run the suite

oloproof run

The output (run ids differ on your machine):

Run run_01M4FC9Q6KT0C7GD33DV6K9628 [DECIDED/COMPLETE]
Gate: ALLOW (exit 0)
Decisions
  Rule         Metric       State  Reasons
  label-floor  exact_label  PASS   lower_bound_meets_minimum
Metrics
  Metric       Estimate  Interval         N
  exact_label  100.0%    [88.4%, 100.0%]  30 / 30 observed, 0 missing, 0 excluded
Cache: execution 0 hit/30 miss; judgment 0 hit/30 miss

How to read it:

• Decisions has one row per rule in release.yaml. label-floor asks that the exact_label pass rate be at least 80%. Its state is PASS because the interval's lower bound, 88.4%, is above 80%; the reason code says exactly that.
• Metrics shows the estimate (30 of 30 right, 100%) and its 95% interval. The interval is what decides, not the estimate: with 30 cases, a system that is truly right 89% of the time could still score 30 of 30.
• N is the denominator: how many cases were observed, missing or excluded. In 30 / 30 observed, the first number is the cases that passed (the estimate's numerator) and the second is the cases observed, so 15 / 20 observed would mean 15 passed out of 20. A case that could not be measured is counted, never silently dropped.
• In a terminal narrower than the table, its columns are cut short (INSUFFICIENT_EVIDE…). Widen the window, or prefix the command with COLUMNS=200, to see every column whole.
• Gate: ALLOW (exit 0) is the release action: nothing the policy blocks on occurred.

Run it again and the system is not called: Cache: execution 30 hit/0 miss, because neither the system nor any case's input changed.

4. Inspect a case

oloproof inspect RUN_ID --failures
oloproof inspect RUN_ID --case refund_00

Here every case passed, so the first prints 0 of 30 cases failed, errored or did not finish. The second shows one case whole: its input, its expected label, the output the system returned (reused from cache on a second run) and each evaluator's verdict (exact_label: passed). On your own application, --failures is where you start.

5. Gate on the decision

oloproof run exits on its gate, so a CI step that runs it needs no further wiring. To see it block, raise the rule's min in release.yaml from 0.80 to 0.95 and run again:

oloproof run
echo "exit $?"

Gate: BLOCK (exit 3)
  label-floor  exact_label  INSUFFICIENT_EVIDENCE  interval_overlaps_threshold
exit 3

The estimate is still 100%, but the interval, [88.4%, 100.0%], straddles 95%: 30 cases cannot show a rate that high with 95% confidence. That is INSUFFICIENT_EVIDENCE, not a failure, and the policy's block_on blocks on it. To ask how many cases would settle it:

oloproof plan RUN_ID --run

label-floor: about 42 more cases would decide it, if the observed rate holds (72 in total)

oloproof gate RUN_ID --policy release.yaml re-decides a stored run against a changed policy without running anything again. The gating guide lists every exit code.

6. Your own application

The example shows the tool works; your application is what you came to evaluate. No baseline is needed: a first evaluation asks whether your application meets your requirements, and comparing versions comes later.

Point system in oloproof.yaml at your application: a Python callable that takes a case's input and returns a dictionary (callable: app:answer, as the example does), or an HTTP endpoint that does the same over JSON (http: {url: http://localhost:8000/answer}). Replace data/example.jsonl with your own cases, one JSON object per line with an id, the input your application takes and the expected answer or label. Fifty real cases are about the floor for an interval narrow enough to decide anything, and they are a better start than a hundred invented ones. Keep an evaluator that can check your expected field (exact_match on a label, contains, regex, json_schema, or an LLM judge with a rubric), and write what acceptable means as a rule in release.yaml: the metric and the rate it must clear, at the confidence level you set.

A rule's metric is an evaluator's criterion, so the two are written together. The scaffold's evaluator is named exact_label: it compares the label your application returns with the label under each case's expected. A rule on it names the same criterion:

# oloproof.yaml
evaluators:
  - type: exact_match
    criterion: exact_label
    field: label

# release.yaml
rules:
  - id: label-correct
    metric: exact_label
    min: 0.90

Rename the criterion if you like, in both places at once; a rule naming a metric no evaluator produces stops the run with a configuration error that lists the metrics there are.

Then oloproof run. PASS means the whole interval cleared your bar on your cases; FAIL means it cannot; INSUFFICIENT_EVIDENCE means the run could not settle it, and the decision's reason says why. Where the reason is too few cases, oloproof plan RUN_ID --run estimates how many more would settle it, for the methods that support sizing; where it is something else, such as executions that failed or a judge not yet validated, the reason names what to fix instead. Each is an answer about your application on the cases you gave it, and nothing more.

Where to go next

• Choose the guide for your kind of system (classifier, text generation, HTTP service, RAG, agent, predictive model, conversation) from the start-here page.
• Change the system and compare the new version with this run: the compare guide.
• Add human review of a judge, CI and a hosted workspace: the judges, CI and gating guides.

Evaluators

The type: values oloproof.yaml accepts:

A judge names its provider (anthropic, openai or openai_compatible), its model, and its rubric as rubric_text or rubric_file. One on a provider that speaks the OpenAI API but is not OpenAI (Groq, Together, a local server) also needs base_url and api_key_env:

evaluators:
  - type: rubric_judge
    criterion: answer_correct
    provider: openai_compatible
    model: openai/gpt-oss-20b
    base_url: https://api.groq.com/openai/v1
    api_key_env: GROQ_API_KEY
    rubric_text: "PASS if the answer conveys the same fact as the reference."

Naming a field an evaluator does not take is answered with the ones it does.

See a run in the browser by pushing it to a hosted workspace. Sign-up at oloproof.com is open; login prints a short code to approve there, and nothing leaves your machine until you push:

oloproof login --url https://oloproof.com
oloproof push RUN_ID

Engine Rules

The evidence chain is:

scenarios -> execution -> artifacts -> evaluators -> judgments
           -> statistical inference -> diagnostics/interventions
           -> decisions -> evidence/provenance
           -> LLM explanation/hypothesis/next experiment

Decision states are exactly PASS, FAIL, INSUFFICIENT_EVIDENCE and MANUAL_REVIEW. Execution states such as RUN_ERROR and CANCELLED are not decision states. Release actions are separate: ALLOW, WARN or BLOCK.

For minimum-threshold interval rules, PASS requires the lower bound to meet the threshold and FAIL requires the upper bound to be below it. Otherwise the result is INSUFFICIENT_EVIDENCE. Unsupported declared structures fail closed.

Current Scope

Built:

• Python SDK and CLI for local evaluation; suites, systems and evaluators are versioned and every record is content-addressed.
• Deterministic, custom evaluators in the Python SDK (@evaluator), LLM-judge, retrieval, agent-trajectory and predictive evaluators, and an evaluator registry that measures a judge against human labels on the answers people judged: agreement, bias and hit rates, each with an interval, and a bar on each before a judge may gate. People label from a file (oloproof labels export and import), the terminal (oloproof review) or, in a hosted workspace, a review queue in the browser; oloproof evaluators try measures a draft judge before it is adopted. Probability judges can be calibrated against labels, and a cascade sends only uncertain cases to a stronger judge.
• Judges measured without labels: oloproof evaluators probe reports how often a judge's verdict moves when it is asked again or shown padding, and oloproof prefer asks a pairwise judge which of two runs answered better, in both orders, with the judge's position bias beside the result.
• Systems of several agents: each trajectory step names the agent that took it, a hand-off is a step, and deterministic criteria judge the route against a declared one, tool calls against a closed permission map, and the number of hand-offs against a bound. No failure is attributed to an agent.
• Binary-rate, bounded-mean and quantile metrics with missingness bounds; paired and cluster-aware analysis where declared.
• Candidate-versus-baseline comparison with superiority, non-inferiority and equivalence rules.
• Declared slices with multiplicity control; a slice rule gates only inside a declared family with a correction and a minimum support.
• Intervention-driven diagnosis of RAG failures (oloproof diagnose) and sample-size planning for undecided rules (oloproof plan).
• Streaming progress with live provisional estimates, bounded concurrency for systems and judges (concurrency:), and cache reuse across execution, judgment, analysis and gate recomputation.
• CI gating through exit codes, sign-off, usage and cost reporting.
• Early stopping (early_stopping: true in a policy): a run, or a comparison, stops after a seeded batch once every rule has decided PASS or FAIL. Cases that never ran count at their worst value, so a stop is valid whenever it happens, and the run reports the work it avoided.
• Prediction-powered inference (ppi_judge_rate@1): a judge's pass rate corrected by a random, blind sample of human labels. A sample drawn on the labeller's machine is good-faith; a hosted workspace draws the sample itself and verifies the interval with its own engine.
• Verified evidence in a hosted workspace: runners an Owner registers sign the runs they execute, an Owner pins the evaluation and the release policy, and the workspace gate decides each system version on its first run of that evaluation, counting independent labellers by account.
• The workbench, as a hosted workspace that oloproof push sends runs to. Sign-up for a hosted workspace is open.
• Hosted review queues: an Owner or Admin assigns reviewers, who give pass/fail verdicts, scores or A/B preferences that do not name the runs, in the browser. Content a project keeps on its own machines reaches the reviewer's browser from oloproof collect, never through Oloproof. Scores and preferences are stored and enter no statistic yet.
• Production traces received locally: oloproof collect accepts OpenTelemetry (GenAI and OpenInference conventions) and pushes them through the project's egress policy. A hosted deployment can also take spans directly, opt-in, redacted at ingest by a dedicated key's policy.
• Production evaluation: workspace-drawn traffic samples over collector commitments, online scoring and anytime-valid monitoring, and turning a trace into a test case.

Not built yet: the rest of the statistical roadmap: conformal risk control, hierarchical dependence, causal intervention analysis, IRT and a method for score and preference labels.

License

Oloproof is licensed in two parts. The engine, CLI, SDK and runner, which are this package, and everything else a developer installs are under the Apache License 2.0 (LICENSE). The hosted control plane and workbench are not part of this package and are under the Elastic License 2.0. NOTICE says which covers what.