Oloproof

Oloproof is a developer-first AI systems evaluation, diagnostics and decision platform. It runs an AI system on a versioned suite of cases, measures it with deterministic, custom, judge, retrieval, agent and predictive evaluators, puts an interval with the right denominator on every number, compares a candidate with a baseline, diagnoses failures by intervening on them, and decides a release as PASS, FAIL, INSUFFICIENT_EVIDENCE or MANUAL_REVIEW from declared rules. The same canonical objects are read in the terminal and in the Next.js workbench.

Oloproof is not a toy eval harness. The intended product direction is credible beside DeepEval/Confident AI, Braintrust, LangSmith, Arize/Phoenix, Galileo, Langfuse and Opik, while differentiating through statistical rigor, evaluator reliability, intervention-driven diagnosis and decision-grade evidence.

Quick Start

Four steps from nothing to a release decision on your own machine, with no account and no network, and a fifth that points it at your own application.

1. Install

python3 -m venv .venv
. .venv/bin/activate
pip install oloproof

Python 3.11 or later. The command-line tool, the engine and the SDK all run locally. Optional extras: oloproof[collect] to receive OpenTelemetry traces with oloproof collect, and oloproof[postgres,worker] for running the hosted store and worker yourself.

2. Scaffold a project

oloproof init oloproof-demo
cd oloproof-demo

init writes a small support bot to evaluate (app.py, declared with @system), 30 cases in data/example.jsonl, the suite in oloproof.yaml (the dataset, the system and an evaluator that checks each answer's label) and the release policy in release.yaml (the confidence level and a rule that the label is right at least 80% of the time).

3. Run the suite

oloproof run

The run prints each rule's decision, PASS, FAIL, INSUFFICIENT_EVIDENCE or MANUAL_REVIEW, and each metric with its interval and how many cases stand behind it. A rule passes only when the whole interval clears its threshold, not the estimate. Run it again and the system is not called: its outputs for an unchanged dataset come from the cache.

4. Gate on the decision

oloproof run
echo "exit $?"

oloproof run exits on its gate: 0 when the policy lets the release go, and a non-zero code naming why when it blocks, so a CI step that runs it needs no further wiring. To see it block, raise the rule's min in release.yaml to 0.95 and run again: 30 cases cannot show a rate that high with 95% confidence, so the rule is INSUFFICIENT_EVIDENCE and the run exits 3. The gating guide lists every code.

5. Your own application

The example shows the tool works; your application is what you came to evaluate. No baseline is needed: a first evaluation asks whether your application meets your requirements, and comparing versions comes later.

Point system in oloproof.yaml at your application: a Python callable that takes a case's input and returns a dictionary (callable: app:answer, as the example does), or an HTTP endpoint that does the same over JSON (http: {url: http://localhost:8000/answer}). Replace data/example.jsonl with a handful of your own cases, one JSON object per line with an id, the input your application takes and the expected answer or label; ten real cases are a better start than a hundred invented ones, and more can come later. Keep an evaluator that can check your expected field (exact_match on a label, contains, regex, json_schema, or an LLM judge with a rubric), and write what acceptable means as a rule in release.yaml: the metric and the rate it must clear, at the confidence level you set.

A rule's metric is an evaluator's criterion, so the two are written together. The scaffold's evaluator is named exact_label: it compares the label your application returns with the label under each case's expected. A rule on it names the same criterion:

# oloproof.yaml
evaluators:
  - type: exact_match
    criterion: exact_label
    field: label

# release.yaml
rules:
  - id: label-correct
    metric: exact_label
    min: 0.90

Rename the criterion if you like, in both places at once; a rule naming a metric no evaluator produces stops the run with a configuration error that lists the metrics there are.

Then oloproof run. PASS means the whole interval cleared your bar on your cases; FAIL means it cannot; INSUFFICIENT_EVIDENCE means the run could not settle it, and the decision's reason says why. Where the reason is too few cases, oloproof plan RUN_ID --run estimates how many more would settle it, for the methods that support sizing; where it is something else, such as executions that failed or a judge not yet validated, the reason names what to fix instead. Each is an answer about your application on the cases you gave it, and nothing more.

Evaluators

The type: values oloproof.yaml accepts:

A judge names its provider (anthropic, openai or openai_compatible), its model, and its rubric as rubric_text or rubric_file. One on a provider that speaks the OpenAI API but is not OpenAI (Groq, Together, a local server) also needs base_url and api_key_env:

evaluators:
  - type: rubric_judge
    criterion: answer_correct
    provider: openai_compatible
    model: openai/gpt-oss-20b
    base_url: https://api.groq.com/openai/v1
    api_key_env: GROQ_API_KEY
    rubric_text: "PASS if the answer conveys the same fact as the reference."

Naming a field an evaluator does not take is answered with the ones it does.

Re-decide an existing run without rerunning the system or evaluators:

oloproof gate RUN_ID --policy release.yaml

Inspect failures and one case:

oloproof inspect RUN_ID --failures
oloproof inspect RUN_ID --case refund_00

Export a bundle and open it in the web workbench:

oloproof export RUN_ID
OLOPROOF_BUNDLE_DIR="$PWD/.oloproof/bundles" pnpm --dir /path/to/Oloproof --filter @oloproof/web dev

The bundle path must be absolute, because pnpm starts the app from apps/web. The workbench opens on its workspace index; a lone OLOPROOF_BUNDLE_DIR appears there as the local/bundles project. To hold several projects on one machine, point OLOPROOF_WORKBENCH_DIR at a directory containing a workbench.json instead (docs/API_CONTRACTS.md).

Install For Development

From the repository root:

python3 -m venv .venv
.venv/bin/python -m pip install -e '.[dev]'
pnpm install --frozen-lockfile
make check
pnpm --filter @oloproof/web check

The public Python surface is:

from oloproof import evaluate, system, current_case
from oloproof.evaluators import ExactMatch, Contains, JsonSchema, Regex, RubricJudge, evaluator

Everything under oloproof_core is engine internals.

Examples

Five runnable projects, none of which needs provider credentials. Each README explains what the run shows and why it decides what it decides.

cd examples/support_bot
../../.venv/bin/python -m oloproof_cli run

The run exits 3. Its keyword-rule bot labels 13 of 18 questions correctly, which is too few cases to show label accuracy of at least 0.70, so the release is blocked with INSUFFICIENT_EVIDENCE.

Benchmark

The Slice A analysis benchmark stores runs of synthetic precomputed evidence and times what oloproof gate does with them: loading, denominators, cold and cached analysis, the clustered bootstrap and decisions. It excludes system and model latency.

.venv/bin/python benchmarks/analysis_bench.py --sizes 1000,10000,100000

Recorded results are in benchmarks/README.md.

Engine Rules

The evidence chain is:

scenarios -> execution -> artifacts -> evaluators -> judgments
           -> statistical inference -> diagnostics/interventions
           -> decisions -> evidence/provenance
           -> LLM explanation/hypothesis/next experiment

Decision states are exactly PASS, FAIL, INSUFFICIENT_EVIDENCE and MANUAL_REVIEW. Execution states such as RUN_ERROR and CANCELLED are not decision states. Release actions are separate: ALLOW, WARN or BLOCK.

For minimum-threshold interval rules, PASS requires the lower bound to meet the threshold and FAIL requires the upper bound to be below it. Otherwise the result is INSUFFICIENT_EVIDENCE. Unsupported declared structures fail closed.

Current Scope

Built:

• Python SDK and CLI for local evaluation; suites, systems and evaluators are versioned and every record is content-addressed.
• Deterministic, custom (@evaluator), LLM-judge, retrieval, agent-trajectory and predictive evaluators, and an evaluator registry that measures a judge against human labels on the answers people judged: agreement, bias and hit rates, each with an interval, and a bar on each before a judge may gate. People label from a file (oloproof labels export and import), the terminal (oloproof review) or, in a hosted workspace, a review queue in the browser; oloproof evaluators try measures a draft judge before it is adopted. Probability judges can be calibrated against labels, and a cascade sends only uncertain cases to a stronger judge.
• Judges measured without labels: oloproof evaluators probe reports how often a judge's verdict moves when it is asked again or shown padding, and oloproof prefer asks a pairwise judge which of two runs answered better, in both orders, with the judge's position bias beside the result.
• Systems of several agents: each trajectory step names the agent that took it, a hand-off is a step, and deterministic criteria judge the route against a declared one, tool calls against a closed permission map, and the number of hand-offs against a bound. No failure is attributed to an agent.
• Binary-rate, bounded-mean and quantile metrics with missingness bounds; paired and cluster-aware analysis where declared.
• Candidate-versus-baseline comparison with superiority, non-inferiority and equivalence rules.
• Declared slices with multiplicity control; a slice rule gates only inside a declared family with a correction and a minimum support.
• Intervention-driven diagnosis of RAG failures (oloproof diagnose) and sample-size planning for undecided rules (oloproof plan).
• Streaming progress with live provisional estimates, bounded concurrency for systems and judges (concurrency:), and cache reuse across execution, judgment, analysis and gate recomputation.
• CI gating through exit codes, sign-off, usage and cost reporting.
• Early stopping (early_stopping: true in a policy): a run, or a comparison, stops after a seeded batch once every rule has decided PASS or FAIL. Cases that never ran count at their worst value, so a stop is valid whenever it happens, and the run reports the work it avoided.
• Prediction-powered inference (ppi_judge_rate@1): a judge's pass rate corrected by a random, blind sample of human labels. A sample drawn on the labeller's machine is good-faith; a hosted workspace draws the sample itself and verifies the interval with its own engine.
• Verified evidence in a hosted workspace: runners an Owner registers sign the runs they execute, an Owner pins the evaluation and the release policy, and the workspace gate decides each system version on its first run of that evaluation, counting independent labellers by account.
• The Next.js workbench, locally over exported bundles or as a hosted workspace that oloproof push sends runs to. The hosted workspace is invitation-only.
• Hosted review queues: an Owner or Admin assigns reviewers, who give pass/fail verdicts, scores or A/B preferences that do not name the runs, in the browser. Content a project keeps on its own machines reaches the reviewer's browser from oloproof collect, never through Oloproof. Scores and preferences are stored and enter no statistic yet.
• Production traces received locally: oloproof collect accepts OpenTelemetry (GenAI and OpenInference conventions) and pushes them through the project's egress policy. A hosted deployment can also take spans directly, opt-in, redacted at ingest by a dedicated key's policy (docs/plans/SLICE_S.md S06).
• Production evaluation: workspace-drawn traffic samples over collector commitments, online scoring and anytime-valid monitoring, and turning a trace into a test case (docs/plans/SLICE_S.md §7).

Not built yet: the rest of the statistical roadmap in Engine §24: conformal risk control, hierarchical dependence, causal intervention analysis, IRT and a method for score and preference labels. The launch checklist (docs/roadmap/INITIAL_COMPETITIVE_RELEASE.md) records what is left before the first invited user: an independent review of the whole product, and watching that user run the golden path.

Project Instructions

Coding agents should start with:

• AGENTS.md - shared instructions and source-of-truth map.
• AGENT_WORKFLOWS.md - Claude/Codex division of labor and task prompts.
• DOCUMENTATION_STATUS.md - canonical synchronized doc versions and status.
• docs/engine/ENGINE_SPECIFICATION.md - statistical and decision contract.
• docs/plans/ - the implementation plans, one per slice, and their accepted decisions.

The top-level local repository folder is Oloproof/ with a capital O. Python packages, npm package names, CLI commands and environment variables remain lowercase or uppercase as documented: oloproof, oloproof_core, @oloproof/..., OLOPROOF_*.

License

Oloproof is licensed in two parts. The engine, CLI, SDK and runner (python/), the UI packages (packages/) and everything else a developer installs are under the Apache License 2.0. The hosted control plane and workbench (apps/web/) are under the Elastic License 2.0: you may use, modify and self-host it, and you may not offer it to others as a managed service. NOTICE says which covers what.