Oloproof

Oloproof is a developer-first AI systems evaluation, diagnostics and decision platform. It runs an AI system on a versioned suite of cases, measures it with deterministic, custom, judge, retrieval, agent and predictive evaluators, puts an interval with the right denominator on every number, compares a candidate with a baseline, diagnoses failures by intervening on them, and decides a release as PASS, FAIL, INSUFFICIENT_EVIDENCE or MANUAL_REVIEW from declared rules. The same canonical objects are read in the terminal and in the Next.js workbench.

Oloproof is not a toy eval harness. The intended product direction is credible beside DeepEval/Confident AI, Braintrust, LangSmith, Arize/Phoenix, Galileo, Langfuse and Opik, while differentiating through statistical rigor, evaluator reliability, intervention-driven diagnosis and decision-grade evidence.

Install For Development

From the repository root:

python3 -m venv .venv
.venv/bin/python -m pip install -e '.[dev]'
pnpm install --frozen-lockfile
make check
pnpm --filter @oloproof/web check

The public Python surface is:

from oloproof import evaluate, system, current_case
from oloproof.evaluators import ExactMatch, Contains, JsonSchema, Regex, RubricJudge, evaluator

Everything under oloproof_core is engine internals.

Quick Start

Generate a small project:

. .venv/bin/activate
oloproof init /tmp/oloproof-demo
cd /tmp/oloproof-demo
oloproof run

Evaluators

The type: values oloproof.yaml accepts:

A judge names its provider (anthropic, openai or openai_compatible), its model, and its rubric as rubric_text or rubric_file. One on a provider that speaks the OpenAI API but is not OpenAI (Groq, Together, a local server) also needs base_url and api_key_env:

evaluators:
  - type: rubric_judge
    criterion: answer_correct
    provider: openai_compatible
    model: openai/gpt-oss-20b
    base_url: https://api.groq.com/openai/v1
    api_key_env: GROQ_API_KEY
    rubric_text: "PASS if the answer conveys the same fact as the reference."

Naming a field an evaluator does not take is answered with the ones it does.

Re-decide an existing run without rerunning the system or evaluators:

oloproof gate RUN_ID --policy release.yaml

Inspect failures and one case:

oloproof inspect RUN_ID --failures
oloproof inspect RUN_ID --case refund_00

Export a bundle and open it in the web workbench:

oloproof export RUN_ID
OLOPROOF_BUNDLE_DIR="$PWD/.oloproof/bundles" pnpm --dir /path/to/Oloproof --filter @oloproof/web dev

The bundle path must be absolute, because pnpm starts the app from apps/web. The workbench opens on its workspace index; a lone OLOPROOF_BUNDLE_DIR appears there as the local/bundles project. To hold several projects on one machine, point OLOPROOF_WORKBENCH_DIR at a directory containing a workbench.json instead (docs/API_CONTRACTS.md).

Examples

Five runnable projects, none of which needs provider credentials. Each README explains what the run shows and why it decides what it decides.

cd examples/support_bot
../../.venv/bin/python -m oloproof_cli run

The run exits 3. Its keyword-rule bot labels 13 of 18 questions correctly, which is too few cases to show label accuracy of at least 0.70, so the release is blocked with INSUFFICIENT_EVIDENCE.

Benchmark

The Slice A analysis benchmark stores runs of synthetic precomputed evidence and times what oloproof gate does with them: loading, denominators, cold and cached analysis, the clustered bootstrap and decisions. It excludes system and model latency.

.venv/bin/python benchmarks/analysis_bench.py --sizes 1000,10000,100000

Recorded results are in benchmarks/README.md.

Engine Rules

The evidence chain is:

scenarios -> execution -> artifacts -> evaluators -> judgments
           -> statistical inference -> diagnostics/interventions
           -> decisions -> evidence/provenance
           -> LLM explanation/hypothesis/next experiment

Decision states are exactly PASS, FAIL, INSUFFICIENT_EVIDENCE and MANUAL_REVIEW. Execution states such as RUN_ERROR and CANCELLED are not decision states. Release actions are separate: ALLOW, WARN or BLOCK.

For minimum-threshold interval rules, PASS requires the lower bound to meet the threshold and FAIL requires the upper bound to be below it. Otherwise the result is INSUFFICIENT_EVIDENCE. Unsupported declared structures fail closed.

Current Scope

Built:

• Python SDK and CLI for local evaluation; suites, systems and evaluators are versioned and every record is content-addressed.
• Deterministic, custom (@evaluator), LLM-judge, retrieval, agent-trajectory and predictive evaluators, and an evaluator registry that measures a judge against human labels on the answers people judged: agreement, bias and hit rates, each with an interval, and a bar on each before a judge may gate. People label from a file (oloproof labels export and import), the terminal (oloproof review) or, in a hosted workspace, a review queue in the browser; oloproof evaluators try measures a draft judge before it is adopted. Probability judges can be calibrated against labels, and a cascade sends only uncertain cases to a stronger judge.
• Judges measured without labels: oloproof evaluators probe reports how often a judge's verdict moves when it is asked again or shown padding, and oloproof prefer asks a pairwise judge which of two runs answered better, in both orders, with the judge's position bias beside the result.
• Systems of several agents: each trajectory step names the agent that took it, a hand-off is a step, and deterministic criteria judge the route against a declared one, tool calls against a closed permission map, and the number of hand-offs against a bound. No failure is attributed to an agent.
• Binary-rate, bounded-mean and quantile metrics with missingness bounds; paired and cluster-aware analysis where declared.
• Candidate-versus-baseline comparison with superiority, non-inferiority and equivalence rules.
• Declared slices with multiplicity control; a slice rule gates only inside a declared family with a correction and a minimum support.
• Intervention-driven diagnosis of RAG failures (oloproof diagnose) and sample-size planning for undecided rules (oloproof plan).
• Streaming progress with live provisional estimates, bounded concurrency for systems and judges (concurrency:), and cache reuse across execution, judgment, analysis and gate recomputation.
• CI gating through exit codes, sign-off, usage and cost reporting.
• Early stopping (early_stopping: true in a policy): a run, or a comparison, stops after a seeded batch once every rule has decided PASS or FAIL. Cases that never ran count at their worst value, so a stop is valid whenever it happens, and the run reports the work it avoided.
• Prediction-powered inference (ppi_judge_rate@1): a judge's pass rate corrected by a random, blind sample of human labels. A sample drawn on the labeller's machine is good-faith; a hosted workspace draws the sample itself and verifies the interval with its own engine.
• Verified evidence in a hosted workspace: runners an Owner registers sign the runs they execute, an Owner pins the evaluation and the release policy, and the workspace gate decides each system version on its first run of that evaluation, counting independent labellers by account.
• The Next.js workbench, locally over exported bundles or as a hosted workspace that oloproof push sends runs to. The hosted workspace is invitation-only.
• Hosted review queues: an Owner or Admin assigns reviewers, who give pass/fail verdicts, scores or A/B preferences that do not name the runs, in the browser. Content a project keeps on its own machines reaches the reviewer's browser from oloproof collect, never through Oloproof. Scores and preferences are stored and enter no statistic yet.
• Production traces received locally: oloproof collect accepts OpenTelemetry (GenAI and OpenInference conventions) and pushes them through the project's egress policy.

Not built yet: production evaluation (hosted trace ingestion, workspace-drawn traffic samples, online scoring and monitoring, turning a trace into a test case; docs/plans/SLICE_S.md S06 to S10), and the rest of the statistical roadmap in Engine §24: conformal risk control, hierarchical dependence, causal intervention analysis, IRT and a method for score and preference labels. Distribution is not built either: there is no PyPI release, and the hosted stack has not yet run on a real host (docs/roadmap/INITIAL_COMPETITIVE_RELEASE.md).

Project Instructions

Coding agents should start with:

• AGENTS.md - shared instructions and source-of-truth map.
• AGENT_WORKFLOWS.md - Claude/Codex division of labor and task prompts.
• DOCUMENTATION_STATUS.md - canonical synchronized doc versions and status.
• docs/engine/ENGINE_SPECIFICATION.md - statistical and decision contract.
• docs/plans/ - the implementation plans, one per slice, and their accepted decisions.

The top-level local repository folder is Oloproof/ with a capital O. Python packages, npm package names, CLI commands and environment variables remain lowercase or uppercase as documented: oloproof, oloproof_core, @oloproof/..., OLOPROOF_*.

License

Oloproof is licensed in two parts. The engine, CLI, SDK and runner (python/), the UI packages (packages/) and everything else a developer installs are under the Apache License 2.0. The hosted control plane and workbench (apps/web/) are under the Elastic License 2.0: you may use, modify and self-host it, and you may not offer it to others as a managed service. NOTICE says which covers what.