Aureproof

An open-source integrity layer for AI-agent evaluations.

Every claim, backed by evidence.

Status: pre-release (0.0.x). The integrity core, the Inspect runner, signing, the comparison gate and one live, signed evidence chain are built; no benchmark result or performance claim is asserted yet. What remains before v0.1.0, and who it waits on, is in the release acceptance matrix; the design is the implementation plan.

What it does

On top of the runners teams already use (Inspect AI first; Harbor next), Aureproof certifies three things about an evaluation:

1. The instrument — a per-task health record proves the grader works (its reference-positive, repaired-negative, no-op, and anti-cheat controls pass) before a task is allowed to count.
2. The record — a signed, re-scorable evidence bundle: content-addressed artifacts, a hash-chained journal, an in-toto statement, and a Sigstore signature. Anyone can verify and re-score it offline.
3. The decision — a pre-registered pass / fail / inconclusive regression gate with cluster-aware statistics, so a result that the data cannot support is reported as inconclusive rather than as a win.

Its flagship benchmark measures whether a security agent's findings are real — false alarms included — the problem the 2026 "AI slop" wave made concrete.

Scope and limits

• In scope (v0.1): security-defect discovery on self-authored synthetic Python fixtures; structured findings graded by a deterministic label matcher in the trusted controller; offline, signed bundles; an A/B decision gate.
• Not yet: executing agent-submitted proof-of-concept code (v0.2), real-CVE tasks, patch grading, chat/RAG/voice, multi-agent systems, training.
• Synthetic fixtures measure controlled skills, not real-world capability.

How it compares

Aureproof is designed to complement these projects, not replace them, and to contribute fixes upstream. To our knowledge, the combination below is not offered as one tool today; see docs/plan-review.md for the cited landscape.

Try it offline

Validate, re-score and report a sealed example bundle. No credentials, no network:

uv sync --frozen
uv run aureproof validate examples/bundles/tutorial-v0.1
uv run aureproof score examples/bundles/tutorial-v0.1 --offline
uv run aureproof report examples/bundles/tutorial-v0.1 --out report.html

Walkthrough: docs/guides/tutorial.md.

Guides: authoring a task (task-health controls, licensing), reading results (verdicts, denominators, exclusions, refusals) and the comparison gate (pass, fail, inconclusive).

Quickstart (development)

Requires uv and Python 3.12+.

uv sync                 # create the environment from the locked dependencies
uv run aureproof --version
uv run pytest           # tests + coverage
uv run ruff check .     # lint
uv run mypy             # types
uv run lint-imports     # architecture boundaries (plan v2 §3.2)

Install the pre-commit hooks once: uvx pre-commit install.

Documentation

• docs/aureproof-plan-v2.md — the full plan (the spec this repo implements)
• docs/plan-review.md — the review and verified landscape
• docs/scope.md — personas, goals, non-goals
• docs/adr/ — architecture decision records
• ROADMAP.md — milestones M0–M5
• docs/project-status.md — audited progress, effort assumptions, and execution order

Contributing

See CONTRIBUTING.md. Commits are signed off under the Developer Certificate of Origin. By participating you agree to the Code of Conduct. Please read RESPONSIBLE_USE.md before adding fixtures or tasks.

Security

Report vulnerabilities privately per SECURITY.md.

License

Code is Apache-2.0. Self-authored fixtures are CC-BY-4.0. The Code of Conduct is adapted from the Contributor Covenant 3.0 (CC-BY-SA-4.0). The repository is REUSE-compliant; per-file terms are in each file's SPDX header or in REUSE.toml.