audit-closed-ai-scientist

Audit-Closed AI Scientist is a benchmark for evaluating the statistical validity of autonomous research systems.

The repository demonstrates failure modes of naive AI scientist pipelines (p-hacking, optional stopping, hypothesis shopping) and provides a reproducible implementation of an audit-closed protocol based on transparency logs and sequential e-process inference.

This benchmark is intended for developers of AI scientist systems, autonomous research agents, and self-driving laboratories.

Start here

An autonomous research system can repeatedly try hypotheses, stop when a promising result appears, or choose a favorable experimental design after seeing earlier results. These adaptive choices can make ordinary p-values look more convincing than they are. This project is a benchmark and protocol implementation for measuring and constraining those failure modes.

In the protocol, a candidate set and the evidence used to judge it are committed to a tamper-evident transparency log. The update decision is then a deterministic function of that declared log. Sequential e-processes and explicit alpha accounting make the stated benchmark tests valid under continuous monitoring. The simulations compare that approach with deliberately naive discovery policies.

This is not a tool that establishes a scientific theory, physical-laboratory safety, or deployment readiness. A successful run supports only the declared synthetic benchmark configuration and its recorded audit artifacts.

Install and run

The published distribution works on Windows, macOS, and Linux with Python 3.10 or later. You only need Python and an internet connection; no Git checkout, conda environment, or API key is required to run the synthetic benchmark.

python -m pip install --upgrade audit-closed-ai-scientist

Run a small integrated benchmark first. The output is a JSON artifact, so it can be archived or compared across runs.

audit-closed-ai-scientist benchmark --runs 20 --output results/benchmark.json

Run every simulation with the lightweight reproducible profile:

audit-closed-ai-scientist run --profile quick --output results/experiment_results.json

benchmark runs the integrated false-discovery, replication, sequential-evidence, and adversarial checks. run runs the complete simulation suite and writes plots to figures/. Use --profile standard for the larger reference configuration. The same command is also available as python -m audit_closed_ai_scientist.

Choose the right command

Every command writes a JSON result file. Keep it together with the command, package version, and any external-agent adapter revision used to produce it.

Read the result correctly

The most useful fields in an integrated benchmark result are:

Compare results only when the seed, profile/configuration, package version, and external-agent adapter are recorded. A low benchmark false-discovery rate is not a universal guarantee and does not validate an unmodeled laboratory, dataset, or agent behavior.

Use it with your own AI scientist

Provide a callable (or an object with evaluate_trial) that returns at least accepted: bool and winner: str | None. The harness counts malformed responses as interface failures rather than silently repairing them.

from benchmarks import benchmark

report = benchmark.evaluate(my_ai_scientist)
print(report["null_world"]["false_discovery_rate"])

The complete adapter contract and a working integration guide are in docs/integration_with_ai_scientist_systems.md.

Agent Skill

This repository includes an Agent Skills-compatible workflow at .agents/skills/audit-closed-ai-scientist/SKILL.md. Compatible agents can discover it from this repository, or copy it to a supported user skills directory such as ~/.agents/skills/audit-closed-ai-scientist/. The canonical implementation remains this repository.

It is designed for:

• AI scientist developers
• autonomous research agents
• automated scientific discovery pipelines
• self-driving laboratories
• agentic AI research frameworks

The benchmark studies where naive autonomous discovery fails, and how an audit-closed protocol mitigates those failures using:

• transparency logs
• deterministic replay
• sequential e-process inference
• explicit alpha accounting

Why this benchmark exists

Naive autonomous research loops are statistically unsafe under adaptive search. In this benchmark, naive pipelines are stress-tested under:

• optional stopping
• p-hacking / many-hypothesis search
• candidate and design shopping
• adversarial experiment submissions

The benchmark compares:

• naive discovery policies (publish when significance appears)
• audit-closed policies (Accept_t = f(Log_0:t))

Reference paper

• Takahashi, K. (2026). Audit-Closed AI Scientist Protocol. Zenodo. https://doi.org/10.5281/zenodo.18728589
• Repository manuscript: paper/audit_closed_ai_scientist_protocol.tex

What is implemented

Core simulations:

1. simulations/p_hacking_simulation.py
2. simulations/candidate_shopping.py
3. simulations/optional_stopping.py
4. simulations/power_curve.py
5. simulations/adversarial_agents.py

Protocol-extension benchmark modules:

1. audit_protocol/physical_sentinels.py + simulations/sentinel_hierarchy.py
2. audit_protocol/drift_localization.py + simulations/drift_localization_simulation.py
3. audit_protocol/certificate_schema.py + simulations/certificate_schema_validation.py

Integrated benchmark:

• benchmarks/discovery_validity_benchmark.py
  • false discovery rate
  • replication success
  • sequential evidence stability
  • adversarial robustness

Reproduce from source

For contributors or users who need the exact repository state:

git clone https://github.com/kadubon/audit-closed-ai-scientist
cd audit-closed-ai-scientist
python -m pip install .
audit-closed-ai-scientist run --profile standard

Conda

conda env create -f environment.yml
conda activate audit-closed-ai-scientist
python run_all_experiments.py --profile standard

Make targets

make reproduce
make benchmark
make figures
make test

Output artifacts:

• results/experiment_results.json
• figures/*.png
• result_summary.md

Figure regeneration from raw JSON:

python regenerate_figures.py --input results/experiment_results.json --output-dir figures

External agent API

Use the benchmark as an evaluation harness for your own AI scientist:

from benchmarks import benchmark

# my_ai_scientist: callable or object with evaluate_trial(...)
report = benchmark.evaluate(my_ai_scientist)
print(report["null_world"]["false_discovery_rate"])
print(report["signal_world"]["replication_probability"])

Adapter contract (minimum return fields):

• accepted: bool
• winner: str | None

See docs/integration_with_ai_scientist_systems.md.

Security and audit integrity

Implemented safeguards:

• append-only hash-chained transparency log
• Merkle checkpoint verification
• fail-closed transcript checks
• candidate-set commitment before evaluation
• deterministic replay validation
• tamper tests for payload/hash/commitment/alpha rewrites

Security tests:

python -m unittest discover -s tests -p "test_*.py"

Scientific integrity statement

This repository is a benchmark-focused implementation. It supports benchmark-level claims about statistical governance and replayability. It does not claim full deployment implementation of all physical protocol components in the paper.

Detailed audits:

• docs/paper_alignment_audit.md
• docs/security_audit.md
• docs/reproducibility.md
• docs/repository_audit_report.md
• result_summary.md

Citation

Software (current release):

• Takahashi, K. (2026). audit-closed-ai-scientist (v0.2.0). GitHub. https://github.com/kadubon/audit-closed-ai-scientist/releases/tag/v0.2.0

Archived software record:

• The Zenodo DOI https://doi.org/10.5281/zenodo.18870261 identifies the archived v0.1.0 software record. It is not presented as a DOI for v0.2.0.

Protocol paper:

• Takahashi, K. (2026). Audit-Closed AI Scientist Protocol. Zenodo. https://doi.org/10.5281/zenodo.18728589

Machine-readable metadata is in CITATION.cff.

Maintainer release procedure

PyPI Trusted Publishing is configured for this exact identity: repository kadubon/audit-closed-ai-scientist, workflow file .github/workflows/workflow.yml, and GitHub environment pypi. No PyPI API token is stored in the repository.

1. Update the version consistently in pyproject.toml, audit_closed_ai_scientist/__init__.py, and CITATION.cff.
2. Commit the release, tag it as v<version> (for example, v0.2.0), and create a GitHub Release from that tag.
3. The workflow tests the installed package on Windows, macOS, and Linux; only after those tests and the version/tag check pass does its isolated pypi job upload the built wheel and source distribution.

The publish job uses OpenID Connect with the required id-token: write permission, which matches PyPI's Trusted Publisher flow. Configure GitHub Environment protection for pypi if a human approval gate is required before publication.