SciMLRigorBench v4.0.0

SciMLRigorBench is the reproducible benchmarking and assurance layer for SciMLRigor. Version 4.0.0 evaluates the released SciMLRigor v1.5.2 engine across controlled methodological failure injections, a preregistered bespoke procedural reference, source-backed external-validation context, and a version-pinned capability landscape.

Purpose

The benchmark asks whether a single standardized methodological-audit interface can recover heterogeneous validation failures at performance comparable to a hand-assembled set of specialized diagnostics. It does not claim that SciMLRigor is universally more accurate than all specialized tools.

What is included

• Frozen comparative protocol: METHODS_PROTOCOL.md
• Exact benchmark runner: src/run_comparative_benchmark.py
• 60 paired controlled scenarios across R01-R10
• Classification and regression benchmark tasks
• SciMLRigor v1.5.2 vs scikit-learn v1.8.0 bespoke procedural reference
• Exact McNemar and paired-bootstrap statistics
• Version-pinned capability mapping for Deepchecks, Evidently, Great Expectations, mlinspect, and mlscorecheck
• Explicit third-party execution status
• Source-backed external-validation context from SciMLRigor v1.5.2
• Publication-oriented reports and figures

Headline comparative results

Paired comparison: exact two-sided McNemar p = 1.0. The SciMLRigor-minus-reference balanced-accuracy difference was -0.0323, with a 10,000-replicate paired bootstrap 95% CI of [-0.1375, 0.0638].

The evidence therefore supports comparable aggregate diagnostic performance with different family-specific strengths and sensitivity/specificity trade-offs, rather than universal superiority.

Failure families

The benchmark covers ten methodological-risk families:

• R01: target-derived leakage
• R02: selection/tuning scope
• R03: repeated/group dependence
• R04: temporal transport sensitivity
• R05: duplicate contamination
• R06: signal/permutation integrity
• R07: identifier shortcut dependence
• R08: proxy dependence
• R09: domain transportability
• R10: reliability/calibration degradation

Evidence grading

External-validation context follows the SciMLRigor v1.5.2 evidence tiers:

• A: authoritative/original source executed end-to-end
• B: verified mirror executed and independently cross-checked
• C: protocol/fixture/parser validation only
• P: protocol prepared but source execution pending/unavailable

Tier A is the primary source-backed external layer. Tier C/P support implementation or readiness claims only.

Reproducing the benchmark

The executed environment recorded in results/environment.json used Python 3.13.5, SciMLRigor 1.5.2, scikit-learn 1.8.0, NumPy 2.3.5, and pandas 2.2.3.

A validated SciMLRigor v1.5.2 wheel is included under artifacts/ for exact reproduction.

Example setup:

python -m venv .venv
# Windows: .venv\\Scripts\\activate
# Linux/macOS: source .venv/bin/activate
pip install artifacts/scimlrigor-1.5.2-py3-none-any.whl
pip install -r requirements.txt
python src/run_all.py

The frozen thresholds and benchmark rules are documented in METHODS_PROTOCOL.md. They were fixed before comparative outcomes were inspected.

Repository structure

SciMLRigorBench/
├── README.md
├── LICENSE
├── CITATION.cff
├── .zenodo.json
├── .gitignore
├── VERSION
├── METHODS_PROTOCOL.md
├── RELEASE_MANIFEST.json
├── SHA256SUMS.txt
├── requirements.txt
├── src/
├── results/
├── metadata/
├── figures/
├── reports/
├── context_v3/
└── artifacts/

Relationship to SciMLRigor

SciMLRigorBench calls the released SciMLRigor audit engine rather than duplicating R01-R10 logic. The benchmarked core software release is:

SciMLRigor v1.5.2 — Zenodo version DOI: 10.5281/zenodo.23001654.

Claim boundaries

The benchmark does not support claims that SciMLRigor is universally more accurate than all competing tools, that unexecuted third-party packages failed the benchmark, or that 60 controlled scenarios establish universal real-world sensitivity and specificity. Non-injected control alerts are reported as benchmark-ground-truth disagreements because natural datasets may contain genuine methodological sensitivity not introduced by the benchmark.

Citation

Use CITATION.cff for citation metadata. A Zenodo DOI will be added for SciMLRigorBench v4.0.0 after archival of the GitHub release.

License

MIT License.