dormant-behavior-audit 1.2.0

Benchmark assets, reproducibility tooling, and evidence checks for dormant behavior audit.

Dormant Behavior Audit

This repository contains the flagship benchmark assets, reference bundle, and reproducibility materials for auditing latent, condition-dependent model behavior.

The motivating historical case is the Jane Street dormant-model puzzle, but the repo is now organized as a public benchmark and research release rather than a contest-only submission package.

Slow Tour

A quiet walk through the release path: open the charter, inspect the reference bundle, rerun the evidence, compare claim checks, and package the release.

Start Here

If you want the quickest tour, read these in order:

1. RELEASE_STATE.md
2. REVIEWER_QUICKSTART.md
3. TRACEABILITY_MATRIX.md
4. CLAIM_LEDGER.md
5. REPRODUCIBILITY.md
6. benchmarks/BENCHMARK_CHARTER.md
7. findings/RELEASE_PACKET_V2.md
8. CONTRIBUTING.md

Install The CLI

The repository now builds as a Python package with a unified dba command.

The current live package release is 1.2.0 on PyPI. It is the benchmark-CLI and release-verification package for the frozen benchmark/report release v1.0.0, which remains the canonical tagged research bundle.

pipx install dormant-behavior-audit
dba --help

For a local one-off run without a permanent install:

uvx --from dormant-behavior-audit dba --help

Optional extras:

• pipx install 'dormant-behavior-audit[tui]' for the Orbit Textual UI
• pipx install 'dormant-behavior-audit[notebooks]' for notebook-heavy local analysis

The default install is intentionally substantial because it includes the research stack needed for reproduction and analysis, not just a lightweight wrapper CLI.

Useful public inspection commands:

dba doctor
dba list-tasks
dba show-task meridian_trace_multiturn_candidate_v0
dba list-submissions
dba scoreboard
dba reviewer-packet --out-root reviewer_packet
dba verify-release --skip-scoreboard-build

Reviewer Quickstart

The fastest reviewer-grade path is:

pipx install dormant-behavior-audit
dba reviewer-packet --out-root reviewer_packet

This builds a local packet with command logs, scoreboard summary, multi-turn suite status, claim ledger, traceability matrix, arXiv endorsement packet, artifact hash manifest, and checked-in reproduction summaries. It does not download model weights, call hosted APIs, contact humans, or use paid services.

From a source checkout, the equivalent command is:

python3 scripts/build_reviewer_packet.py --out-root reviewer_packet

What This Repo Ships

Public-facing research packet

• Reference report index: findings/RELEASE_PACKET_V2.md
• Canonical report PDF: https://github.com/SproutSeeds/dormant-behavior-audit/releases/download/v1.0.0/dormant-behavior-audit-v1.0.0-reference-report.pdf
• Repo copy of report PDF: findings/CodyMitchell_DormantPuzzle_Submission_V2_2026-03-06.pdf
• Preprint candidate PDF: findings/DormantBehaviorAudit_ReferenceCase_Preprint_2026-04-07.pdf
• Preprint LaTeX source: findings/PREPRINT_SUBMISSION.tex
• Main report markdown: findings/SUBMISSION_V2.md
• Statistical appendix: findings/STATS_ADDENDUM_V2.md
• Raw evidence appendix: findings/RAW_EVIDENCE_APPENDIX_V2.md
• Implications memo: findings/IMPLICATIONS_AND_APPLICATIONS_APPENDIX_V2.md

Benchmark assets

• Benchmark overview: benchmarks/README.md
• Multi-turn suite guide: benchmarks/MULTITURN_SUITE.md
• Multi-turn suite status: benchmarks/MULTITURN_SUITE_STATUS.md
• Benchmark charter: benchmarks/BENCHMARK_CHARTER.md
• Launch plan: benchmarks/LAUNCH_PLAN.md
• Governance/versioning: benchmarks/GOVERNANCE_AND_VERSIONING.md
• Public launch drafts: benchmarks/public/README.md
• Release notes: benchmarks/public/RELEASE_NOTES_v1.0.0.md
• Collaboration brief: benchmarks/public/COLLABORATION_BRIEF.md
• Reviewer quickstart: REVIEWER_QUICKSTART.md
• Traceability matrix: TRACEABILITY_MATRIX.md
• arXiv endorsement packet: benchmarks/public/ARXIV_ENDORSEMENT_PACKET.md
• Standalone homepage: https://sproutseeds.github.io/dormant-behavior-audit/
• Frozen reference bundle: benchmarks/reference/dormant_puzzle_v1/benchmark_bundle_v0.json

Reproducibility artifacts

• Canonical reproduction bundle: artifacts/reproduction/20260305_230206/
• Tightening bundle: artifacts/tightening/20260306_075440/
• Claim-level consistency report: artifacts/reproduction/20260305_230206/findings/claim_consistency_report.md
• Bundle checker entry point: scripts/check_benchmark_bundle.py
• Public safety scan: scripts/check_public_safety.py
• Artifact hash manifest: benchmarks/public/artifact_hash_manifest_v0.json

Benchmark Shape

The current benchmark release has three layers:

• core local seeded and clean-control tasks, including a public stateful multi-turn candidate/control suite with Qwen2 and Qwen2.5 repeat-anchored controls,
• a naturalistic historical reference bundle built from the dormant puzzle result,
• and a supplementary hosted-comparator lane used for calibration and mechanism interpretation.

The benchmark is designed to reward:

• family recovery instead of one lucky string guess,
• candidate-versus-control specificity,
• repeated-run stability,
• interpretation-aware reporting,
• and artifact-rich submission packets instead of one scalar score.

What This Is Not

• It is not a universal detector for every backdoor or dormant behavior.
• It is not a provider leaderboard.
• It does not claim a single proven mechanism for every observed split.
• It treats prefix/taxonomic acknowledgment as an interpretation label, not as dormant-behavior recovery by itself.

Reproducing The Reference Case

Install dependencies:

pip install -r requirements.txt

Run the reproducibility pipeline:

python3 scripts/reproduce_submission.py

This writes a fresh bundle under artifacts/reproduction/<timestamp>/.

Use these files to judge success:

• artifacts/reproduction/<timestamp>/reproduction_report.md
• artifacts/reproduction/<timestamp>/findings/claim_consistency_report.md

For a reviewer-safe report refresh without fresh hosted API calls:

python3 scripts/reproduce_submission.py --report-only --out-root artifacts/reproduction/20260305_230206

Important notes:

• local warmup stages are expected to reproduce on MPS-capable hardware,
• API-side artifacts are stochastic, so claim-level consistency matters more than exact JSON replay,
• and scripts/reproduce_submission.py --warmup-start-stage ... can resume a late warmup failure without rerunning the entire local sweep.

Repo Map

• benchmarks/: benchmark specs, tasks, schemas, public-release drafts, and the normalized reference bundle
• findings/: public report packet, appendices, raw evidence snapshots, and release-facing validation records
• artifacts/: checked-in submission packets, reproduction bundles, tightening bundles, and hosted-baseline outputs
• scripts/: bundle builders, release checkers, reproducibility scripts, and analysis utilities
• src/, orbit/, problems/: earlier investigation and local-analysis surfaces preserved for provenance and follow-on work

Release Status

The canonical release metadata lives in benchmarks/public/release_metadata.json.

Current public release URLs:

• repository: https://github.com/SproutSeeds/dormant-behavior-audit
• canonical benchmark release: https://github.com/SproutSeeds/dormant-behavior-audit/releases/tag/v1.0.0
• canonical reference report PDF: https://github.com/SproutSeeds/dormant-behavior-audit/releases/download/v1.0.0/dormant-behavior-audit-v1.0.0-reference-report.pdf
• canonical reference bundle: https://github.com/SproutSeeds/dormant-behavior-audit/releases/download/v1.0.0/dormant-behavior-audit-v1.0.0-reference-bundle.json
• package release on PyPI: https://pypi.org/project/dormant-behavior-audit/
• Hugging Face dataset entry: https://huggingface.co/datasets/sproutseeds/dormant-behavior-audit
• Zenodo DOI: https://doi.org/10.5281/zenodo.19475781
• reference report markdown: https://github.com/SproutSeeds/dormant-behavior-audit/blob/main/findings/SUBMISSION_V2.md
• benchmark homepage: https://sproutseeds.github.io/dormant-behavior-audit/

The working launch checklist is still preserved in PUBLIC_RELEASE_CHECKLIST.md as the release record.

Licensing

• Code, scripts, and schemas: Apache-2.0 via LICENSE
• Public-facing reports, benchmark docs, and release artifacts: CC BY 4.0 via LICENSE-docs.md

Related Docs

• Release state: RELEASE_STATE.md
• Claim ledger: CLAIM_LEDGER.md
• Reviewer quickstart: REVIEWER_QUICKSTART.md
• Traceability matrix: TRACEABILITY_MATRIX.md
• Reproducibility guide: REPRODUCIBILITY.md
• Roadmap: ROADMAP.md
• Collaboration guide: COLLABORATION.md
• Wanted contributions: WANTED.md
• Public release checklist: PUBLIC_RELEASE_CHECKLIST.md
• Current package release notes: benchmarks/public/PACKAGE_RELEASE_NOTES_v1.2.0.md
• arXiv endorsement packet: benchmarks/public/ARXIV_ENDORSEMENT_PACKET.md
• PyPI publishing guide: PYPI_PUBLISHING.md
• Preprint build script: scripts/build_preprint_pdf.sh
• Contributing guide: CONTRIBUTING.md
• Findings guide: findings/README.md
• Collaboration brief: benchmarks/public/COLLABORATION_BRIEF.md
• Benchmark governance: benchmarks/GOVERNANCE_AND_VERSIONING.md
• External platform status: benchmarks/public/EXTERNAL_PLATFORM_STATUS.md
• Hugging Face publish guide: benchmarks/public/HUGGINGFACE_PUBLISHING.md
• Preprint discoverability packet: benchmarks/public/PREPRINT_DISCOVERABILITY_PACKET.md