plainproof

Archived on Zenodo: version 0.1.0 is DOI 10.5281/zenodo.23647650; all versions are DOI 10.5281/zenodo.23647649. Also on PyPI: pip install plainproof.

Reproduction package for "How many follow-up questions distinguish genuine understanding from apparent mastery?" (EDM 2027 submission, simulation study). It asks how many adaptive follow-up questions a classification test needs to tell genuine understanding from apparent mastery at a calibrated confidence target, and how that changes as a learner outsources a share d of the follow-ups to an AI. A second, sealed study (E0b) tests whether planted hard probe items separate outsourcing learners from honest ones.

Everything here is CPU-only and needs no paid service. Code: MIT (LICENSE). Docs, figures and tables: CC BY 4.0 (LICENSE-docs.md).

From install to the figure (about 10 minutes)

pip install 'plainproof[test,run]'
plainproof init ~/plainproof        # writes the working directory: sealed scripts, freezes, receipts, docs, tests
cd ~/plainproof
plainproof test                     # 43 tests: SPRT against catIrt 0.5.1's saved steps, exact stopping time, simulator, signatures, probes
plainproof verify                   # every sealed file against both freezes (undoes the documented substitutions, then sha256)
plainproof fetch heldout            # the sealed held-out outputs, hash-checked
plainproof figures                  # tables and figures from the held-out summaries (needs R 4.x on PATH; base graphics only)

The figures land in reports/edm-v2-e0/figures/ and reports/edm-v2-e0b/figures/, byte-identical to the published copies. plainproof init writes the files in the exact layout the sealed scripts expect; nothing sealed changes. Every other command runs inside that directory, or takes --root DIR. The local path inside the sealed scripts/edm_v2_e0/r_env.sh is part of the frozen record; the scripts/rscript.sh wrapper, which plainproof figures calls, is the one to use.

Full rerun (about 4 hours of CPU)

Only if you want to regenerate everything, including the held-out outputs.

plainproof fetch bobcat     # BOBCAT (MIT) at its pinned commit, with the preprocessed ASSISTments 2009-2010 file
plainproof fit              # 2PL item fit on the development half of learners; checked against the receipt's hash
plainproof retrain          # BOBCAT checkpoint, about 2 hours (author defaults, fold 1, seed 221), compared with the original run
plainproof run e0           # E0 held-out run and summary; refuses to overwrite an existing cell
plainproof run e0b          # E0b held-out run and summary

The ASSISTments terms of use forbid passing the data on, so this package never contains it: fetch bobcat downloads it from its source, and you accept those terms yourself. Two caveats, stated plainly. The sealed code checks the item fit's sha256, so a different numpy can stop fit; and it asserts that the BOBCAT checkpoint's best epoch is 105, so a retrain that is not bit-identical stops run e0. In either case, use the published held-out outputs (plainproof fetch heldout). The development stages (scripts/edm_v2_e0/study.py tune|calibrate|calibrate_sprt_full|pilot) are included for the record; their outputs (synthetic learners only) are in data/edm-v2-e0/study/.

Fingerprints (sha256)

What is in here

• src/plainproof/: item model, SPRT arm, exact stopping time, simulator (uniform and uncertainty-triggered outsourcing), item fit, signatures, planted probes (with the give-up model), matched-coverage scoring.
• src/plainproof/cli.py: the plainproof command. setup.py packs the working directory into the wheel for plainproof init.
• scripts/edm_v2_e0/, scripts/edm_v2_e0b/: the runners, report builders and R plots. scripts/published_cat_reference_v1/: the BOBCAT loading code the runners call.
• reports/: both sealed freezes, receipts, figures and tables. docs/: the E0 protocol and the pre-registration log (verbatim extract).
• substitutions.json: the only differences between this code and the sealed originals (package name, file locations, data paths). scripts/verify_frozen.py undoes them and checks every hash. scripts/edm_v2_e0/r_env.sh is kept verbatim as the record of the original R runtime; use scripts/rscript.sh instead.

Recorded departures

All are in docs/PREREG_EDM_V2.md: the SPRT tuning rule was revised after its first development pass (labelled post hoc); SPRT's calibration map was refitted once on full paths before the freeze; the E0 held-out run exceeded its 2 GiB memory cap (no result depends on it; the runner now enforces the cap); E0b was sealed before its pilot table had been reviewed (no result affected).

Appendix: the raw script calls

What each plainproof command runs, from the working directory (REPRO_DATA defaults to ./data):

python -m pytest -n auto -q tests                                       # plainproof test
python scripts/verify_frozen.py                                         # plainproof verify
ZENODO_URL=https://github.com/cloudronin/plainproof/releases/download/v0.1.0 bash scripts/fetch_heldout.sh   # plainproof fetch heldout
python scripts/edm_v2_e0/report.py data/edm-v2-e0/heldout/summary.json  # plainproof figures (E0 tables)
bash scripts/rscript.sh scripts/edm_v2_e0/plot_error_vs_d.R reports/edm-v2-e0/error-vs-d.csv reports/edm-v2-e0/figures/error-vs-d.png <E0 summary sha256>
python scripts/edm_v2_e0b/report.py data/edm-v2-e0b/heldout/summary.json
bash scripts/rscript.sh scripts/edm_v2_e0b/plot_probe_auc.R reports/edm-v2-e0b/probe-auc-vs-d.csv reports/edm-v2-e0b/figures/probe-auc-vs-d.png <E0b summary sha256>
bash scripts/bobcat/fetch_bobcat.sh                                     # plainproof fetch bobcat
python scripts/edm_v2_e0/fit_assist2009.py data/edm-v2-e0/item-fit      # plainproof fit
python scripts/bobcat/retrain.py && python scripts/bobcat/check_checkpoint.py   # plainproof retrain
python scripts/edm_v2_e0/heldout.py run data/edm-v2-e0/heldout && python scripts/edm_v2_e0/heldout.py summarize data/edm-v2-e0/heldout   # plainproof run e0
python scripts/edm_v2_e0b/run.py run data/edm-v2-e0b/heldout && python scripts/edm_v2_e0b/run.py summarize data/edm-v2-e0b/heldout       # plainproof run e0b
scripts/rscript.sh scripts/edm_v2_e0/catirt_sprt_steps.R tests/fixtures/edm_v2_catirt_sprt   # regenerate the catIrt fixture (R with catIrt 0.5.1)