plainproof

Archived on Zenodo: version 0.1.0 is DOI 10.5281/zenodo.23647650; all versions are DOI 10.5281/zenodo.23647649. Also on PyPI: pip install plainproof.

Reproduction package for "How many follow-up questions distinguish genuine understanding from apparent mastery?" (EDM 2027 submission, simulation study). It asks how many adaptive follow-up questions a classification test needs to tell genuine understanding from apparent mastery at a calibrated confidence target, and how that changes as a learner outsources a share d of the follow-ups to an AI. A second, sealed study (E0b) tests whether planted hard probe items separate outsourcing learners from honest ones.

Everything here is CPU-only and needs no paid service. Code: MIT (LICENSE). Docs, figures and tables: CC BY 4.0 (LICENSE-docs.md).

From clone to the figure (about 10 minutes)

The plainproof command runs each step (plainproof --help); the raw script calls below are what it runs. The local path inside the sealed scripts/edm_v2_e0/r_env.sh is part of the frozen record; use the scripts/rscript.sh wrapper, which plainproof figures calls.

git clone <this repository> && cd plainproof
python3.11 -m venv .venv && . .venv/bin/activate && pip install -e '.[test,run]'   # or: pip install -r requirements.txt

# 1. Tests (43): SPRT against catIrt 0.5.1's saved steps, exact stopping time, simulator, signatures, planted probes
python -m pytest -n auto -q tests

# 2. Check that the code here is the sealed code (undoes the documented path substitutions, then compares sha256)
plainproof verify

# 3. Fetch the sealed held-out outputs (from the v0.1.0 release assets) and check their hashes
plainproof fetch heldout

# 4. Rebuild the tables and figures from the held-out summaries (needs R 4.x for the plots; base graphics only)
python scripts/edm_v2_e0/report.py data/edm-v2-e0/heldout/summary.json
scripts/rscript.sh scripts/edm_v2_e0/plot_error_vs_d.R reports/edm-v2-e0/error-vs-d.csv reports/edm-v2-e0/figures/error-vs-d.png aff9a8c49d48821c0e9343cd058c4e0e2f1659ffabcaa719824412f3b14abc90
python scripts/edm_v2_e0b/report.py data/edm-v2-e0b/heldout/summary.json
scripts/rscript.sh scripts/edm_v2_e0b/plot_probe_auc.R reports/edm-v2-e0b/probe-auc-vs-d.csv reports/edm-v2-e0b/figures/probe-auc-vs-d.png 617be427feac24df2e342da00d34e91c78d5de91592cee390ef3516ae9e2ff3f

The figures land in reports/edm-v2-e0/figures/ and reports/edm-v2-e0b/figures/; the committed copies are the same files.

Full rerun (about 4 hours of CPU)

Only if you want to regenerate everything, including the held-out outputs.

# a. BOBCAT (MIT) at the pinned commit, with the preprocessed ASSISTments 2009-2010 file; the data's own terms forbid
#    passing it on, so you fetch it from its source and accept them yourself
scripts/bobcat/fetch_bobcat.sh
# b. Item fit on the development half of ASSISTments learners (the output is checked against the receipt's hash)
python scripts/edm_v2_e0/fit_assist2009.py data/edm-v2-e0/item-fit
# c. Retrain the BOBCAT checkpoint (about 2 hours; author defaults, fold 1, seed 221), then compare it with the original
python scripts/bobcat/retrain.py && python scripts/bobcat/check_checkpoint.py
# d. catIrt fixture, development stages, held-out runs (each refuses to overwrite an existing cell)
scripts/rscript.sh scripts/edm_v2_e0/catirt_sprt_steps.R tests/fixtures/edm_v2_catirt_sprt
python scripts/edm_v2_e0/heldout.py run data/edm-v2-e0/heldout && python scripts/edm_v2_e0/heldout.py summarize data/edm-v2-e0/heldout
python scripts/edm_v2_e0b/run.py run data/edm-v2-e0b/heldout && python scripts/edm_v2_e0b/run.py summarize data/edm-v2-e0b/heldout

Two caveats, stated plainly. The sealed code checks the item fit's sha256, so a different numpy can stop step b; and it asserts that the BOBCAT checkpoint's best epoch is 105, so a retrain that is not bit-identical can stop step d. In either case, use the published held-out outputs (plainproof fetch heldout). The development stages (study.py tune|calibrate|calibrate_sprt_full|pilot) are included for the record; their outputs (synthetic learners only) are in data/edm-v2-e0/study/.

Fingerprints (sha256)

What is in here

• src/plainproof/: item model, SPRT arm, exact stopping time, simulator (uniform and uncertainty-triggered outsourcing), item fit, signatures, planted probes (with the give-up model), matched-coverage scoring.
• scripts/edm_v2_e0/, scripts/edm_v2_e0b/: the runners, report builders and R plots. scripts/published_cat_reference_v1/: the BOBCAT loading code the runners call.
• reports/: both sealed freezes, receipts, figures and tables. docs/: the E0 protocol and the pre-registration log (verbatim extract).
• substitutions.json: the only differences between this code and the sealed originals (package name, file locations, data paths). scripts/verify_frozen.py undoes them and checks every hash. scripts/edm_v2_e0/r_env.sh is kept verbatim as the record of the original R runtime; use scripts/rscript.sh instead.

Recorded departures

All are in docs/PREREG_EDM_V2.md: the SPRT tuning rule was revised after its first development pass (labelled post hoc); SPRT's calibration map was refitted once on full paths before the freeze; the E0 held-out run exceeded its 2 GiB memory cap (no result depends on it; the runner now enforces the cap); E0b was sealed before its pilot table had been reviewed (no result affected).