Clinical Site Statistics Rollup & Predictive Modeling Pipeline

Lewis Katz School of Medicine at Temple University Center for Biostatistics & Epidemiology 2026-08-12

Project Overview

This repository contains the multi-center pipeline, statistical models, and manuscript deliverables consolidating longitudinal Electronic Health Record (EHR) inpatient data across five clinical health systems: - TEMPLE (Lewis Katz School of Medicine at Temple University) - PSU (Penn State University) - Hopkins (Johns Hopkins University) - PITT (University of Pittsburgh Medical Center) - Geisinger (Geisinger Health System)

The pipeline standardizes encounter types and procedure coding, executes longitudinal lookback window aggregations (1-year and 2-year), maps lab values via LOINC codes, generates Table 1 descriptive statistics rollups, and trains patient-level traditional machine learning models (Logistic Regression, Random Forest, HistGBM, and XGBoost) using Stratified 5-Fold Cross-Validation on $N = 273,755$ qualifying index patient encounters.

✅ Cohort-Construction Fix & Server Re-Validation (Completed August 2026)

A confirmed data-leakage bug in Table1_Consolidated_Labs.py’s cohort construction (load_and_process_site) was identified and resolved. The original outcome definition evaluated a patient-level “ever, anywhere in history” flag rather than scoping readmission to 30 days post-discharge following the selected index encounter. This structurally entangled the outcome with readmission_count_twelve_months (feature > 0 implied outcome = 1 with zero exceptions across all 270,565 non-readmitted patients in the original dataset).

The fix protocol (select_index_encounters_and_outcome()) combines random index encounter sampling among eligible candidates only (verifying a 30-day post-discharge follow-up window) with an outcome strictly scoped to $[t_{\text{discharge}}, t_{\text{discharge}} + 30 \text{ days}]$. Because the lookback window ($[t_{\text{admit}} - 365 \text{ days}, t_{\text{admit}}]$) and outcome window are strictly disjoint, zero data leakage is guaranteed regardless of index selection.

Server Re-Validation Completed: The hardened pipeline was executed on the dassie server (consolidated_out_no_n_px_ccs_20260811_154116.zip + Table1_run_20260811_094355.log). The final 5-health-system cohort consists of $N = 273,755$ unique patients (73,786 excluded for lacking verified follow-up) with a leak-free 30-day readmission rate of 11.37% (31,113 readmitted).

• Headline Full-Model AUCs: XGBoost = 0.706, HistGBM = 0.705, Random Forest = 0.691, Logistic Regression = 0.682.
• 2-Feature Utilization Benchmark (prior_IP_OS_ED_count + readmission_count_twelve_months): AUCs = 0.581–0.583 across all architectures, retaining 82.3%–85.4% of full-model discrimination capacity while remaining 100% leak-free.
• Unit Tests: All 97 unit tests across the test suite pass (95 passed, 2 skipped by design).

✅ Manuscript Content-Integrity Fix (v1.12.1, August 2026)

A v1.12.0 manuscript “re-compilation” pass silently dropped several sections from eDERRI_Traditional_ML_Models_Manuscript.qmd with no trace in CHANGELOG.md/NEWS.md: all 5 SHAP/DALEX explainability figures, the Author Contributions & Acknowledgments section, the Software/package-citations section, and Appendices A & B. A follow-up restore pass brought those sections back structurally, but in doing so reverted large parts of the narrative prose (Abstract, Introduction, Methods, cohort flowchart, Discussion) to a stale, pre-leakage-fix version — citing the old $N = 347{,}541$ cohort, pre-fix AUCs (0.831/0.823/0.821/0.783), and several numbers (a Table 4 odds-ratio example, a Table 2 Brier score) that did not match the actual exported table CSVs, even though every table and figure in the same document was already built from the corrected, leak-free $N = 273{,}755$ cohort.

All narrative numbers were re-verified directly against the exported CSVs (Manuscript_Table1_Descriptive_Stats.csv, Manuscript_Table3_Model_Performance.csv, Table3_2Feature_vs_FullModel_Comparison.csv, Manuscript_Table2_Logistic_Regression.csv) and corrected in place; the CONSORT flowchart’s per-stage encounter/patient dual counts were restored; a Limitations subsection addressing potential survivorship/loss-to-follow-up bias (raised in the 2026-08-12 meeting) was added; and test_manuscript_content_integrity.py was added to guard against this exact failure mode recurring — it asserts required sections/figures are present, that known stale pre-fix values are absent, and that headline AUC/Brier numbers in the prose match the exported CSVs.

Repository Structure & Extract Downloads Directory

To maintain a clean working repository, server extracts and download archives are organized under the extracts/ directory:

kyle_out_zip/
├── extracts/
│   ├── from_server/          # Drop newly downloaded server ZIP archives here (*.zip)
│   └── archive/              # Archived historic server extracts and legacy datasets
├── old_logs/                 # Archived pipeline execution logs and raw run dumps
├── Dr_Rubin_Manuscript_Deliverables_20260729/   # Formatted manuscript tables & figures
├── Dr_Rubin_Manuscript_Deliverables_20260729.zip
├── Dr_Rubin_Data_Leakage_Autopsy_Deliverables_20260811.zip # Technical Autopsy & Distribution Suite
├── Dr_Rubin_Manuscript_Review.zip               # Lightweight manuscript & table package
├── eDERRI_Traditional_ML_Models_Manuscript.qmd  # Quarto manuscript draft
├── Table1_Consolidated_Labs.py                   # Cohort construction + model training
├── generate_manuscript_tables.py
├── generate_shap_4panel.py
├── generate_dalex_4panel.py
├── build_dr_rubin_deliverables.py
├── build_autopsy_deliverables.py
├── create_review_zip.py
├── pyproject.toml
├── test_table1.py
├── test_pipeline_and_paths.py
├── test_manuscript_data_fixes.py
├── test_table3_and_manuscript_hygiene.py
├── test_index_selection_outcome.py
└── test_table1b_missingness_and_ordering.py    # Unit tests for Table 1b and logical row ordering

All Python helper scripts (generate_manuscript_tables.py, generate_shap_4panel.py, generate_dalex_4panel.py, build_dr_rubin_deliverables.py, diagnose_lr_utilization.py) automatically search extracts/from_server/ and extracts/ recursively to detect and process the latest extract ZIP files.

Deliverables & Manuscript Table Specifications

1. Formatted Manuscript Tables (Dr_Rubin_Manuscript_Deliverables_20260729/01_Manuscript_Tables/)

• Manuscript_Tables_Combined.xlsx: Holds sheets matching Dr. Rubin’s exact shells:
  • Table 1. Descriptive: Baseline characteristics for all 83 features (Whole cohort, Readmission subset, Non-readmission subset).
  • Table 2. LogReg: Multivariable Logistic Regression risk factors for all 93 features (Dual Unscaled Marginal ORs & Normalized ORs, 95% CIs, P-values).
  • Table 3. Model performance: Model discrimination and calibration measures (AUC, F1, Sensitivity, Specificity, Brier Score across model types; all cross-validation $\text{SD} < 0.01$).
• Manuscript_Tables_By_Site.xlsx: Per-site Table 1s, Table 2s, and master site performance metrics.

2. Publication Figures (Dr_Rubin_Manuscript_Deliverables_20260729/02_Manuscript_Figures/)

• Figure_Top10_DALEX_4Panel.png: High-resolution 300 DPI 4-panel summary plot comparing top 10 feature importances via DALEX variable dropout loss across models.
• Figure_Top10_SHAP_4Panel.png: High-resolution 300 DPI summary plot comparing top 10 features via SHAP Shapley values across models.
• Figure_Prior_Utilization_SHAP_Blowup.png: 4-panel architectural comparison for the top utilization feature (prior_IP_OS_ED_count) showing how LR (linear tail), RF (bimodal butterfly), HistGBM (smooth saturation), and XGBoost (regularized step) process risk differently.
• Figure_SHAP_4Panel.png: Combined 4-panel full SHAP summary figure.
• Figure_HistGBM_SHAP.png: Standalone 300 DPI HistGBM SHAP summary plot.

Empirical Diagnostic Findings

A 2-feature Logistic Regression diagnostic experiment (diagnose_lr_utilization.py / benchmark_2feature_all_models.py) was conducted on the leak-free combined dataset ($N = 273,755$ encounters), benchmarked against the canonical Table 2 full-model results: - Full 93-Feature Model (Logistic Regression): $\text{AUC-ROC} = 0.682$, $\text{Brier Score} = 0.223$ - 2-Feature Utilization Sub-Model (prior_IP_OS_ED_count + readmission_count_twelve_months): $\text{AUC-ROC} = 0.583$, $\text{Brier Score} = 0.099$ - Finding: Historical healthcare utilization alone accounts for 85.4% of the total discriminatory power of Logistic Regression. The same 2-feature benchmark repeated across Random Forest, HistGBM, and XGBoost captures 82.3%-84.1% of each architecture’s full-model AUC-ROC (see manuscript Table 3), always benchmarked against the same canonical full-model numbers reported in Table 2 rather than independently recomputed.

Running Pipeline Scripts

1. Generating Manuscript Tables & Figures

python generate_manuscript_tables.py
python generate_shap_4panel.py
python generate_shap_blowup_analysis.py
python build_dr_rubin_deliverables.py
python benchmark_2feature_all_models.py  # Regenerates Table 3 (2-feature sub-model benchmark)

2. Running Unit Tests

pytest -v

License & Authorship

Developed by the Lewis Katz School of Medicine at Temple University Center for Biostatistics & Epidemiology in collaboration with the eDERRI Research Group.