Clinical Site Statistics Rollup & Predictive Modeling Pipeline

Lewis Katz School of Medicine at Temple University Center for Biostatistics & Epidemiology 2026-08-05

Project Overview

This repository contains the multi-center pipeline, statistical models, and manuscript deliverables consolidating longitudinal Electronic Health Record (EHR) inpatient data across five clinical health systems: - TEMPLE (Lewis Katz School of Medicine at Temple University) - PSU (Penn State University) - Hopkins (Johns Hopkins University) - PITT (University of Pittsburgh Medical Center) - Geisinger (Geisinger Health System)

The pipeline standardizes encounter types and procedure coding, executes longitudinal lookback window aggregations (1-year and 2-year), maps lab values via LOINC codes, generates Table 1 descriptive statistics rollups, and trains patient-level traditional machine learning models (Logistic Regression, Random Forest, HistGBM, and XGBoost) using Stratified 5-Fold Cross-Validation on $N = 347,541$ qualifying patient encounters.

⚠️ Known Issue: Cohort-Construction Fix Pending Server Re-Validation

A confirmed data-leakage bug was found in Table1_Consolidated_Labs.py’s cohort construction (load_and_process_site): the index encounter used to represent each patient was chosen by uniform random sampling rather than a principled first-qualifying-admission rule, and the 30-day readmission outcome was defined as a patient-level “ever, anywhere in history” flag rather than being scoped to 30 days after the selected index encounter. Together these structurally entangled the outcome with readmission_count_twelve_months (feature > 0 implied outcome = 1 with zero exceptions across all 270,565 non-readmitted patients in the current dataset).

Fixed as of v1.11.9 in select_index_encounters_and_outcome(), with 15 regression tests (test_index_selection_outcome.py) against synthetic data covering the exact failure mode. The fix has not yet been run against the real data (the data lives on the eDERRI server, not this machine) — until that re-run completes, all AUC/OR/feature-importance figures below and throughout the current manuscript deliverables reflect the pre-fix, affected cohort and should be treated as pending re-validation, not final.

Repository Structure & Extract Downloads Directory

To maintain a clean working repository, server extracts and download archives are organized under the extracts/ directory:

kyle_out_zip/
├── extracts/
│   ├── from_server/          # Drop newly downloaded server ZIP archives here (*.zip)
│   └── archive/              # Archived historic server extracts and legacy datasets
├── old_logs/                 # Archived pipeline execution logs and raw run dumps
├── Dr_Rubin_Manuscript_Deliverables_20260729/   # Formatted manuscript tables & figures
├── Dr_Rubin_Manuscript_Deliverables_20260729.zip
├── eDERRI_Traditional_ML_Models_Manuscript.qmd  # Quarto manuscript draft (v1.11.9 with DALEX & Appendices A & B)
├── tasks/add/appendix_include/                   # Biostatistical & metric appendix modules
├── Table1_Consolidated_Labs.py                   # Cohort construction + model training (see Known Issue above)
├── generate_manuscript_tables.py
├── generate_shap_4panel.py
├── generate_dalex_4panel.py
├── generate_shap_blowup_analysis.py
├── build_dr_rubin_deliverables.py
├── benchmark_2feature_all_models.py
├── diagnose_lr_utilization.py
├── pyproject.toml
├── test_table1.py
├── test_pipeline_and_paths.py
├── test_manuscript_data_fixes.py
├── test_table3_and_manuscript_hygiene.py
└── test_index_selection_outcome.py               # Regression tests for the cohort-construction fix

All Python helper scripts (generate_manuscript_tables.py, generate_shap_4panel.py, generate_dalex_4panel.py, build_dr_rubin_deliverables.py, diagnose_lr_utilization.py) automatically search extracts/from_server/ and extracts/ recursively to detect and process the latest extract ZIP files.

Deliverables & Manuscript Table Specifications

1. Formatted Manuscript Tables (Dr_Rubin_Manuscript_Deliverables_20260729/01_Manuscript_Tables/)

• Manuscript_Tables_Combined.xlsx: Holds sheets matching Dr. Rubin’s exact shells:
  • Table 1. Descriptive: Baseline characteristics for all 83 features (Whole cohort, Readmission subset, Non-readmission subset).
  • Table 2. LogReg: Multivariable Logistic Regression risk factors for all 93 features (Dual Unscaled Marginal ORs & Normalized ORs, 95% CIs, P-values).
  • Table 3. Model performance: Model discrimination and calibration measures (AUC, F1, Sensitivity, Specificity, Brier Score across model types; all cross-validation $\text{SD} < 0.01$).
• Manuscript_Tables_By_Site.xlsx: Per-site Table 1s, Table 2s, and master site performance metrics.

2. Publication Figures (Dr_Rubin_Manuscript_Deliverables_20260729/02_Manuscript_Figures/)

• Figure_Top10_DALEX_4Panel.png: High-resolution 300 DPI 4-panel summary plot comparing top 10 feature importances via DALEX variable dropout loss across models.
• Figure_Top10_SHAP_4Panel.png: High-resolution 300 DPI summary plot comparing top 10 features via SHAP Shapley values across models.
• Figure_Prior_Utilization_SHAP_Blowup.png: 4-panel architectural comparison for the top utilization feature (prior_IP_OS_ED_count) showing how LR (linear tail), RF (bimodal butterfly), HistGBM (smooth saturation), and XGBoost (regularized step) process risk differently.
• Figure_SHAP_4Panel.png: Combined 4-panel full SHAP summary figure.
• Figure_HistGBM_SHAP.png: Standalone 300 DPI HistGBM SHAP summary plot.

Empirical Diagnostic Findings

A 2-feature Logistic Regression diagnostic experiment (diagnose_lr_utilization.py / benchmark_2feature_all_models.py) was conducted on the full combined dataset ($N = 347,541$ encounters), benchmarked against the canonical Table 2 full-model results: - Full 93-Feature Model (Logistic Regression): $\text{AUC-ROC} = 0.783$, $\text{Brier Score} = 0.183$ - 2-Feature Utilization Sub-Model (prior_IP_OS_ED_count + 12-month readmissions): $\text{AUC-ROC} = 0.761$, $\text{Brier Score} = 0.136$ - Finding: Historical healthcare utilization alone accounts for 97.2% of the total discriminatory power of Logistic Regression. The same 2-feature benchmark repeated across Random Forest, HistGBM, and XGBoost captures 91.5-92.6% of each architecture’s full-model AUC-ROC (see manuscript Table 3), always benchmarked against the same canonical full-model numbers reported in Table 2 rather than independently recomputed.

Running Pipeline Scripts

1. Generating Manuscript Tables & Figures

python generate_manuscript_tables.py
python generate_shap_4panel.py
python generate_shap_blowup_analysis.py
python build_dr_rubin_deliverables.py
python benchmark_2feature_all_models.py  # Regenerates Table 3 (2-feature sub-model benchmark)

2. Running Unit Tests

pytest -v

License & Authorship

Developed by the Lewis Katz School of Medicine at Temple University Center for Biostatistics & Epidemiology in collaboration with the eDERRI Research Group.