cohort-diff

Keywords: EHR · MIMIC-IV · cohort definitions · phenotyping · acute kidney injury · diabetes · 30-day readmission · reproducibility · odds ratio · Jaccard index · bootstrap confidence intervals

Skills: Python · SQL · DuckDB · pandas · NumPy · patient-level bootstrap · uv packaging · PyPI (Trusted Publishing) · GitHub Actions CI · pytest · Hypothesis (property-based testing) · test oracles · mypy (strict) · ruff · AI-assisted engineering (Claude Code with CLAUDE.md, tests first, AI log)

How different are "the same" EHR cohorts? cohort-diff runs several definitions of one condition side by side on MIMIC-IV-shaped data and reports how much the cohorts, and the results built on them, disagree.

Description

A cohort definition is a modelling decision, but papers usually report it in a sentence. Two studies that both say "AKI patients in MIMIC-IV" may share only a fraction of their admissions: one counts discharge diagnosis codes, another a creatinine rise within 48 hours, another a rise over 7 days. A finding such as "AKI is associated with higher in-hospital mortality" can change size with the definition. cohort-diff stores each definition as a versioned SQL file with its source, runs them on DuckDB directly over MIMIC's .csv.gz files, and reports overlap, who is in each cohort, and how a typical downstream association moves.

Objective

1. Build a tested, pip-installable library that runs cohort definitions as reviewable SQL and checks each one against an independent implementation.
2. Measure how much definitions of the same condition (AKI, diabetes, 30-day readmission) disagree, in membership and in a downstream result (the odds ratio between membership and in-hospital mortality), with confidence intervals.
3. (Extension, branch cohort-diff-llm.) Measure how often a large language model, given a paper's methods text, drafts a cohort definition that passes the same oracle tests as the hand-written one.

Hypotheses

• H1. Definitions of the same condition select substantially different admissions (pairwise Jaccard well below 1).
• H2. Code-based and lab-based definitions differ systematically: codes under-capture, and lab rules with fixed thresholds over-capture when measurements are frequent and noisy.
• H3. The mortality odds ratio reported for "the same" condition changes materially with the definition: for some pairs, the 95% interval of the ratio of their odds ratios excludes 1.
• H4. A definition limits which questions its cohort can answer (for example, readmission exclusions that remove every death).
• H5. (Extension.) LLM-drafted definitions often run without error but fail the oracle tests at clinically important details (thresholds, time windows, specimen type, boundary conditions), so tests, not inspection, decide whether a draft can be used.

Samples

The demo shares the full MIMIC-IV schema, so the same definitions run on full MIMIC-IV (credentialed) unchanged.

Methods

• Definitions as data. Each definition is one SQL file in src/cohort_diff/definitions/ with a header (name, condition, description, reference) and returns one column, hadm_id. Adding a definition does not touch Python.
• Built-in definitions.

• Engine. DuckDB, in process, reads the MIMIC tables in place (demo folder layout or flat CSVs) as views. Lab items are found by label and fluid, not by itemid.
• Correctness. Every rule-based definition is compared with an independent, deliberately slow pandas implementation (a test oracle written with loops rather than joins), on the synthetic data and on hundreds of random creatinine series generated by Hypothesis. Hand-picked regression tests pin threshold edge cases that both implementations could get wrong together.
• Comparison. For every pair of definitions: admissions in both, in only one, and the Jaccard index |A ∩ B| / |A ∪ B|. For each cohort: size, prevalence, mean age, % female, in-hospital mortality.
• Downstream result. Unadjusted odds ratio between membership and in-hospital mortality, with a Woolf 95% CI (log OR ± 1.96·√(1/a + 1/b + 1/c + 1/d)); 0.5 is added to every cell when any cell is 0 (Haldane–Anscombe).
• Uncertainty. Patient-level bootstrap (1,000 resamples; a patient drawn k times contributes all their admissions k times) gives 95% percentile intervals for each pair's Jaccard index and for the ratio of their mortality odds ratios (OR of a / OR of b). An interval for the ratio that excludes 1 means the two definitions disagree about the association, beyond sampling noise. A test checks that patients, not admissions, are resampled.
• Scale check. Wall time of compare for each condition, on the demo and (once credentialed) on full MIMIC-IV, on one laptop.
• Extension: LLM-drafted definitions (planned, branch cohort-diff-llm). An LLM receives a paper's methods text and the table and column names only (never patient data) and returns a definition through JSON-schema structured output. Each draft is graded by the same oracle and property tests, and by exact agreement with the hand-written definition on synthetic data. Pass rates for 3+ models on a 20-item labeled set, with 95% intervals; every failure is categorized.
• Engineering. Strict typing, lint and tests on Python 3.11–3.13 in CI; the wheel is built in CI; releases go to PyPI by Trusted Publishing from version tags.

Results

Synthetic data (2,000 patients, 2,738 admissions, seed 0)

Pairwise overlap and disagreement in the mortality association (patient-level bootstrap, 1,000 resamples, 95% intervals):

Bold: the interval excludes 1, so the two definitions disagree about the mortality association beyond sampling noise.

• Code vs lab definitions barely overlap (Jaccard 0.09–0.16), while two lab rules (KDIGO 48 h vs 7 d) or two code-like rules (ICD vs medications) overlap 81–88%.
• High overlap does not mean the same answer. The two KDIGO definitions share 88% of admissions, yet their mortality ORs differ about threefold (ratio 2.87 [1.87, 5.15]).
• Different-looking ORs can be compatible. aki_icd (3.06) and aki_kdigo_48h (4.23) look different, but the interval for their ratio includes 1.
• KDIGO-48h flags 78% of synthetic admissions: the generator draws creatinine about every 12 h with noise (SD 0.3 mg/dL), and a 0.3 mg/dL rule fires on noise when labs are frequent. Whether real data shows the same is what the demo and full MIMIC-IV can answer.
• Readmission cohorts contain no deaths by construction (an admission followed by another one cannot end in death), so their mortality ORs come only from the 0.5 zero-cell correction. The definition limits which questions the cohort can answer.

MIMIC-IV demo (real data: 100 patients, 275 admissions)

• Real definitions also select different patients. The AKI definitions have similar prevalence (19–25%) but share only 31–56% of their combined admissions. Diabetes by HbA1c ≥ 6.5% (6% of admissions) overlaps the code and medication definitions at Jaccard 0.11–0.13.
• An ICD-10-only diabetes definition loses almost half the cohort to the coding switch. Cheng et al. 2025 define type 2 diabetes as ICD-10 E11. On the demo this finds 60 of the 111 admissions with a diabetes code (Jaccard 0.54 with the Charlson definition); of the 51 it misses, 50 are coded only in ICD-9 (before October 2015) and 1 has a non-E11 ICD-10 code. The eMERGE definition (Zhang & Cao 2023) uses ICD-9 codes only, yet it is not blind to the ICD-10 years (next point).
• A multi-source phenotype is not its code list, and in-hospital insulin decides who qualifies. eMERGE as adapted by Zhang & Cao 2023 classifies patients, not admissions, and returns every admission of a case: 51 admissions (18.5%) from only 9 patients. It shares 46 with dm_icd (Jaccard 0.40) and, despite its ICD-9-only codes, 31 with the ICD-10 Cheng cohort, because path 5 (type 2 drug + abnormal glucose or HbA1c) needs no code and a case patient's ICD-10-era admissions come with them. Its 5 admissions outside dm_icd all belong to patients with a diabetes code on another admission. No case qualified by path 1 (type 2 code + type 2 drug, no insulin). Of the 65 dm_icd admissions it misses, 53 (19 patients) are lost to insulin, which the algorithm counts as a type 1 drug: 36 because insulin came no later than the first type 2 drug, and 17 because there was insulin, no type 2 drug and a type 2 code on only one admission. The other 12 (7 patients) have no ICD-9 type 2 code and no type 2 drug. Insulin is the usual treatment for high glucose in hospital, so the rule's "type 1 medication" step removes most coded inpatients. Measured with notes/check_emerge.py (counts only).
• eMERGE's cases include no in-hospital deaths. Its ratio intervals against Cheng (10.09 [1.07, 34.24]) and the medication definition (9.12 [1.29, 37.06]) exclude 1, but only because of the zero-cell correction on 9 patients; they are not evidence of a different association.
• The community definition matches the textbook one. mimic-code's Charlson diabetes flags select exactly the same 111 admissions as the built-in dm_icd (Jaccard 1.0).
• The medication phenotype captures the most (49% of admissions vs 40% by codes), consistent with insulin given for stress hyperglycemia; every HbA1c-defined admission is also in the medication cohort.
• KDIGO-48h flags 25% of demo admissions, not the 78% seen on synthetic data, so the synthetic over-capture came from the generator's frequent, noisy creatinine, not from the rule itself.
• Anchoring to the ICU changes the cohort and the answer. The published mimic-code KDIGO definition flags 44 admissions (16%) against 69 (25%) for the same creatinine rule over the whole hospital stay; 43 of its 44 are in the hospital-wide cohort. Its members are sicker (mortality 25% vs 16%), and the ratio of mortality ORs, 0.51 [0.36, 0.71], is the one demo interval that excludes 1. The cohorts are nested, which keeps the paired interval narrow; with 100 patients this shows how much the anchoring choice matters, not a real-world effect size.
• A prediction-study definition selects a different population. Kamel Rahimi et al.'s Baseline 3 keeps only new AKI on ICU day 2 or 3 in first ICU stays with creatinine on days 1–3, and excludes AKI on day 1. On the demo it flags 15 admissions (5.5%), all inside the hospital-wide KDIGO cohort but overlapping the code-based cohort at Jaccard 0.06. Its members have 6.7% mortality (1 death) against 16–25% for the other AKI definitions, and every ratio of mortality ORs against it excludes 1. With one death the intervals are very wide (up to about 100), and the comparison group includes the sicker patients the definition excludes (see Limitations).
• Even one paper's own baselines disagree on who has AKI. Kamel Rahimi et al.'s three baselines flag 18, 16 and 15 demo admissions, the same order as the incidences the paper reports on full MIMIC-IV (21.0%, 18.3%, 16.7%; the paper's percentages are of its eligible cohort, the demo's of all admissions). They share only 62–82% of their combined members (Jaccard 0.62–0.82); their mortality associations cannot be told apart with 1–2 deaths each.
• For readmission, the window matters more than the source. The community pipeline's 30-day rule (Gupta et al.) and the built-in readmit30_any select almost the same admissions (52 and 53, Jaccard 0.98; the one difference is an admission whose only 30-day "readmission" either overlaps it or ends in death). The two 90-day papers flag 32% (Prempeh) and 27% (Shen) of demo admissions against 19% at 30 days, and overlap the 30-day cohorts at Jaccard 0.51–0.60. Every 30-day readmission is also a Prempeh readmission. Shen's exclusions (stays under 2 days, a next admission under 1 day after discharge, only the next admission counted) remove 15 of Prempeh's 88 and 8 of the 53 30-day readmissions.
• Readmission ratio intervals that exclude 1 are an artifact. Several readmission pairs give ratio intervals that exclude 1 (for example 30-day vs Prempeh, 1.99 [1.65, 2.46]). No readmission cohort contains a death, so each OR comes only from the 0.5 zero-cell correction, which shrinks as the cohort grows; the ratio measures cohort size, not a different association. They are reported as not meaningful.
• Otherwise, no disagreement in the mortality association can be shown at this size. The other AKI ORs range from 9.4 to 14.3 and the diabetes ORs other than eMERGE from 0.45 to 1.32, and their ratio intervals include 1, some 20-fold wide.

With 100 patients, cohorts are small and intervals wide; these numbers show the tool runs on real data, not how much definitions disagree in practice.

Scale check

Full MIMIC-IV (about 546,000 admissions) will be timed once credentialed.

Extension: LLM-drafted definitions (planned)

Not started. Planned on branch cohort-diff-llm after v0.1.0: pass rates of LLM-drafted definitions on the oracle tests for 3+ models on a 20-item labeled set, with 95% intervals and every failure categorized.

Correctness

88 tests pass with 99% line coverage on Python 3.11–3.13 (CI). Deliberate breaks show what each kind of test catches:

Discussion

On synthetic data, where the structure is known, definitions of the same condition overlap little when they use different sources (codes vs labs, Jaccard 0.09–0.16; H1), and four of six pairs disagree about the mortality association beyond sampling noise (ratio intervals excluding 1; H3). High overlap does not guarantee agreement: the two KDIGO rules share 88% of admissions but their ORs differ about threefold.

The open demo shows the same pattern in membership on real data: AKI definitions with almost identical prevalence share at most 56% of their admissions, and diabetes by HbA1c overlaps the other definitions at about 0.12 (H1). The medication phenotype captures the most admissions, as expected (H2). The published diabetes definitions differ mostly in two choices: which side of the 2015 ICD-9/ICD-10 switch their codes cover (Cheng's ICD-10-only rule misses 50 ICD-9-only admissions), and how they treat insulin (eMERGE counts it as a type 1 drug, which removes most coded inpatients). For AKI, H2 was not supported on the demo: the lab rule flagged the same share as the codes (25%), and the synthetic data's 78% came from the generator. Whether the mortality association itself changes with the definition (H3) can mostly not be tested with 100 patients: every ratio interval among the textbook definitions includes 1. The exception is a published one: the mimic-code KDIGO definition, anchored to ICU stays, gives a mortality OR about half that of the hospital-wide rule (ratio 0.51 [0.36, 0.71]). Readmission definitions exclude deaths by construction, so they cannot answer a mortality question at all (H4). Among the published readmission definitions, the window (30 vs 90 days) changes the cohort far more than the choice of source: a community pipeline's 30-day rule reproduces the textbook one almost exactly.

Strengths. Definitions are versioned, cited SQL rather than prose; each is checked against an independent implementation and by property tests; known-structure validation on synthetic data; installable with pip; reproducible by anyone from open data, a lock file and CI on Python 3.11–3.13.

Limitations

• No real-world estimate of disagreement. The demo has 100 patients; its cohorts and intervals are too small to say how much definitions disagree in practice.
• Synthetic structure is planted. Overlaps on synthetic data reflect the generator's assumptions, not real coding or lab practice.
• KDIGO is creatinine-only (no urine output) with an in-admission baseline; many studies use pre-admission baselines.
• Readmission definitions are simplified; the CMS planned-readmission algorithm is far more detailed.
• Shen et al.'s comparison group cannot be reproduced as a cohort. The paper also excludes patients "who lacked readmission time", so its controls were all readmitted after more than 90 days and patients never readmitted are in neither group. readmit_shen2025 returns the paper's cases; here, as for every definition, non-members are all other admissions.
• The downstream comparison is members vs all other admissions. For definitions with their own eligibility rules (such as Kamel Rahimi et al.'s first-ICU-stay, day-1-AKI-excluded cohort), the non-members include patients the paper excluded, so the odds ratio is not the comparison the paper made. Restricting the comparison to each definition's eligible population is a planned extension.
• Per-patient phenotypes are compared admission by admission. eMERGE classifies patients and returns all their admissions, so its cohort is weighted toward patients with many admissions (51 admissions from 9 patients on the demo), and its overlap with admission-level definitions mixes the two units.
• The odds ratio is unadjusted. It shows how a reported association can move with the definition, not a causal effect.

Trial and error

What went wrong while building this, and what changed as a result. Analysis problems come first because they affect results; engineering problems follow because they affect reproducibility.

Analysis and data

Engineering

Conclusions

"The same" cohort is often not the same patients: on real MIMIC-IV demo data, AKI definitions agree on at most 56% of their combined admissions diabetes definitions on as little as 6%, and 30- and 90-day readmission definitions on 51–60%. cohort-diff makes that difference measurable, with each definition as tested, cited SQL and paired, patient-level intervals for overlap and for the downstream association. Whether those differences change published associations on real data is the question for full MIMIC-IV, where the same code runs unchanged.

Repository layout

Reproduce

uv sync
uv run cohort-diff synth --out data/synth --patients 2000 --seed 0
uv run cohort-diff list
uv run cohort-diff compare --condition aki --data data/synth --out reports/aki.md
uv run cohort-diff compare --condition aki --data path/to/mimic-iv-clinical-database-demo-2.2

Development: uv run ruff format --check src tests, uv run ruff check src tests, uv run mypy src (strict), uv run pytest, uv build.

Citations

Data

• Johnson, A., Bulgarelli, L., Pollard, T., Horng, S., Celi, L. A., & Mark, R. (2023). MIMIC-IV Clinical Database Demo (version 2.2). PhysioNet. RRID:SCR_007345. https://doi.org/10.13026/dp1f-ex47
• Johnson, A. E. W., Bulgarelli, L., Shen, L., et al. (2023). MIMIC-IV, a freely accessible electronic health record dataset. Scientific Data, 10, 1. https://doi.org/10.1038/s41597-022-01899-x
• Pollard, T., Moody, B. E., Lehman, L., Gow, B., Fernandes, C., Xie, C., Johnson, A., Mark, R. G., & Heldt, T. (2026). PhysioNet as a global platform for biomedical research. Nature Health. https://doi.org/10.1038/s44360-026-00096-z

Published cohort definitions implemented here (each SQL file's reference: line names the exact source)

• aki_kdigo_mimiccode: MIT-LCP mimic-code, mimic-iv/concepts/organfailure/kdigo_creatinine.sql and kdigo_stages.sql, https://github.com/MIT-LCP/mimic-code (read 2026-11-04). Johnson, A. E. W., Stone, D. J., Celi, L. A., & Pollard, T. J. (2018). The MIMIC Code Repository: enabling reproducibility in critical care research. Journal of the American Medical Informatics Association, 25(1), 32–39. https://doi.org/10.1093/jamia/ocx084

• aki_rahimi2023_b1, _b2, _b3: Kamel Rahimi, A., Ghadimi, M., van der Vegt, A. H., Canfell, O. J., Pole, J. D., Sullivan, C., & Shrapnel, S. (2023). Machine learning clinical prediction models for acute kidney injury: the impact of baseline creatinine on prediction efficacy. BMC Medical Informatics and Decision Making, 23, 207. https://doi.org/10.1186/s12911-023-02306-0

• dm_charlson_mimiccode: MIT-LCP mimic-code, mimic-iv/concepts/comorbidity/charlson.sql, https://github.com/MIT-LCP/mimic-code (read 2026-11-05); Johnson et al. (2018), above. Codes from Quan, H., Sundararajan, V., Halfon, P., et al. (2005). Coding algorithms for defining comorbidities in ICD-9-CM and ICD-10 administrative data. Medical Care, 43(11), 1130–1139.

• dm_cheng2025: Cheng, J., Cheng, Q., Wu, Y., Yin, J., & He, F. (2025). Association between triglyceride-glucose-body mass index and adverse prognosis in elderly patients with severe heart failure and type 2 diabetes: a retrospective study based on the MIMIC-IV database. Cardiovascular Diabetology, 24, 299. https://doi.org/10.1186/s12933-025-02870-x

• dm_zhang2023_emerge: Zhang, W., & Cao, T. (2023). Automated type 2 diabetes case and control identification from the MIMIC-IV database. AMIA Joint Summits on Translational Science Proceedings, 2023, 602–611. PMCID: PMC10283086. Algorithm: Kho, A. N., Hayes, M. G., Rasmussen-Torvik, L., et al. (2012). Use of diverse electronic medical record systems to identify genetic risk for type 2 diabetes within a genome-wide association study. Journal of the American Medical Informatics Association, 19(2), 212–218. https://doi.org/10.1136/amiajnl-2011-000439

• readmit_gupta2022: Gupta, M., Gallamoza, B., Cutrona, N., Dhakal, P., Poulain, R., & Beheshti, R. (2022). An extensive data processing pipeline for MIMIC-IV. Proceedings of Machine Learning Research, 193, 311–325. Logic from the code: https://github.com/healthylaife/MIMIC-IV-Data-Pipeline, preprocessing/day_intervals_preproc/day_intervals_cohort_v3.py (commit afcd8a3, read 2026-11-06).

• readmit_prempeh2026: Prempeh, A. G. A., & Tenfelde, A. (2026). Opioid escalation in the ICU and 90-day readmission after major surgery: a retrospective cohort study using the MIMIC-IV database. Journal of Intensive Care Medicine, 41(10), 1008–1012. https://doi.org/10.1177/08850666261425613

• readmit_shen2025: Shen, X., Yang, J., & Zhang, X. (2025). Influencing factors of 90-day readmission of patients with heart failure based on the MIMIC-IV database. Journal of Thoracic Disease, 17(3), 1207–1216. https://doi.org/10.21037/jtd-24-1738

Clinical criteria behind the built-in definitions

• Kidney Disease: Improving Global Outcomes (KDIGO) Acute Kidney Injury Work Group (2012). KDIGO Clinical Practice Guideline for Acute Kidney Injury. Kidney International Supplements, 2(1), 1–138.

License

Code: MIT. The MIMIC-IV demo data is distributed by PhysioNet under the Open Data Commons Open Database License v1.0 and is not included in this repository.