cohort-diff

Keywords: EHR · MIMIC-IV · cohort definitions · phenotyping · acute kidney injury · diabetes · 30-day readmission · reproducibility · odds ratio · Jaccard index · bootstrap confidence intervals

Skills: Python · SQL · DuckDB · pandas · NumPy · patient-level bootstrap · uv packaging · PyPI (Trusted Publishing) · GitHub Actions CI · pytest · Hypothesis (property-based testing) · test oracles · mypy (strict) · ruff · AI-assisted engineering (Claude Code with CLAUDE.md, tests first, AI log)

How different are "the same" EHR cohorts? cohort-diff runs several definitions of one condition side by side on MIMIC-IV-shaped data and reports how much the cohorts, and the results built on them, disagree.

Description

A cohort definition is a modelling decision, but papers usually report it in a sentence. Two studies that both say "AKI patients in MIMIC-IV" may share only a fraction of their admissions: one counts discharge diagnosis codes, another a creatinine rise within 48 hours, another a rise over 7 days. A finding such as "AKI is associated with higher in-hospital mortality" can change size with the definition. cohort-diff stores each definition as a versioned SQL file with its source, runs them on DuckDB directly over MIMIC's .csv.gz files, and reports overlap, who is in each cohort, and how a typical downstream association moves.

Objective

1. Build a tested, pip-installable library that runs cohort definitions as reviewable SQL and checks each one against an independent implementation.
2. Measure how much definitions of the same condition (AKI, diabetes, 30-day readmission) disagree, in membership and in a downstream result (the odds ratio between membership and in-hospital mortality), with confidence intervals.
3. (Extension, branch cohort-diff-llm.) Measure how often a large language model, given a paper's methods text, drafts a cohort definition that passes the same oracle tests as the hand-written one.

Hypotheses

• H1. Definitions of the same condition select substantially different admissions (pairwise Jaccard well below 1).
• H2. Code-based and lab-based definitions differ systematically: codes under-capture, and lab rules with fixed thresholds over-capture when measurements are frequent and noisy.
• H3. The mortality odds ratio reported for "the same" condition changes materially with the definition: for some pairs, the 95% interval of the ratio of their odds ratios excludes 1.
• H4. A definition limits which questions its cohort can answer (for example, readmission exclusions that remove every death).
• H5. (Extension.) LLM-drafted definitions often run without error but fail the oracle tests at clinically important details (thresholds, time windows, specimen type, boundary conditions), so tests, not inspection, decide whether a draft can be used.

Samples

The demo shares the full MIMIC-IV schema, so the same definitions run on full MIMIC-IV (credentialed) unchanged.

Methods

• Definitions as data. Each definition is one SQL file in src/cohort_diff/definitions/ with a header (name, condition, description, reference) and returns one column, hadm_id. Adding a definition does not touch Python.
• Built-in definitions.

• Engine. DuckDB, in process, reads the MIMIC tables in place (demo folder layout or flat CSVs) as views. Lab items are found by label and fluid, not by itemid.
• Correctness. Every rule-based definition is compared with an independent, deliberately slow pandas implementation (a test oracle written with loops rather than joins), on the synthetic data and on hundreds of random creatinine series generated by Hypothesis. Hand-picked regression tests pin threshold edge cases that both implementations could get wrong together.
• Comparison. For every pair of definitions: admissions in both, in only one, and the Jaccard index |A ∩ B| / |A ∪ B|. For each cohort: size, prevalence, mean age, % female, in-hospital mortality.
• Downstream result. Unadjusted odds ratio between membership and in-hospital mortality, with a Woolf 95% CI (log OR ± 1.96·√(1/a + 1/b + 1/c + 1/d)); 0.5 is added to every cell when any cell is 0 (Haldane–Anscombe).
• Uncertainty. Patient-level bootstrap (1,000 resamples; a patient drawn k times contributes all their admissions k times) gives 95% percentile intervals for each pair's Jaccard index and for the ratio of their mortality odds ratios (OR of a / OR of b). An interval for the ratio that excludes 1 means the two definitions disagree about the association, beyond sampling noise. A test checks that patients, not admissions, are resampled.
• Scale check. Wall time of compare for each condition, on the demo and (once credentialed) on full MIMIC-IV, on one laptop.
• Extension: LLM-drafted definitions (planned, branch cohort-diff-llm). An LLM receives a paper's methods text and the table and column names only (never patient data) and returns a definition through JSON-schema structured output. Each draft is graded by the same oracle and property tests, and by exact agreement with the hand-written definition on synthetic data. Pass rates for 3+ models on a 20-item labeled set, with 95% intervals; every failure is categorized.
• Engineering. Strict typing, lint and tests on Python 3.11–3.13 in CI; the wheel is built in CI; releases go to PyPI by Trusted Publishing from version tags.

Results

Synthetic data (2,000 patients, 2,738 admissions, seed 0)

Pairwise overlap and disagreement in the mortality association (patient-level bootstrap, 1,000 resamples, 95% intervals):

Bold: the interval excludes 1, so the two definitions disagree about the mortality association beyond sampling noise.

• Code vs lab definitions barely overlap (Jaccard 0.09–0.16), while two lab rules (KDIGO 48 h vs 7 d) or two code-like rules (ICD vs medications) overlap 81–88%.
• High overlap does not mean the same answer. The two KDIGO definitions share 88% of admissions, yet their mortality ORs differ about threefold (ratio 2.87 [1.87, 5.15]).
• Different-looking ORs can be compatible. aki_icd (3.06) and aki_kdigo_48h (4.23) look different, but the interval for their ratio includes 1.
• KDIGO-48h flags 78% of synthetic admissions: the generator draws creatinine about every 12 h with noise (SD 0.3 mg/dL), and a 0.3 mg/dL rule fires on noise when labs are frequent. Whether real data shows the same is what the demo and full MIMIC-IV can answer.
• Readmission cohorts contain no deaths by construction (an admission followed by another one cannot end in death), so their mortality ORs come only from the 0.5 zero-cell correction. The definition limits which questions the cohort can answer.

MIMIC-IV demo (real data: 100 patients, 275 admissions)

• Real definitions also select different patients. The AKI definitions have similar prevalence (19–25%) but share only 31–56% of their combined admissions. Diabetes by HbA1c ≥ 6.5% (6% of admissions) overlaps the code and medication definitions at Jaccard 0.11–0.13.
• The medication phenotype captures the most (49% of admissions vs 40% by codes), consistent with insulin given for stress hyperglycemia; every HbA1c-defined admission is also in the medication cohort.
• KDIGO-48h flags 25% of demo admissions, not the 78% seen on synthetic data, so the synthetic over-capture came from the generator's frequent, noisy creatinine, not from the rule itself.
• No disagreement in the mortality association can be shown at this size. The AKI ORs range from 9.4 to 14.3 and the diabetes ORs from 0.45 to 1.20, but every ratio interval includes 1 and some are 20-fold wide.

With 100 patients, cohorts are small and intervals wide; these numbers show the tool runs on real data, not how much definitions disagree in practice.

Scale check

Full MIMIC-IV (about 546,000 admissions) will be timed once credentialed.

Extension: LLM-drafted definitions (planned)

Not started. Planned on branch cohort-diff-llm after v0.1.0: pass rates of LLM-drafted definitions on the oracle tests for 3+ models on a 20-item labeled set, with 95% intervals and every failure categorized.

Correctness

27 tests pass with 99% line coverage on Python 3.11–3.13 (CI). Two deliberate breaks show what each kind of test catches:

Discussion

On synthetic data, where the structure is known, definitions of the same condition overlap little when they use different sources (codes vs labs, Jaccard 0.09–0.16; H1), and four of six pairs disagree about the mortality association beyond sampling noise (ratio intervals excluding 1; H3). High overlap does not guarantee agreement: the two KDIGO rules share 88% of admissions but their ORs differ about threefold.

The open demo shows the same pattern in membership on real data: AKI definitions with almost identical prevalence share at most 56% of their admissions, and diabetes by HbA1c overlaps the other definitions at about 0.12 (H1). The medication phenotype captures the most admissions, as expected (H2). For AKI, H2 was not supported on the demo: the lab rule flagged the same share as the codes (25%), and the synthetic data's 78% came from the generator. Whether the mortality association itself changes with the definition (H3) cannot be tested with 100 patients: every interval for the ratio of ORs includes 1. Readmission definitions exclude deaths by construction, so they cannot answer a mortality question at all (H4).

Strengths. Definitions are versioned, cited SQL rather than prose; each is checked against an independent implementation and by property tests; known-structure validation on synthetic data; installable with pip; reproducible by anyone from open data, a lock file and CI on Python 3.11–3.13.

Limitations

• No real-world estimate of disagreement. The demo has 100 patients; its cohorts and intervals are too small to say how much definitions disagree in practice.
• Synthetic structure is planted. Overlaps on synthetic data reflect the generator's assumptions, not real coding or lab practice.
• KDIGO is creatinine-only (no urine output) with an in-admission baseline; many studies use pre-admission baselines.
• Readmission definitions are simplified; the CMS planned-readmission algorithm is far more detailed.
• The odds ratio is unadjusted. It shows how a reported association can move with the definition, not a causal effect.

Trial and error

What went wrong while building this, and what changed as a result. Analysis problems come first because they affect results; engineering problems follow because they affect reproducibility.

Analysis and data

Engineering

Conclusions

"The same" cohort is often not the same patients: on real MIMIC-IV demo data, AKI definitions agree on at most 56% of their combined admissions and diabetes definitions on as little as 11%. cohort-diff makes that difference measurable, with each definition as tested, cited SQL and paired, patient-level intervals for overlap and for the downstream association. Whether those differences change published associations on real data is the question for full MIMIC-IV, where the same code runs unchanged.

Repository layout

Reproduce

uv sync
uv run cohort-diff synth --out data/synth --patients 2000 --seed 0
uv run cohort-diff list
uv run cohort-diff compare --condition aki --data data/synth --out reports/aki.md
uv run cohort-diff compare --condition aki --data path/to/mimic-iv-clinical-database-demo-2.2

Development: uv run ruff format --check src tests, uv run ruff check src tests, uv run mypy src (strict), uv run pytest, uv build.

Citations

• Johnson, A., Bulgarelli, L., Pollard, T., Horng, S., Celi, L. A., & Mark, R. (2023). MIMIC-IV Clinical Database Demo (version 2.2). PhysioNet. RRID:SCR_007345. https://doi.org/10.13026/dp1f-ex47
• Johnson, A. E. W., Bulgarelli, L., Shen, L., et al. (2023). MIMIC-IV, a freely accessible electronic health record dataset. Scientific Data, 10, 1. https://doi.org/10.1038/s41597-022-01899-x
• Pollard, T., Moody, B. E., Lehman, L., Gow, B., Fernandes, C., Xie, C., Johnson, A., Mark, R. G., & Heldt, T. (2026). PhysioNet as a global platform for biomedical research. Nature Health. https://doi.org/10.1038/s44360-026-00096-z
• Kidney Disease: Improving Global Outcomes (KDIGO) Acute Kidney Injury Work Group (2012). KDIGO Clinical Practice Guideline for Acute Kidney Injury. Kidney International Supplements, 2(1), 1–138.

License

Code: MIT. The MIMIC-IV demo data is distributed by PhysioNet under the Open Data Commons Open Database License v1.0 and is not included in this repository.