eICU MEDS Extraction ETL

This repository contains the code for downloading the eICU dataset from PhysioNet and transforming it into the Medical Event Data Standard (MEDS) format.

pip install eICU-MEDS
export DATASET_DOWNLOAD_USERNAME=$PHYSIONET_USERNAME
export DATASET_DOWNLOAD_PASSWORD=$PHYSIONET_PASSWORD
MEDS_extract-eICU root_output_dir=data/eicu_meds

When you run this, the program will:

1. Download the raw eICU-CRD tables this ETL reads (see Which eICU tables are extracted) for the currently supported version into $ROOT_OUTPUT_DIR/raw_input (via MEDS-Extract's download layer; files that already exist and verify against PhysioNet's SHA256SUMS.txt are skipped, so an interrupted run resumes rather than restarting).
2. Construct the MEDS cohort directly from those raw files — every transformation, join, and pseudotime derivation is declared in src/eICU_MEDS/configs/event_configs.yaml — and write it to $ROOT_OUTPUT_DIR/MEDS_output.

The public demo needs no credentials:

MEDS_extract-eICU root_output_dir=data/eicu_demo do_demo=True

You can also point the raw and output directories somewhere else directly. root_output_dir is still required, since the intermediate pre_MEDS tables and the logs are written under it:

MEDS_extract-eICU root_output_dir=$ROOT_OUTPUT_DIR \
    raw_input_dir=$RAW_INPUT_DIR MEDS_output_dir=$MEDS_OUTPUT_DIR

A note on time in eICU

eICU contains no absolute timestamps. Every table records an integer offset in minutes relative to unit admission, and health-system stays are ordered only at year granularity. This pipeline therefore:

• uses the health system stay (patienthealthsystemstayid) as the MEDS subject_id, since events are well ordered only within a stay; and
• anchors each stay at a constant, arbitrary pseudo-date (December 31 of the recorded discharge year, at the recorded discharge clock time) and derives every event time by offsetting from it.

Only relative time differences within a subject are meaningful. Absolute dates in the output are not real and must not be interpreted as such.

Death

MEDS_DEATH is emitted at hospital discharge when hospitalDischargeStatus == "Expired", and from no other field. The documentation describes Expired as the patient's condition on leaving the hospital, and on the full release it agrees with the independently populated hospitalDischargeLocation == "Death" on every one of the 200,859 rows (18,004 deaths).

Hospital rather than unit status is the key because it is the more complete of the two: 7,315 rows record a unit discharge Alive followed by a hospital death, which unit status would miss.

The converse is deliberately not used. 222 rows (0.1%) have unitDischargeStatus == "Expired" while hospital status is Alive or blank, and these emit no MEDS_DEATH. They contradict themselves: in 142 of them the hospital stay continues past the unit discharge at which the patient supposedly died, some to a discharge home. Rows with a blank hospital status (1,751) are likewise not treated as deaths. See #11.

Vital signs: only measurements that happened

vitalPeriodic and vitalAperiodic are wide tables — one row per timestamp, one column per vital sign — and most cells are empty. A row that recorded a heart rate still carries columns for intracranial pressure, end-tidal CO2 and the pulmonary-artery pressures, which need invasive lines that most patients never have.

Each vital's code is therefore guarded so that a missing cell emits no event, rather than an event asserting the measurement was taken with no value attached. Without the guard the full cohort carries 2.60 billion vitals rows of which 69.8% are empty; with it, 785 million, all of them real. Per-vital emptiness in the raw data ranges from 0.5% (heart rate) to 98.2% (intracranial pressure).

The guard is written if $x == $x, which is true for any present value — including 0, inf and NaN — and null for a missing one, which nulls the code and drops the row. It reads oddly because dftly has no null test; if $x will not type-check against a float column and a bare null parses as the string "null". Tracked in dftly#113; this becomes if is_present($x) once that lands.

NaN and inf are deliberately preserved rather than dropped — they are recorded values, and changing that is a data-modelling decision rather than a bug fix.

Numeric-value guards

numeric_value is strict-cast downstream, so a String source column containing a single non-numeric row aborts the stage. Which columns are String is not inferable from the demo — it depends on what junk happens to appear — so the guarded set comes from the official eICU DDL (MIT-LCP/eicu-code, postgres_create_tables.sql): age, medication.dosage, and infusiondrug.{infusionrate, patientweight}. Numeric-typed columns are passed through bare, since the schema forbids junk in them.

The guard regex is exactly Rust's f64::from_str grammar, so it is value-for-value identical to polars' non-strict cast. A narrower hand-rolled pattern silently drops real values such as 1e5, 25. and +5; tests/test_numeric_guard.py pins the equivalence by differential fuzzing.

Which eICU tables are extracted

eICU-CRD v2.0 ships 31 tables. This pipeline reads 15 of them. The rest are excluded deliberately — the reasoning is recorded here rather than in a code comment, because "this table is missing" is otherwise indistinguishable from an oversight.

Extracted: patient, hospital (joined for site attributes), admissionDx, allergy, carePlanEOL, carePlanGeneral, carePlanGoal, carePlanInfectiousDisease, diagnosis, infusionDrug, lab, medication, treatment, vitalAperiodic, vitalPeriodic.

Excluded — deliberate, on the merits:

Excluded — not yet done, not on the merits. These carry real clinical signal and were stubbed out but never finished (the original config shipped commented-out blocks for them marked NOT YET DONE, because their cell-label/value structure needs a code scheme of its own):

MEDS-transforms settings

If you want to convert a large dataset, you can use parallelization with MEDS-transforms (the MEDS-transformation step that takes the longest).

Local parallelization uses the hydra-joblib-launcher package, which is installed as a dependency of eICU-MEDS. You can set the number of workers as an environment variable:

export N_WORKERS=8

The number of subjects per shard is fixed at 1,000 in the split_and_shard_subjects stage of src/eICU_MEDS/configs/ETL.yaml; no environment variable or command-line option changes it. Other MEDS ETLs do the same (MIMIC-IV fixes it in its spec's etl: block): MEDS-Extract treats shard size as a property of the dataset, while worker counts belong to the machine running it. To use a different shard size, edit that file in a local checkout (see Development).

Development

Contributors should use uv to set up a development environment from the lockfile:

git clone https://github.com/Medical-Event-Data-Standard/eICU_MEDS.git
cd eICU_MEDS
uv sync --group dev
uv run pre-commit install
uv run pytest src/ tests/ --doctest-modules

The end-to-end test (tests/e2e_demo_test.py) downloads the public eICU demo from PhysioNet, so it is slow on a first run.