eICU MEDS Extraction ETL

This repository contains the code for downloading the eICU dataset from PhysioNet and transforming it into the Medical Event Data Standard (MEDS) format.

pip install eICU-MEDS
export DATASET_DOWNLOAD_USERNAME=$PHYSIONET_USERNAME
export DATASET_DOWNLOAD_PASSWORD=$PHYSIONET_PASSWORD
MEDS_extract-eICU root_output_dir=data/eicu_meds

When you run this, the program will:

1. Download the raw eICU-CRD tables this ETL reads (see Which eICU tables are extracted) for the currently supported version into $ROOT_OUTPUT_DIR/raw_input (via MEDS-Extract's download layer; files that already exist and verify against PhysioNet's SHA256SUMS.txt are skipped, so an interrupted run resumes rather than restarting).
2. Construct the MEDS cohort directly from those raw files — every transformation, join, and pseudotime derivation is declared in src/eICU_MEDS/configs/event_configs.yaml — and write it to $ROOT_OUTPUT_DIR/MEDS_output.

The public demo needs no credentials:

MEDS_extract-eICU root_output_dir=data/eicu_demo do_demo=True

You can also point the raw and output directories somewhere else directly. root_output_dir is still required, since the intermediate pre_MEDS tables and the logs are written under it:

MEDS_extract-eICU root_output_dir=$ROOT_OUTPUT_DIR \
    raw_input_dir=$RAW_INPUT_DIR MEDS_output_dir=$MEDS_OUTPUT_DIR

A note on time in eICU

eICU contains no absolute timestamps. Every table records an integer offset in minutes relative to unit admission, and health-system stays are ordered only at year granularity. This pipeline therefore:

• uses the health system stay (patienthealthsystemstayid) as the MEDS subject_id, since events are well ordered only within a stay; and
• anchors each stay at a constant, arbitrary pseudo-date (December 31 of the recorded discharge year, at the recorded discharge clock time) and derives every event time by offsetting from it.

Only relative time differences within a subject are meaningful. Absolute dates in the output are not real and must not be interpreted as such: because every stay ends on December 31, the date of an event reveals how long until hospital discharge. See docs/data_leakage.md for this and every other way the output can leak information, table by table.

Death

MEDS_DEATH is emitted at hospital discharge when hospitalDischargeStatus == "Expired", and from no other field. The documentation describes Expired as the patient's condition on leaving the hospital, and on the full release it agrees with the independently populated hospitalDischargeLocation == "Death" on every one of the 200,859 rows (18,004 deaths).

Hospital rather than unit status is the key because it is the more complete of the two: 7,315 rows record a unit discharge Alive followed by a hospital death, which unit status would miss.

The converse is deliberately not used. 222 rows (0.1%) have unitDischargeStatus == "Expired" while hospital status is Alive or blank, and these emit no MEDS_DEATH. They contradict themselves: in 142 of them the hospital stay continues past the unit discharge at which the patient supposedly died, some to a discharge home. Rows with a blank hospital status (1,751) are likewise not treated as deaths. See #11.

Vital signs: only measurements that happened

vitalPeriodic and vitalAperiodic are wide tables — one row per timestamp, one column per vital sign — and most cells are empty. A row that recorded a heart rate still carries columns for intracranial pressure, end-tidal CO2 and the pulmonary-artery pressures, which need invasive lines that most patients never have.

Each vital's code is therefore guarded so that a missing cell emits no event, rather than an event asserting the measurement was taken with no value attached. Without the guard the full cohort carries 2.60 billion vitals rows of which 69.8% are empty; with it, 785 million, all of them real. Per-vital emptiness in the raw data ranges from 0.5% (heart rate) to 98.2% (intracranial pressure).

The guard is written if $x == $x, which is true for any present value — including 0, inf and NaN — and null for a missing one, which nulls the code and drops the row. It reads oddly because dftly has no null test; if $x will not type-check against a float column and a bare null parses as the string "null". Tracked in dftly#113; this becomes if is_present($x) once that lands.

NaN and inf are deliberately preserved rather than dropped — they are recorded values, and changing that is a data-modelling decision rather than a bug fix.

Numeric-value guards

numeric_value is strict-cast downstream, so a String source column containing a single non-numeric row aborts the stage. Which columns are String is not inferable from the demo — it depends on what junk happens to appear — so the guarded set comes from the official eICU DDL (MIT-LCP/eicu-code, postgres_create_tables.sql): age, medication.dosage, infusiondrug.{infusionrate, patientweight}, nurseCharting.nursingChartValue, respiratoryCharting.respChartValue, nurseAssessment.cellAttributeValue, nurseCare.cellAttributeValue and physicalExam.physicalExamText. Numeric-typed columns are passed through bare, since the schema forbids junk in them.

Where the non-numeric remainder is worth keeping (charted values, below), the same pattern also appears negated as a regex match, $x if not (/<guard>/ in $x), to fill text_value, so each event carries exactly one of the two values.

The guard regex is exactly Rust's f64::from_str grammar, so it is value-for-value identical to polars' non-strict cast. A narrower hand-rolled pattern silently drops real values such as 1e5, 25. and +5; tests/test_numeric_guard.py pins the equivalence by differential fuzzing.

Charted observations

Several tables record each observation as one varchar cell under picklist labels or a label path. Each row becomes one event whose code is the table's prefix followed by those labels:

• A value the numeric guard accepts goes to numeric_value. Anything else, such as GCS descriptors (4-->(E4) spontaneous), Oral, WDL or Nasal cannula, goes to text_value. A value written as a percentage (35%) is text: stripping the % would need a second guard pattern.
• A blank cell emits no event.
• The time is the entry offset, not the charted offset, so a value is not visible before it was recorded. The last two columns show how far apart the two are.
• The category is part of the code because the same label can sit under more than one category: PA under both Invasive and Vital Signs in nurseCharting (96 codes from 83 label/name pairs on the full release), and 198 codes from 194 labels in respiratoryCharting.
• In the path-coded tables the path ends in the attribute (…|Neurologic|Pupils|Right) and determines the label, so the path alone is the code. They cover few stays: nurseAssessment has rows for 13,001 and nurseCare for 13,052 of the 200,859 unit stays on the full release.
• Vitals charted here overlap with vitalPeriodic/vitalAperiodic. They keep their own source-prefixed codes, and nothing is deduplicated across tables.
• The two charted tables overlap in one measurement: on the full release, 79% of respiratoryCharting's RR (patient) values repeat nurseCharting's Respiratory Rate for the same stay at the same minute. Ventilator settings and measurements (PEEP, tidal volume, pressure support, plateau pressure, compliance) appear only in respiratoryCharting.

Intake and output

intakeOutput is the nursing intake/output flowsheet. Each row becomes one INTAKE_OUTPUT//<cellPath> event, for example INTAKE_OUTPUT//flowsheet|Flowsheet Cell Labels|I&O|Output (ml)|Urine, valued by the row's own cellValueNumeric and timed at intakeOutputEntryOffset, as for the charted tables. The path ends in the row's label, so the label adds nothing to the code.

• intakeTotal, outputTotal, dialysisTotal and netTotal are not emitted. The documentation describes them as cumulative up to the row's offset and repeated across rows entered at the same time, so emitting them would count the same fluid many times.
• cellValueText is not emitted. It is the same number written as text.
• Rows are not deduplicated. On the full release 9,619 rows repeat another row in everything but their ID, and the table gives no way to tell a repeated entry from two equal ones.
• The documentation says cellValueNumeric is always positive. 1,476 full-release rows are negative, and they are emitted as recorded.

Physical exam

physicalExam holds the structured physical exam entered in eCareManager. Each row becomes one PHYSICAL_EXAM//<physicalExamPath> event, for example PHYSICAL_EXAM//notes/Progress Notes/Physical Exam/Physical Exam/Neurologic/GCS/Motor Score/6, timed at physicalExamOffset, the table's only offset, which the documentation describes as the time the item was entered.

• The path ends in the picklist choice (physicalExamValue), so the choice is already in the code.
• physicalExamText is split like the charted values. For 63 vital-sign, weight and I&O items (HR Current, Weight (kg)/Admission, …) it is the measurement and goes to numeric_value; for the other items it mostly repeats the choice and goes to text_value.
• A blank text does not drop the event, because the path is the finding. On the full release 41,327 rows have a blank text, all in items that are not measurements, and their events carry no value.
• The vital signs here are the exam's summaries (current, lowest and highest over 24 hours). Like the charted vitals, they keep their own codes and are not deduplicated against vitalPeriodic.

Code metadata: descriptions

metadata/codes.parquet carries code, description, parent_codes and code_template. eICU ships no code dictionary (nothing like MIMIC-IV's d_labitems): its tables record the names themselves (labName, drugName, treatmentString, diagnosisString), so the codes are human-readable on their own and releases up to 0.1.0 wrote no description. That is enough for a reader, not for tooling that maps codes across datasets or builds features from descriptions: vital codes are column names (VITALS//PERIODIC//HEARTRATE), lab names are site picklist abbreviations (Hgb, -polys, WBC x 1000), and vocabulary-based enrichment (OHDSI Athena) cannot resolve either. So every code now gets the documented meaning of what it encodes, through three routes, with the source of each description recorded:

The lab table carries descriptions only. eICU records no LOINC code for its lab names, so the ETL writes no LOINC parent for them; the lab-name to LOINC crosswalk is part of suMEDS's versioned ontology data and is applied after extraction (see Enriching code metadata with suMEDS). Diagnoses keep their ICD parents (below), which come from eICU's own icd9code column.

The lab table lists only lab names that are already public: the 147 in the open eICU-CRD demo (ODbL) and one more in eicu-code (MIT). It ships nothing that exists only in the credentialed release. A lab name outside the table gets a generic description from the ETL's last step, built from the code itself, for example Lab result (eICU lab.labName '<labname>').

Why three routes: MEDS-Extract attaches _metadata by joining on the code's components, so a literal code has none to join on, and the pre-existing metadata/codes.parquet it documents for that case is not read by the standard extraction pipeline (extract_code_metadata reads its prior metadata stage's output, split_and_shard_subjects, which writes no codes.parquet). The post-pipeline fill only ever writes a description where the pipeline left null, never extends the vocabulary, and is idempotent.

On the demo (eICU-CRD demo 2.0.1) every one of the 8,122 codes is described: 294 lab codes, 34 literal codes, and 7,794 component-coded codes. Event data is untouched: per-code event counts are identical to a build without this metadata.

On the full release (eICU-CRD 2.0) every one of the 15,528 codes is described: 316 lab codes (a RESULT and a SPECIMEN_DRAWN code for each of the 158 lab names; 148 names are in the table and the other 10 get the generic description), 34 literal codes, and 15,178 component-coded codes. Every code in the data is in codes.parquet, and the file validates against meds.CodeMetadataSchema.

MEDS 0.4 makes description optional, so consumers must still not rely on it being non-null for datasets built with older releases.

ICD parent codes for diagnoses are the other piece of code metadata: each DIAGNOSIS//... code gets ICD9CM/... and/or ICD10CM/... parent codes from the icd9code column of diagnosis (via the dx_icd_map table built in pre-MEDS). In the demo, 860 of the 1,091 diagnosis codes have one; in the full release, 3,129 of the 3,933. parent_codes is written sorted, so that two runs over the same data write the same file.

Enriching code metadata with suMEDS

The ETL writes what eICU itself records. Links to external vocabularies that eICU does not record, and parents among eICU's own codes, are added by suMEDS with the rules file shipped in this package, src/eICU_MEDS/sumeds_rules.yaml. The crosswalk tables and ontology packs come from suMEDS's versioned ontology data (sumeds-ontologies, pinned release), so this repository does not ship them, and no Athena download or LOINC account is needed.

To enrich as part of the ETL, install the sumeds extra and turn on do_enrich:

pip install "eICU-MEDS[sumeds]"
MEDS_extract-eICU root_output_dir=data/eicu_meds do_enrich=True

The last step then rewrites MEDS_output/metadata/codes.parquet in place. The run checks that suMEDS is installed before it starts. If the enrichment itself fails, for example without network access, the ETL's codes.parquet is left as it was. Enrichment is off by default because suMEDS is an extra dependency, because it downloads its ontology data from GitHub on first use (cached under ~/.cache/sumeds; on a machine without internet access, point SUMEDS_ONTOLOGY_SOURCE at a local copy of the pinned sumeds-ontologies release), and because the default output then holds only what eICU records.

To enrich an existing output instead, run suMEDS on its own. Given the same codes.parquet, it writes the same table as do_enrich=True:

pip install "sumeds>=0.4"
suMEDS-enrich $MEDS_DIR/metadata/codes.parquet -o codes_enriched.parquet \
  --rules "$(python -c 'from importlib.resources import files; print(files("eICU_MEDS") / "sumeds_rules.yaml")')"

This needs suMEDS 0.4 or later, the first release with ontology rules. The rules add:

• LOINC for lab results (eicu-lab-loinc, a crosswalk): LAB//RESULT//{labname}//{unit} gets LOINC//<code> in parent_codes, plus vocabulary_id and concept_code, for the 50 lab names in the crosswalk table eicu/lab_loinc, each validated against the LOINC pack. The links are an inference across two curated mappings (eICU name → WeavEHR concept ← MIMIC-IV item with its LOINC code in mimic-code d_labitems_to_loinc.csv), so the LOINC code inherits the MIMIC item's specimen (serum/plasma versus blood); the table records both ends.
• Parents among eICU codes (eicu-diagnosis-paths, eicu-admission-diagnosis-paths, eicu-treatment-paths, hierarchy rules): a |-separated path gets the nearest shorter path that is itself a code, e.g. TREATMENT//ENTERED//a|b|c gets TREATMENT//ENTERED//a|b. On the full release that adds 2,929 diagnosis and 2,096 treatment parents; admission diagnosis paths are recorded only as leaves, so that rule adds none today.

The ETL alone no longer emits LOINC parents for lab codes (builds of main after #27 wrote LOINC/<code> for the same 50 lab names), and it never emitted parents among eICU codes: metadata/codes.parquet from the ETL has ICD parents for diagnoses only, and suMEDS supplies the rest: it links the same 50 lab codes to the same LOINC codes the ETL used to write, and it never removes or rewrites the ETL's own parents or descriptions.

Which eICU tables are extracted

eICU-CRD v2.0 ships 31 tables. This pipeline reads 22 of them. The rest are excluded deliberately — the reasoning is recorded here rather than in a code comment, because "this table is missing" is otherwise indistinguishable from an oversight.

Extracted: patient, hospital (joined for site attributes), admissionDx, allergy, carePlanEOL, carePlanGeneral, carePlanGoal, carePlanInfectiousDisease, diagnosis, infusionDrug, intakeOutput, lab, medication, nurseAssessment, nurseCare, nurseCharting, pastHistory, physicalExam, respiratoryCharting, treatment, vitalAperiodic, vitalPeriodic.

Excluded — deliberate, on the merits:

Excluded — not yet done, not on the merits. These carry real clinical signal and were stubbed out but never finished (the original config shipped commented-out blocks for them marked NOT YET DONE, because their cell-label/value structure needs a code scheme of its own):

MEDS-transforms settings

If you want to convert a large dataset, you can use parallelization with MEDS-transforms (the MEDS-transformation step that takes the longest).

Local parallelization uses the hydra-joblib-launcher package, which is installed as a dependency of eICU-MEDS. You can set the number of workers as an environment variable:

export N_WORKERS=8

Size N_WORKERS to memory, not cores: each worker is a separate process with its own working set. On the full release, N_WORKERS=4 exhausted a 24 GB laptop, with each of the four workers holding 7.5 to 8 GB during split_and_shard_subjects. That is a metadata stage, so every worker repeats the whole computation rather than taking a share of it. The same build run serially (N_WORKERS unset, polars limited to four threads with POLARS_MAX_THREADS=4) peaked at 4.8 GB and took 20 minutes on a 10-core machine.

The number of subjects per shard is fixed at 1,000 in the split_and_shard_subjects stage of src/eICU_MEDS/configs/ETL.yaml; no environment variable or command-line option changes it. Other MEDS ETLs do the same (MIMIC-IV fixes it in its spec's etl: block): MEDS-Extract treats shard size as a property of the dataset, while worker counts belong to the machine running it. To use a different shard size, edit that file in a local checkout (see Development).

Development

Contributors should use uv to set up a development environment from the lockfile:

git clone https://github.com/Medical-Event-Data-Standard/eICU_MEDS.git
cd eICU_MEDS
uv sync --group dev
uv run pre-commit install
uv run pytest src/ tests/ --doctest-modules

The end-to-end test (tests/e2e_demo_test.py) downloads the public eICU demo from PhysioNet, so it is slow on a first run. DEVELOPMENT.md covers testing, running the ETL on the full release, and cutting a release with uv version --bump.

Citation

If you use this code in your research, please cite the following:

@article{PhysioNet-eicu-crd-2.0,
  author = {Pollard, Tom and Johnson, Alistair and Raffa, Jesse and Celi, Leo Anthony and Badawi, Omar and Mark, Roger},
  title = {{eICU Collaborative Research Database}},
  journal = {{PhysioNet}},
  year = {2019},
  month = apr,
  note = {Version 2.0},
  doi = {10.13026/C2WM1R},
  url = {https://doi.org/10.13026/C2WM1R}
}
@article{Pollard2018-ln,
  title     = "The {eICU} Collaborative Research Database, a freely available
               multi-center database for critical care research",
  author    = "Pollard, Tom J and Johnson, Alistair E W and Raffa, Jesse D and
               Celi, Leo A and Mark, Roger G and Badawi, Omar",
  journal   = "Sci. Data",
  publisher = "Springer Science and Business Media LLC",
  volume    =  5,
  number    =  1,
  pages     = "180178",
  month     =  sep,
  year      =  2018,
  copyright = "https://creativecommons.org/licenses/by/4.0",
  language  = "en"
}

@software{McDermott_eICU_MEDS_eICU_MEDS,
author = {McDermott, Matthew and Zhu, Mingcheng and van de Water, Robin Philippus},
license = {MIT},
title = {{eICU\_MEDS: eICU MEDS Extraction ETL}},
url = {https://github.com/Medical-Event-Data-Standard/eICU_MEDS/}
}