HiRID MEDS Extraction ETL

This pipeline extracts the HiRID high time-resolution ICU dataset v1.1.1 from PhysioNet into the MEDS format. HiRID holds 33,905 admissions to the Department of Intensive Care Medicine of the Bern University Hospital, Switzerland, between January 2008 and June 2016, with physiological variables, diagnostic test results and treatment parameters recorded as often as every two minutes. It was built from the ETL_MEDS_Template and uses the config-only layout introduced by MEDS-Extract 0.7.

The ETL maps the patient table, the variable reference and the two raw_stage tables (observations and pharma records). It never downloads or reads the merged_stage / imputed_stage derivatives or the CSV copies of the raw tables.

Usage

HiRID is a credentialed-access dataset. You need a credentialed PhysioNet account that has completed the required training and signed the data use agreement.

pip install HIRID-MEDS
export DATASET_DOWNLOAD_USERNAME=$PHYSIONET_USERNAME
export DATASET_DOWNLOAD_PASSWORD=$PHYSIONET_PASSWORD
export POLARS_ENGINE_AFFINITY=streaming  # see "Resources" below
meds-extract-download spec=HIRID output_dir=$RAW_INPUT_DIR
meds-extract-run spec=HIRID output_dir=$MEDS_OUTPUT_DIR do_download=false input_dir=$RAW_INPUT_DIR

When you run this, the program will:

1. Download the three release files the ETL reads (reference_data.tar.gz, raw_stage/observation_tables_parquet.tar.gz, raw_stage/pharma_records_parquet.tar.gz, ~7 GB) into $RAW_INPUT_DIR, and unpack them.
2. Build the MEDS cohort from the unpacked files and write it to $MEDS_OUTPUT_DIR (data/ and metadata/). All transformations are declared in src/HIRID_MEDS/configs/messy.yaml.

meds-extract-run spec=HIRID output_dir=$MEDS_OUTPUT_DIR without do_download=false does both steps in one command.

Using an existing download

If you already have the release on disk, skip the download command and point input_dir at a directory holding the unpacked files, laid out the way the download step leaves them:

$RAW_INPUT_DIR/
├── general_table.csv                    # from reference_data.tar.gz
├── hirid_variable_reference.csv         # from reference_data.tar.gz
└── raw_stage/
    ├── observation_tables/parquet/part-*.parquet
    └── pharma_records/parquet/part-*.parquet

All three archives are required. The two raw_stage parquet archives alone are not enough: patients, demographics and variable names come from reference_data.tar.gz. To unpack a manual download:

cd $RAW_INPUT_DIR
tar -xzf reference_data.tar.gz
(cd raw_stage && tar -xzf observation_tables_parquet.tar.gz && tar -xzf pharma_records_parquet.tar.gz)

The download step only unpacks files it fetches itself: archives already present with a matching checksum are skipped and therefore not unpacked.

Resources

A full run reads ~777M observation rows. Run it with polars' streaming engine (export POLARS_ENGINE_AFFINITY=streaming). With it, a full run takes ~13 minutes and peaks at ~9.5 GB RSS (single worker, 10-core / 24 GB laptop). Without it, split_and_shard_subjects is killed for running out of memory on the same machine (#9).

To parallelize the stages, pass a runner file:

pip install "HIRID-MEDS[local_parallelism]"
printf 'parallelize:\n  n_workers: 8\n  launcher: joblib\n' >runner.yaml
meds-extract-run spec=HIRID output_dir=$MEDS_OUTPUT_DIR do_download=false \
	input_dir=$RAW_INPUT_DIR stage_runner_fp=runner.yaml

Run meds-extract-run --help for every option.

How this ETL is defined

The whole ETL is one MESSY (MEDS-Extract Specification Syntax YAML) file, src/HIRID_MEDS/configs/messy.yaml. The package contains no ETL code. The file declares:

• sources:: where the raw data lives (the PhysioNet release, restricted to the three archives above, unpacked after download).
• etl:: the dataset name and sharding options.
• One block per raw table: how each row becomes one or more MEDS events, and which tables it is joined with.

pyproject.toml registers the file with MEDS-Extract's MEDS_extract.pipelines entry-point group under the name HIRID. That registration is what makes spec=HIRID work.

Everything the pre-0.7 pre_MEDS.py did is now config:

Events

Variable Name and Unit come from hirid_variable_reference, matched on the raw id and its Source Table, since one id is listed as both an observation and a drug. An id with no reference entry for its own table keeps its raw id in the code (OBSERVATION//<variableid>//UNK, MEDICATION//<pharmaid>). A missing sex or unit becomes UNK.

A full run of HiRID 1.1.1 yields 33,905 admissions, 33,904 discharges, 2,062 deaths, 16.25M medication events and 758.65M observation events. See DESIGN_DECISIONS.md for every modelling choice, the numbers behind it, and the full-run validation.

Data notes

• Deidentified dates. HiRID shifts dates into the 22nd century while keeping chronology within a patient; they are emitted as supplied.
• ICU, not hospital. HiRID covers ICU admissions. The HOSPITAL_ADMISSION / HOSPITAL_DISCHARGE codes are kept from earlier releases; discharge and death are the last observation, not a recorded time (#4).
• Pseudo date of birth. age is binned (15 values, 20–90), so MEDS_BIRTH can be off by up to ~5 years.
• Duplicates are collapsed. MEDS-Extract drops events identical in every column. This removes 3.1M exact raw duplicates and merges 15.1M coincident readings of different variable ids that share a name and unit (#8).
• Unmapped columns. Validity and status columns (status, type, recordstatus, typeid, …) are not extracted (#5).

Testing

tests/test_etl.py builds a small hand-written fixture with the release's layout and dtypes, runs meds-extract-run over it, and checks the shared-id join, dtype alignment, fallback codes, text values and patient events. The synthetic ETL workflow also generates raw files straight from the config with MESSY-synth and fails if any event block yields no events.

uv sync
uv run pytest tests/ -v
uv run pre-commit run --all-files

See CONTRIBUTING.md for the development and release workflow. To run the synthetic ETL locally:

uv pip install "MESSY_synth @ git+https://github.com/rvandewater/MESSY_synth@de7a888d1045e943aabb218bcf83d2e83f4d3219"
uv run MESSY-synth ./src/HIRID_MEDS/configs/messy.yaml -o raw_synthetic/ --explain
uv run meds-extract-run spec=HIRID output_dir=./HIRID_meds_synth input_dir=raw_synthetic do_download=false

Citation

If you use this ETL, please cite the HiRID database:

@article{PhysioNet-hirid-1.1.1,
  author = {Faltys, Martin and Zimmermann, Marc and Lyu, Xinrui and Hüser, Matthias and Hyland, Stephanie and Rätsch, Gunnar and Merz, Tobias},
  title = {{HiRID, a high time-resolution ICU dataset}},
  journal = {{PhysioNet}},
  year = {2021},
  month = feb,
  note = {Version 1.1.1},
  doi = {10.13026/nkwc-js72},
  url = {https://doi.org/10.13026/nkwc-js72}
}

And its original publication:

@ARTICLE{Hyland2020-vq,
  title     = "Early prediction of circulatory failure in the intensive care
               unit using machine learning",
  author    = "Hyland, Stephanie L and Faltys, Martin and Hüser, Matthias
               and Lyu, Xinrui and Gumbsch, Thomas and Esteban, Cristóbal
               and Bock, Christian and Horn, Max and Moor, Michael and Rieck,
               Bastian and Zimmermann, Marc and Bodenham, Dean and Borgwardt,
               Karsten and Rätsch, Gunnar and Merz, Tobias M",
  journal   = "Nat. Med.",
  publisher = "Springer Science and Business Media LLC",
  volume    =  26,
  number    =  3,
  pages     = "364--373",
  month     =  mar,
  year      =  2020,
  language  = "en"
}

And this ETL:

@software{van_de_Water_HIRID_MEDS_ETL_2025,
author = {van de Water, Robin Philippus},
doi = {10.5281/zenodo.17370179},
license = {MIT},
month = oct,
title = {{HIRID MEDS ETL}},
url = {https://github.com/rvandewater/HIRID_MEDS},
year = {2025}
}

See also CITATION.cff.