MIMIC-IV MEDS Extraction ETL
This pipeline extracts the MIMIC-IV dataset (from physionet) into the MEDS format.
Usage:
pip install MIMIC_IV_MEDS
export DATASET_DOWNLOAD_USERNAME=$PHYSIONET_USERNAME
export DATASET_DOWNLOAD_PASSWORD=$PHYSIONET_PASSWORD
MEDS_extract-MIMIC_IV root_output_dir=$ROOT_OUTPUT_DIR
When you run this, the program will:
- Download the needed raw MIMIC files for the currently supported version into
$ROOT_OUTPUT_DIR/raw_input. - Perform initial, pre-MEDS processing on the raw MIMIC files, saving the results in
$ROOT_OUTPUT_DIR/pre_MEDS. - Construct the final MEDS cohort, and save it to
$ROOT_OUTPUT_DIR/MEDS_cohort.
You can also specify the target directories more directly, with
export DATASET_DOWNLOAD_USERNAME=$PHYSIONET_USERNAME
export DATASET_DOWNLOAD_PASSWORD=$PHYSIONET_PASSWORD
MEDS_extract-MIMIC_IV raw_input_dir=$RAW_INPUT_DIR pre_MEDS_dir=$PRE_MEDS_DIR MEDS_cohort_dir=$MEDS_COHORT_DIR
Examples and More Info:
You can run MEDS_extract-MIMIC_IV --help for more information on the arguments and options. You can also run
MEDS_extract-MIMIC_IV root_output_dir=$ROOT_OUTPUT_DIR do_demo=True
to run the entire pipeline over the publicly available, fully open MIMIC-IV demo dataset.
Expected runtime and compute needs
This pipeline can be successfully run over the full MIMIC-IV on a 5-core machine leveraging around 165GB of memory in approximately 7 hours (note this time includes the time to download all of the MIMIC-IV files as well, and this test was run on a machine with poor network transfer speeds and without any parallelization applied to the transformation steps, so these speeds can likely be greatly increased). The output folder of data is 9.8 GB. This can be reduced significantly as well as intermediate files not necessary for the final MEDS dataset are retained in additional folders. See this github issue for tracking on ensuring these directories are automatically cleaned up in the future.
🔧 Common Issues / FAQ
❓ Issue: FileNotFoundError or pipeline errors during the pre_MEDS step on Ubuntu (symlinks not recognized)
Problem:
Some users running the pipeline encounter errors during the pre_MEDS step, where the scripts attempt to create symlinks but later fails to recognize or access them — even though the symlinks appear to exist in the file system.
Solution:
A do_copy=True option is available in the CLI that allows the pipeline to copy files instead of symlinking, avoiding this issue entirely (at the cost of additional disk usage). You can enable this by adding do_copy=True to your command:
MEDS_extract-MIMIC_IV root_output_dir=$ROOT_OUTPUT_DIR do_copy=True
1 maintainer
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Filter files by name, interpreter, ABI, and platform.
If you're not sure about the file name format, learn more about wheel file names.
Copy a direct link to the current filters
File details
Details for the file mimic_iv_meds-0.0.4.1.tar.gz.
File metadata
- Download URL: mimic_iv_meds-0.0.4.1.tar.gz
- Upload date:
- Size: 21.4 kB
- Tags: Source
- Uploaded using Trusted Publishing? Yes
- Uploaded via:
twine/7.0.0 CPython/3.13.14
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
e55c69ec377c49b0d3c0411c0ef3d57aa436a92e8fccb492bad4a4d225b112f7
|
|
| MD5 |
2b3c134943d1f13e910a1147752f714c
|
|
| BLAKE2b-256 |
4086f2076a39cb8ef01e72ed976eda150e48b34662a457042f2cc79854946484
|
Provenance
The following attestation bundles were made for mimic_iv_meds-0.0.4.1.tar.gz:
Publisher:
python-build.yaml on Medical-Event-Data-Standard/MIMIC_IV_MEDS
-
Statement:
-
Statement type:
https://in-toto.io/Statement/v1 -
Predicate type:
https://docs.pypi.org/attestations/publish/v1 -
Subject name:
mimic_iv_meds-0.0.4.1.tar.gz -
Subject digest:
e55c69ec377c49b0d3c0411c0ef3d57aa436a92e8fccb492bad4a4d225b112f7 - Sigstore transparency entry: 2373267870
- Sigstore integration time:
-
Permalink:
Medical-Event-Data-Standard/MIMIC_IV_MEDS@dff0d4c30eff745d4ae2a0dab3a86b200c6d76c2 -
Branch / Tag:
refs/tags/0.0.4.1 - Owner: https://github.com/Medical-Event-Data-Standard
-
Access:
public
-
Token Issuer:
https://token.actions.githubusercontent.com -
Runner Environment:
github-hosted -
Publication workflow:
python-build.yaml@dff0d4c30eff745d4ae2a0dab3a86b200c6d76c2 -
Trigger Event:
push
-
Statement type:
File details
Details for the file mimic_iv_meds-0.0.4.1-py3-none-any.whl.
File metadata
- Download URL: mimic_iv_meds-0.0.4.1-py3-none-any.whl
- Upload date:
- Size: 17.1 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? Yes
- Uploaded via:
twine/7.0.0 CPython/3.13.14
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
2c9e3eb233d78885b47d4696adcc08c4e5d7139974c65f9d3984d2beab15cbfe
|
|
| MD5 |
c2eb83bdce58f97e8c1364676b7bed70
|
|
| BLAKE2b-256 |
0ee8832199d75542ea38ca275fedcfc67e6754794a450464ab166c32e442473a
|
Provenance
The following attestation bundles were made for mimic_iv_meds-0.0.4.1-py3-none-any.whl:
Publisher:
python-build.yaml on Medical-Event-Data-Standard/MIMIC_IV_MEDS
-
Statement:
-
Statement type:
https://in-toto.io/Statement/v1 -
Predicate type:
https://docs.pypi.org/attestations/publish/v1 -
Subject name:
mimic_iv_meds-0.0.4.1-py3-none-any.whl -
Subject digest:
2c9e3eb233d78885b47d4696adcc08c4e5d7139974c65f9d3984d2beab15cbfe - Sigstore transparency entry: 2373268138
- Sigstore integration time:
-
Permalink:
Medical-Event-Data-Standard/MIMIC_IV_MEDS@dff0d4c30eff745d4ae2a0dab3a86b200c6d76c2 -
Branch / Tag:
refs/tags/0.0.4.1 - Owner: https://github.com/Medical-Event-Data-Standard
-
Access:
public
-
Token Issuer:
https://token.actions.githubusercontent.com -
Runner Environment:
github-hosted -
Publication workflow:
python-build.yaml@dff0d4c30eff745d4ae2a0dab3a86b200c6d76c2 -
Trigger Event:
push
-
Statement type:
