MIMIC-IV MEDS Extraction ETL

PyPI - Version Documentation Status codecov tests code-quality python license PRs contributors

This pipeline extracts the MIMIC-IV dataset (from physionet) into the MEDS format.

Usage:

pip install MIMIC_IV_MEDS
export DATASET_DOWNLOAD_USERNAME=$PHYSIONET_USERNAME
export DATASET_DOWNLOAD_PASSWORD=$PHYSIONET_PASSWORD
MEDS_extract-MIMIC_IV root_output_dir=$ROOT_OUTPUT_DIR

When you run this, the program will:

  1. Download the needed raw MIMIC files for the currently supported version into $ROOT_OUTPUT_DIR/raw_input.
  2. Perform initial, pre-MEDS processing on the raw MIMIC files, saving the results in $ROOT_OUTPUT_DIR/pre_MEDS.
  3. Construct the final MEDS cohort, and save it to $ROOT_OUTPUT_DIR/MEDS_cohort.

You can also specify the target directories more directly, with

export DATASET_DOWNLOAD_USERNAME=$PHYSIONET_USERNAME
export DATASET_DOWNLOAD_PASSWORD=$PHYSIONET_PASSWORD
MEDS_extract-MIMIC_IV raw_input_dir=$RAW_INPUT_DIR pre_MEDS_dir=$PRE_MEDS_DIR MEDS_cohort_dir=$MEDS_COHORT_DIR

Examples and More Info:

You can run MEDS_extract-MIMIC_IV --help for more information on the arguments and options. You can also run

MEDS_extract-MIMIC_IV root_output_dir=$ROOT_OUTPUT_DIR do_demo=True

to run the entire pipeline over the publicly available, fully open MIMIC-IV demo dataset.

Expected runtime and compute needs

This pipeline can be successfully run over the full MIMIC-IV on a 5-core machine leveraging around 165GB of memory in approximately 7 hours (note this time includes the time to download all of the MIMIC-IV files as well, and this test was run on a machine with poor network transfer speeds and without any parallelization applied to the transformation steps, so these speeds can likely be greatly increased). The output folder of data is 9.8 GB. This can be reduced significantly as well as intermediate files not necessary for the final MEDS dataset are retained in additional folders. See this github issue for tracking on ensuring these directories are automatically cleaned up in the future.

Project links

Key dates

PyPI data
Data sourced directly from PyPI's database.
  • Released:
Newer release available (0.2.0)

1 maintainer

PyPI data
Data sourced directly from PyPI's database.
Avatar for mmd_pypi from gravatar.com mmd_pypi

Credits

PyPI data
Data sourced directly from PyPI's database.

Author: Matthew McDermott

License

MIT License

Requires

Python >=3.11

Provides Extra

dev tests local-parallelism slurm-parallelism

Classifiers

Report project as malware

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

mimic_iv_meds-0.0.3.1.tar.gz (20.6 kB view details)

Uploaded Source

Built Distribution

If you're not sure about the file name format, learn more about wheel file names.

mimic_iv_meds-0.0.3.1-py3-none-any.whl (16.7 kB view details)

Uploaded Python 3

File details

Details for the file mimic_iv_meds-0.0.3.1.tar.gz.

File metadata

  • Download URL: mimic_iv_meds-0.0.3.1.tar.gz
  • Upload date:
  • Size: 20.6 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for mimic_iv_meds-0.0.3.1.tar.gz
Algorithm Hash digest
SHA256 93b9eb7415fc3ed28df95378fa407223adccf6acae8a36bf1c10da5bdb26509f
MD5 e384355584f4b082d9c6b619f4dc08c7
BLAKE2b-256 cae54ab62549d7a54051bf2866037bb051bd104faa5dcb478829a3a3599885b8

See more details on using hashes here.

Provenance

The following attestation bundles were made for mimic_iv_meds-0.0.3.1.tar.gz:

Publisher: python-build.yaml on Medical-Event-Data-Standard/MIMIC_IV_MEDS

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file mimic_iv_meds-0.0.3.1-py3-none-any.whl.

File metadata

File hashes

Hashes for mimic_iv_meds-0.0.3.1-py3-none-any.whl
Algorithm Hash digest
SHA256 e53271ca2efc538ec7a693331413017f06989417eef4f34167985ce5cd714bd1
MD5 1dc468c2098f16d6c1f102f64ba69388
BLAKE2b-256 83e92a3f5cd991189a3c2de297c9e50f7e57da2eda810028761b2208f2dba177

See more details on using hashes here.

Provenance

The following attestation bundles were made for mimic_iv_meds-0.0.3.1-py3-none-any.whl:

Publisher: python-build.yaml on Medical-Event-Data-Standard/MIMIC_IV_MEDS

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Release history Release notifications | RSS feed

0.2.0

2 files

0.1.3

2 files

0.1.2

2 files

0.1.1.1

2 files

0.1.1

2 files

0.1.0.1

2 files

0.1.0

2 files

0.0.7

2 files

0.0.6

2 files

0.0.5

2 files

0.0.4.1

2 files

0.0.4

2 files

This release

0.0.3.1 This release

2 files

0.0.3

2 files

0.0.2.1

2 files

0.0.2

2 files

0.0.1.1

2 files

0.0.1

2 files

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page