INSPIRE-MEDS 0.0.12

An ETL pipeline to extract INSPIRE data into the MEDS format.

INSPIRE-MEDS

This pipeline extracts the INSPIRE dataset (from physionet, https://physionet.org/content/inspire/) into the MEDS format.

Usage:

pip install INSPIRE_MEDS
export DATASET_DOWNLOAD_USERNAME=$PHYSIONET_USERNAME
export DATASET_DOWNLOAD_PASSWORD=$PHYSIONET_PASSWORD
MEDS_extract-INSPIRE root_output_dir=$ROOT_OUTPUT_DIR

When you run this, the program will:

1. Download the needed raw INSPIRE files for the currently supported version into $ROOT_OUTPUT_DIR/raw_input.
2. Perform initial, pre-MEDS processing on the raw INSPIRE files, saving the results in $ROOT_OUTPUT_DIR/pre_MEDS.
3. Construct the final MEDS cohort, and save it to $ROOT_OUTPUT_DIR/MEDS_cohort.

You can also specify the target directories more directly, with

export DATASET_DOWNLOAD_USERNAME=$PHYSIONET_USERNAME
export DATASET_DOWNLOAD_PASSWORD=$PHYSIONET_PASSWORD
MEDS_extract-INSPIRE raw_input_dir=$RAW_INPUT_DIR pre_MEDS_dir=$PRE_MEDS_DIR MEDS_cohort_dir=$MEDS_COHORT_DIR

Examples and More Info:

You can run MEDS_extract-INSPIRE --help for more information on the arguments and options. You can also run

MEDS_extract-INSPIRE root_output_dir=$ROOT_OUTPUT_DIR

to run the entire pipeline.

MEDS-transforms settings

If you want to convert a large dataset, you can use parallelization with MEDS-transforms (the MEDS-transformation step that takes the longest).

Using local parallelization with the hydra-joblib-launcher package, you can set the number of workers:

pip install hydra-joblib-launcher --upgrade

Then, you can set the number of workers as environment variable:

export N_WORKERS=8

Moreover, you can set the number of subjects per shard to balance the parallelization overhead based on how many subjects you have in your dataset:

export N_SUBJECTS_PER_SHARD=100000

The MIMIC-IV OMOP Dataset

We use the demo dataset for MIMIC-IV in the OMOP format, which is a subset of the MIMIC-IV dataset. This dataset downloaded from Physionet does not include the standard dictionary linking definitions but should otherwise be functional

Particularities

• Care site is added to the visit as text
• Add support for care_site table (visit_detail)