OMOP-MEDS 0.0.10

An ETL to convert OMOP data to the MEDS format.

MEDS OMOP ETL with MEDS-Transforms

An ETL pipeline for transforming OMOP datasets into the MEDS format using the MEDS-Transforms library. Thanks to the developers of the first OMOP MEDS ETL, from which we took inspiration, which can be found here: https://github.com/Medical-Event-Data-Standard/meds_etl. We currently support OMOP 5.3 and 5.4 datasets.

pip install OMOP_MEDS
OMOP_MEDS root_output_dir=$ROOT_OUTPUT_DIR

To try with the MIMIC-IV OMOP demo dataset, you can run:

OMOP_MEDS root_output_dir=/path/to/your/output do_download=True ++do_demo=True

Example config for an OMOP dataset:

dataset_name: MIMIC_IV_OMOP
raw_dataset_version: 1.0
omop_version: 5.3

urls:
  dataset:
    - https://physionet.org/content/mimic-iv-demo-omop/0.9/
    - url: EXAMPLE_CONTROLLED_URL
      username: ${oc.env:DATASET_DOWNLOAD_USERNAME}
      password: ${oc.env:DATASET_DOWNLOAD_PASSWORD}
  demo:
    - https://physionet.org/content/mimic-iv-demo-omop/0.9/
  common:
    - EXAMPLE_SHARED_URL # Often used for shared metadata files

Pre-MEDS settings

The following settings can be used to configure the pre-MEDS steps.

OMOP_MEDS \
	root_output_dir=/sc/arion/projects/hpims-hpi/projects/foundation_models_ehr/cohorts/meds_debug/small_demo \
	raw_input_dir=/sc/arion/projects/hpims-hpi/projects/foundation_models_ehr/cohorts/full_omop \
	do_download=False ++do_overwrite=True ++limit_subjects=50

• root_output_dir: Set the root output directory.
• raw_input_dir: Path to the raw input directory.
• do_download: Set to False to skip downloading the dataset.
• ++do_overwrite: Set to True to overwrite existing files.
• ++limit_subjects: Limit the number of subjects to process.

MEDS-transforms settings

If you want to convert a large dataset, you can use parallelization with MEDS-transforms (the MEDS-transformation step that takes the longest).

Using local parallelization with the hydra-joblib-launcher package, you can set the number of workers:

pip install hydra-joblib-launcher --upgrade

Then, you can set the number of workers as environment variable:

export N_WORKERS=16

Moreover, you can set the number of subjects per shard to balance the parallelization overhead based on how many subjects you have in your dataset:

export N_SUBJECTS_PER_SHARD=1000

Citation

If you use this dataset, please use the citation link in Github.