suMEDS

suMEDS (pronounced "summed-s", past tense of "sum" + s) creates a compact code catalog for performing tasks Medical Event Data Standard (MEDS) datasets. MEDS data should be "summed" in this way in a storage-efficient format, without the need to access the full dataset.

It can be seen as a form of "extended metadata," as we sometimes need more information than vanilla MEDS metadata provides, but don't want to expose our complete dataset (either because of privacy concerns, storage capacity, or token use efficiency for LLMs). It joins canonical descriptions and configurable parent, child, and sibling codes to event and unique-subject counts, then masks or removes rare codes before writing Parquet, CSV, or JSON.

[!WARNING] This package is in early development and is human-guided, but AI-agent-generated. It is not yet reviewed for production use. Please report issues and suggestions.

Polars scans, aggregations, joins, and writes stay lazy. This should mean that it could run with little resources on large MEDS datasets.

Install and run

Normal install:

pip install sumeds

Datasets with more than 2³² rows require Polars' 64-bit runtime:

pip install "sumeds[rt64]"

Development

uv sync
uv run suMEDS tests/resources/MIMICIV_DEMO/MEDS_cohort \
  --output code-summary.parquet \
  --min-subjects 20 \
  --split-columns

Or use YAML:

summary:
  per_split: false
  split_columns: true
  partitions: 256
privacy:
  min_subjects: 20
  min_split_subjects: 1
  rare_code_action: bucket
  rare_code_label: __RARE__
  round_counts_to: 5

uv run suMEDS /path/to/MEDS -o summary.parquet -c examples/summary.yaml
# The suffix selects CSV or JSON instead:
uv run suMEDS /path/to/MEDS -o summary.json -c examples/summary.yaml

CLI flags override YAML. The defaults bucket codes seen in fewer than 20 unique subjects and retain exact counts.

Optionally fill missing descriptions, all ancestors, OMOP concept IDs, domains, and standard-concept markers from OHDSI Athena. Child and sibling expansion are opt-in:

uv run suMEDS /path/to/MEDS -o summary.parquet \
  --athena-csv /path/to/athena --child-codes --child-depth 3 --sibling-codes
# Or use PostgreSQL through the installed psql client:
uv run suMEDS /path/to/MEDS -o summary.parquet \
  --athena-postgres postgresql://postgres@localhost/omop

Standalone enrichment avoids rescanning event data:

uv run suMEDS-enrich metadata/codes.parquet -o codes-enriched.parquet \
  --athena-csv /path/to/athena

The standalone command shows phase progress and reports Athena matches and before/after metadata coverage.

Both VOCABULARY//CODE//... and VOCABULARY//VOCABULARY_VERSION//CODE layouts are resolved. See the Athena enrichment guide.

Python API

from sumeds import EnrichmentConfig, SummaryConfig, summarize

path = summarize(
    "/path/to/MEDS",
    "code-summary.parquet",
    SummaryConfig(
        min_subjects=20,
        rare_code_action="bucket",
        enrichment=EnrichmentConfig(csv_dir="/path/to/athena"),
    ),
)

The output format is inferred from .parquet, .csv, or .json (with .jsonl/.ndjson also supported). It preserves all code-metadata extension columns and adds event_count, subject_count, and is_masked. per_split=True emits one row per split. split_columns=True keeps the total row and adds columns such as event_count_train and subject_count_train from subject_splits.parquet. Set min_split_subjects above 1 to suppress rare split-level cells.

Documentation

uv run mkdocs serve
uv run mkdocs build --strict

See the documentation site for the data flow, complete CLI and YAML reference, output schema, Python API, and privacy limitations.

Development

uv run pre-commit run --all-files
uv run pytest
uv run mkdocs build --strict

The included integration test uses tests/resources/MIMICIV_DEMO/MEDS_cohort when present.