ECGEN-Eval

Evaluation metrics and interactive HTML reports for synthetic ECG quality assessment. Part of the ECGEN research suite (ECGEN-FM, ECGEN-VAE, Pulse2Pulse).

📖 Documentation: https://vajira.info/ECGEN-Eval/

Installation

Install the latest release from PyPI:

pip install ecgen-eval

Optional extras:

pip install "ecgen-eval[features]"        # clinical feature extraction (neurokit2 + scipy)
pip install "ecgen-eval[fed]"             # Fréchet ECG Distance (torch, transformers, huggingface_hub)
pip install "ecgen-eval[features,fed]"    # both

Other optional packages:

pip install wfdb           # for loading PTB-XL / WFDB records
pip install scipy          # for Welch PSD, resampling, and R-peak detection (recommended)

To work on ECGEN-Eval itself, install from source instead; see the contributing guide.

Quick start

import numpy as np
from ecgen_eval import load_ecg_npy, AVAILABLE_METRICS

# Load model outputs (shape: N × L × T)
real  = load_ecg_npy("real_ecgs.npy",  label="PTB-XL")
synth = load_ecg_npy("synth_ecgs.npy", label="ECGEN-FM")

# Compute a metric
mmd = AVAILABLE_METRICS["mmd"]()
result = mmd.compute(real.data, synth.data, lead_names=real.lead_names)
print(f"MMD = {result.score:.6f}")
print(result.per_lead_scores)

Data loaders

load_ecg_folder auto-detects the format based on file extensions.

Metrics

All metrics are available via AVAILABLE_METRICS["key"]() and return a MetricResult with .score, .per_lead_scores, and .extra.

Original metrics

New metrics (literature review 2018 – 2025)

Feature Extraction

Extract 10 clinical ECG features per recording and compare their distributions between real and synthetic datasets.

from ecgen_eval import load_ecg_npy
from ecgen_eval.features import extract_features

real  = load_ecg_npy("real_ecgs.npy",  label="PTB-XL")
synth = load_ecg_npy("synth_ecgs.npy", label="ECGEN-FM")

# Returns (beat_df, summary_df)
_, real_feat  = extract_features(real)
_, synth_feat = extract_features(synth)

print(real_feat[["rr_mean_ms", "hr_bpm", "qrs_duration_ms",
                  "qt_interval_ms", "hrv_sdnn_ms"]].describe())

Extracted features (one value per recording):

2D and 3D distribution plots

from ecgen_eval.visualization.feature_plots import (
    plot_feature_2d,
    plot_feature_pair_grid,
    plot_feature_3d,
    plot_feature_triple_grid,
)

# Single 2D scatter + KDE contour
fig = plot_feature_2d(real_feat, synth_feat, "rr_mean_ms", "hr_bpm")
fig.show()

# Grid of default feature pairs
fig = plot_feature_pair_grid(real_feat, synth_feat)
fig.show()

# Interactive 3D scatter
fig = plot_feature_3d(real_feat, synth_feat, "rr_mean_ms", "hr_bpm", "qrs_duration_ms")
fig.show()

CLI

# Run all metrics (except fed) and generate interactive HTML report
ecgen-eval eval \
  --real   /path/to/real_ecgs.npy \
  --synthetic /path/to/model1_ecgs.npy \
  --synthetic /path/to/model2_ecgs.npy \
  --output report.html

# Include clinical feature distributions in the report (requires neurokit2)
ecgen-eval eval \
  --real /path/to/real_ecgs.npy \
  --synthetic /path/to/synth_ecgs.npy \
  --feature-plots \
  --output report.html

# Extract features only — save summary CSV
ecgen-eval features \
  --real /path/to/real_ecgs.npy \
  --synthetic /path/to/synth_ecgs.npy \
  --output features.csv

# List available metrics
ecgen-eval list-metrics

# ECG waveform comparison only (no metrics)
ecgen-eval visualize \
  --real /path/to/real.npy \
  --synthetic /path/to/synth.npy \
  --compare-mode overlay

Key options for eval:

• -m mmd -m prd -m psd — run a subset of metrics (repeat -m per metric; default: all except fed)
• --feature-plots — add clinical feature extraction + 2D/3D distribution plots
• --fs 500 — sampling frequency (also passed to psd, spectral_entropy, hrv, heart_rate)
• --n-samples 5 — ECG samples shown in gallery
• --offline — embed Plotly.js for offline viewing
• --max-samples 500 — cap loaded recordings per folder

Report

The HTML report includes:

• Dataset summary table
• Interactive ECG gallery (overlay / side-by-side / grid)
• Per-metric: score table, per-lead bar chart, PSD overlay (PSD metric), violin plot (DTW)
• Radar chart summarising all metrics
• Metrics summary table with Download CSV button
• Clinical Feature Summary table (mean ± std, real vs synthetic) — with --feature-plots
• 2D feature distribution plots (scatter + KDE contours) — with --feature-plots
• 3D feature distribution plots (interactive rotation) — with --feature-plots
• References / citations

Running tests

pytest tests/ -v

Project structure

ecgen_eval/
├── data/
│   ├── dataset.py          ECGDataset container
│   └── loader.py           npy / WFDB / CSV loaders
├── features/               Clinical ECG feature extraction (optional: neurokit2)
│   ├── __init__.py         Exports FeatureExtractor, extract_features
│   ├── extractor.py        FeatureExtractor class → (beat_df, summary_df)
│   ├── qrs_detection.py    R-peak detection (neurokit2 primary, scipy fallback)
│   ├── interval_features.py  RR, HR, PR, QRS, QT, QTc, P-wave duration
│   ├── amplitude_features.py ST deviation, T-wave amplitude
│   ├── hrv_features.py     SDNN, RMSSD, pNN50
│   └── utils.py            Bandpass filter, peak helpers
├── metrics/
│   ├── base.py             BaseMetric + MetricResult
│   ├── mmd.py              MMD
│   ├── dtw.py              DTW
│   ├── prd.py              PRD
│   ├── psd.py              PSD divergence
│   ├── fd.py               Fréchet Distance
│   ├── psnr.py             PSNR
│   ├── ssim.py             1-D SSIM
│   ├── swd.py              Sliced Wasserstein Distance
│   ├── mae.py              MAE / RMSE
│   ├── hrv.py              HRV statistics (SDNN, rMSSD, pNN50)
│   ├── pr_dist.py          Distribution Precision & Recall
│   ├── sqi.py              Signal Quality Index
│   ├── nn_dist.py          Nearest-Neighbour Distance
│   ├── heart_rate.py       Heart Rate Distribution
│   └── spectral_entropy.py Spectral Entropy Divergence
├── visualization/
│   ├── ecg_paper.py        ECG graph-paper figure factory
│   ├── ecg_waveform.py     Waveform comparison plots
│   ├── metric_plots.py     Bar, violin, PSD overlay, radar
│   └── feature_plots.py    2D/3D clinical feature distribution plots
├── report/
│   └── html_report.py      HTML report generator
└── cli.py                  Click CLI entry point (eval, visualize, features, list-metrics)
tests/
├── conftest.py
├── test_dataset.py
├── test_loader.py
├── test_metrics.py
└── test_features.py
docs/
└── ecg_features_review.md  Literature review of the 10 clinical features