medeval 0.1.7

PyTorch-native library for medical imaging evaluation metrics

MedEval

A PyTorch-native library for computing, aggregating, and visualizing evaluation metrics for medical imaging tasks with 2D/3D support, correct physical spacing handling, confidence intervals, and clean APIs (Python + CLI).

Features

• 🏥 Medical Imaging Focus: Designed specifically for medical imaging evaluation tasks
• 📐 2D/3D Support: Handle both 2D and 3D medical images with proper dimension handling
• 📏 Physical Spacing: Correct handling of anisotropic voxels and physical dimensions
• 📊 Confidence Intervals: Bootstrap and jackknife methods for statistical reporting
• 🔄 Interoperability: Works seamlessly with MONAI and torchmetrics
• 💻 Clean APIs: Both Python API and CLI for batch evaluation
• 🚫 No PHI: Designed with privacy in mind, no protected health information handling

Conventions

Input Shape Support

Input Shape	Interpretation	Normalized Shape
(H, W)	2D single image	(1, 1, H, W)
(B, H, W)	2D batch (B≤4)	(B, 1, H, W)
(B, C, H, W)	2D batch+channel	unchanged
(Z, Y, X)	3D single volume (Z>4)	(1, 1, Z, Y, X)
(B, Z, Y, X)	3D batch (Z>4)	(B, 1, Z, Y, X)
(B, C, Z, Y, X)	3D batch+channel	unchanged

Note: For 3-dim inputs, a heuristic distinguishes 2D-batched from 3D-unbatched by the first dimension size (threshold: 4). Use explicit 4D/5D shapes or provide spacing to disambiguate.

Spacing Axis Order

• 2D: (dy, dx) — row spacing, column spacing
• 3D: (dz, dy, dx) — slice spacing, row spacing, column spacing

Surface Metrics

Surface-based metrics (Hausdorff, ASSD, Surface Dice) require physical spacing. Without spacing:

• Default: metrics are skipped (no error, logged as skipped)
• Strict mode: use require_spacing=True to raise ValueError

Installation

# Install from PyPI (recommended)
pip install medeval

# With DICOM support
pip install medeval[dicom]

# Minimal install (metrics only, no IO / plotting)
pip install medeval[minimal]

# For development
pip install -e ".[dev]"

What MedEval Is (and Is Not)

MedEval is an evaluation library, not a training framework.

It assumes you already have model predictions and ground truth labels. MedEval’s responsibility is to:

• Compute correct, spacing-aware medical metrics
• Aggregate results across cases, patients, and strata
• Produce reproducible CSV / JSON artifacts and plots

MedEval deliberately does not:

• Train models
• Tune hyperparameters
• Declare a “best model”

This separation mirrors real clinical research and deployment workflows.

End-to-End Demo (External User Perspective)

This demo reflects how MedEval is intended to be used after a model has already produced predictions.

The demo_external_user/ directory provides a realistic, end-to-end example of how an external user would interact with MedEval using their own data.
It demonstrates data preparation → evaluation → aggregation → visualization using both the Python API and the CLI.

TL;DR (10 lines)

1. python demo_external_user/gen_data.py generates synthetic segmentation + classification data (no PHI).
2. It writes a segmentation manifest with 100 cases, patient IDs, and strata (site/scanner-like).
3. python demo_external_user/run_example.py is the recommended “real user” entry point.
4. It loads manifest_seg.csv, then iterates cases and reads each NIfTI (pred/target) + spacing metadata.
5. It normalizes shapes to (B, C, *spatial) and computes per-case metrics (Dice/IoU/HD95/ASSD/Surface Dice).
6. It saves per-case results to out_example/segmentation_results.csv.
7. It aggregates by strata and saves out_example/segmentation_summary_by_strata.csv.
8. It loads classification probs/labels, computes sample-level metrics + calibration (AUROC/AUPRC/Accuracy/ECE).
9. It aggregates classification to patient-level (grouped AUROC) and saves out_example/classification_results.json.
10. If matplotlib is installed, it also writes plots (Dice hist/box, HD95 hist, ROC, reliability diagram).

What This Demo Simulates

• Multiple patients, cases, and strata (e.g. sites/scanners)
• 3D segmentation with physical voxel spacing
• Binary classification with per-patient aggregation
• Per-case metrics, stratified summaries, and plots

Demo Structure

demo_external_user/
├── gen_data.py              # Generate synthetic medical data + manifests
├── manifest_seg.csv         # Segmentation manifest (100 cases, patients, strata)
├── run_example.py           # Main end-to-end Python workflow (recommended)
├── run_python_api.py        # Minimal direct API usage example
├── run_cli.sh               # CLI-based evaluation example
├── config_seg.yaml          # CLI configuration
├── data/                    # Generated synthetic inputs
└── out_example/             # Outputs: CSVs, JSON, plots

Recommended Entry Point

Run the full example as a real user would:

python demo_external_user/gen_data.py
python demo_external_user/run_example.py

This will:

• Load a segmentation manifest with 100 cases
• Compute per-case metrics (Dice, HD95, ASSD, etc.)
• Aggregate results by strata
• Compute classification metrics (sample-level + patient-level)
• Save CSV/JSON outputs
• Generate plots (Dice distribution, HD95 histogram, ROC, calibration)

Typical Outputs

• segmentation_results.csv — per-case metrics
• segmentation_summary_by_strata.csv — aggregated statistics
• classification_results.json — sample-level and patient-level metrics
• dice_hist.png, dice_by_strata_box.png, hd95_hist.png
• roc_curve.png, reliability_diagram.png

This demo is the best starting point to understand MedEval’s design, APIs, and evaluation philosophy.

Evaluation Scope and Design Philosophy

MedEval evaluates one experiment (one model’s predictions) per run.

It is intentionally designed to be:

• Model-agnostic: MedEval does not know or assume how predictions were generated.
• Metric-neutral: It computes metrics but does not decide which metric “matters most”.
• Decision-neutral: It does not declare a “best model”.

This design reflects real-world medical research and deployment practice, where:

• Different metrics capture different clinical trade-offs.
• Performance may vary across patients, strata (e.g. sites/scanners), or endpoints.
• Model selection and conclusions depend on study-specific goals, not on the evaluation tool itself.

MedEval’s role is to produce reliable, comparable, and statistically sound evaluation artifacts
(CSV/JSON summaries, confidence intervals, plots) that enable downstream analysis.

Multi-Model Comparison (External Analysis Layer)

While MedEval evaluates one model per run, it is explicitly designed to support multi-model comparison at a higher analysis layer.

The recommended workflow is:

1. Run MedEval once per model, using the same evaluation protocol.
2. Store each model’s outputs (CSVs, JSON summaries, plots).
3. Compare models using a separate analysis script or notebook.

Recommended Directory Structure

experiments/
├── model_A/
│   └── out_example/
│       ├── segmentation_results.csv
│       ├── segmentation_summary_by_strata.csv
│       ├── classification_results.json
│       └── plots/
├── model_B/
│   └── out_example/
├── model_C/
│   └── out_example/
└── compare_models.ipynb

In this structure:

• MedEval is responsible only for generating per-model evaluation results.
• compare_models.ipynb (or an equivalent script) performs:
  • Cross-model aggregation
  • Statistical testing
  • Visualization of differences
  • Study-specific conclusions

This separation keeps MedEval reusable, transparent, and scientifically neutral.

This design avoids hidden coupling between models and metrics and ensures that every comparison is explicit, auditable, and study-specific.

Quick Start

Segmentation Evaluation

import torch
from medeval.metrics.segmentation import dice_score, hausdorff_distance_95, compute_segmentation_metrics

# Your predictions and ground truth
pred = torch.rand(1, 1, 64, 64, 64) > 0.5
target = torch.rand(1, 1, 64, 64, 64) > 0.5
spacing = (2.0, 1.0, 1.0)  # Physical spacing in mm (z, y, x)

# Compute individual metrics
dice = dice_score(pred, target, reduction="mean-case")
print(f"Dice: {dice.item():.4f}")

# Compute spacing-aware surface distance
hd95 = hausdorff_distance_95(pred, target, spacing=spacing)
print(f"HD95: {hd95.item():.2f} mm")

# Compute all metrics at once
results = compute_segmentation_metrics(
    pred, target,
    spacing=spacing,
    include_surface=True,
    include_calibration=True,
)

Classification Evaluation

import numpy as np
from medeval.metrics.classification import auroc, compute_classification_metrics

# Your predictions and labels
probs = np.random.rand(1000)
labels = (np.random.rand(1000) > 0.7).astype(int)

# Compute AUROC with DeLong confidence interval
score, ci_lower, ci_upper = auroc(probs, labels, compute_ci=True)
print(f"AUROC: {score:.4f} [{ci_lower:.4f}, {ci_upper:.4f}]")

# Comprehensive metrics with calibration
results = compute_classification_metrics(
    probs, labels,
    compute_ci=True,
    include_calibration=True,
)

Detection Evaluation

import torch
from medeval.metrics.detection import mean_average_precision, froc

# Predicted and ground truth boxes
pred_boxes = [torch.tensor([[10, 10, 50, 50], [60, 60, 100, 100]])]
pred_scores = [torch.tensor([0.9, 0.7])]
pred_labels = [torch.tensor([0, 1])]
target_boxes = [torch.tensor([[10, 10, 50, 50], [60, 60, 100, 100]])]
target_labels = [torch.tensor([0, 1])]

# Compute mAP
results = mean_average_precision(
    pred_boxes, pred_scores, pred_labels,
    target_boxes, target_labels,
)
print(f"mAP@[.50:.95]: {results['mAP@[.50:.95]']:.4f}")

Registration Evaluation

import numpy as np
from medeval.metrics.registration import target_registration_error, normalized_mutual_information

# Landmark-based evaluation
pred_landmarks = np.array([[10, 20, 30], [40, 50, 60]])
target_landmarks = np.array([[11, 21, 31], [41, 51, 61]])
spacing = (1.0, 1.0, 2.0)

tre_results = target_registration_error(pred_landmarks, target_landmarks, spacing=spacing)
print(f"TRE: {tre_results['mean']:.2f} mm (95th: {tre_results['95th_percentile']:.2f} mm)")

CLI Usage

# Evaluate segmentation from manifest
medeval evaluate --manifest data/manifest.csv --task segmentation --output results/

# With configuration file
medeval evaluate --manifest data/manifest.csv --config config.yaml -v

Manifest Format

prediction,target,spacing,patient_id,strata
/data/pred1.nii.gz,/data/gt1.nii.gz,"1.0,1.0,2.0",patient_001,site_A
/data/pred2.nii.gz,/data/gt2.nii.gz,"1.0,1.0,2.0",patient_002,site_B

Supported Metrics

Segmentation

Metric	Description
Dice Score (F1)	Overlap-based similarity
Jaccard Index (IoU)	Intersection over union
Precision/Recall	True positive rates
Volumetric Similarity	Volume-based similarity
Hausdorff Distance	Maximum surface distance
HD95	95th percentile Hausdorff
ASSD	Average symmetric surface distance
Surface Dice	Surface overlap at tolerance

Classification

Metric	Description
AUROC	Area under ROC curve (with DeLong CI)
AUPRC	Area under precision-recall curve
Accuracy	Overall correctness
Balanced Accuracy	Class-balanced accuracy
F1, MCC, Cohen's κ	Agreement metrics
ECE, AECE, TACE	Calibration errors
Brier Score	Probabilistic accuracy

Detection

Metric	Description
IoU	Box intersection over union (2D/3D)
mAP@[.50:.95]	Mean average precision
FROC	Free-response ROC
Average Recall	Detection recall

Registration

Metric	Description
TRE	Target registration error
NMI	Normalized mutual information
NCC	Normalized cross-correlation
Jacobian	Deformation field quality

Visualization

from medeval.vis import plot_roc_curve, plot_reliability_diagram, plot_segmentation_overlay

# Plot ROC curve
fig, ax = plot_roc_curve(fpr, tpr, auc=auc_score, title="Model Performance")

# Plot calibration diagram
fig, ax = plot_reliability_diagram(
    bin_centers, accuracies, confidences, counts,
    ece=ece_value,
)

# Plot segmentation overlay
fig, ax = plot_segmentation_overlay(image, mask=ground_truth, prediction=pred)

Statistical Reporting

MedEval provides comprehensive statistical reporting:

from medeval.core.aggregate import aggregate_metrics, stratified_aggregate

# Aggregate with bootstrap CI
results = aggregate_metrics(
    {"dice": dice_scores, "hd95": hd95_scores},
    method="mean",
    compute_ci=True,
    confidence=0.95,
    n_bootstrap=1000,
)

# Stratified by site/scanner
stratified = stratified_aggregate(
    {"dice": dice_scores},
    strata=site_labels,
    compute_ci=True,
)

MONAI Integration

from medeval.core.interop import MedEvalMetricWrapper
from medeval.metrics.segmentation import dice_score

# Use medeval metrics with torchmetrics-style interface
metric = MedEvalMetricWrapper(dice_score, reduction="none")
metric.update(preds, targets)
result = metric.compute()

Documentation

• API Reference
• Metric Definitions
• CLI Reference
• Tutorials

Examples

See the examples/ directory for:

• segmentation_demo.py - Comprehensive segmentation evaluation
• classification_demo.py - Classification with calibration analysis
• cli_usage.sh - Command-line interface examples

Development

# Clone and install
git clone https://github.com/your-repo/medeval.git
cd medeval
pip install -e ".[dev]"

# Install pre-commit hooks
pre-commit install

# Run tests
pytest

# Run tests with coverage
pytest --cov=medeval --cov-report=html

# Build docs
mkdocs serve

Contributing

We welcome contributions! Please see CONTRIBUTING.md for guidelines.

Citation

If you use MedEval in your research, please cite:

@software{medeval2024,
  title = {MedEval: Medical Imaging Evaluation Metrics},
  year = {2024},
  url = {https://github.com/Eisenherz00/medeval}
}

License

Apache-2.0. See LICENSE for details.