ultrabench 1.0.1

A standardized benchmark of ultrasound image analysis tasks.

UltraBench

UltraBench is a standardized benchmark of 14 different convex ultrasound image classification and semantic segmentation tasks (7 classification and 7 segmentation tasks) drawn from the 10 different publicly available datasets covering 11 regions of the body. This repository contains the processing scripts to transform raw datasets from their original sources into a standardized format for training and evaluating machine learning models.

This benchmark was published alongside the article Revisiting Data Augmentation for Ultrasound Images. If you use this benchmark in your research please cite this article and the articles for each of the included datasets. Instructions for how to do so are provided at the end.

Motivation

A common limitation of studies on ultrasound image analysis using machine learning is a lack of evaluations across multiple domains. However, this is made difficult by a lack of ultrasound image analysis tasks in existing medical image analysis benchmarks, such as MedMNIST, MedSegBench and the Medical Segmentation Decathlon. To address this issue, we created UltraBench. Our aim is to make it easier for researchers by standardizing the preprocessing of many publicly available ultrasound datasets and providing tools (e.g., dataset implementations for common deep learning frameworks) that (a) reduce the effort required to use them and (b) make it easier for researchers to compare results between papers.

Installation

Install UltraBench using:

pip install ultrabench

Getting Started

UltraBench is packaged as a commandline application. After installing, run ultrabench --help to get started.

For more information, checkout the Documentation on the wiki!

Current Datasets & Tasks

The following table contains the list of supported datasets and tasks. Click on the links to checkout the appropriate sections of the documentation or on the article/repository links to visit the original publications. We do not have the rights to republish the datasets, but links to the original articles and datasets are included here and in the Documentation.

Dataset	Region	Classification Tasks	Segmentation Tasks
Annotated Ultrasound Liver (article, data)	Liver	Liver mass classification	Liver segmentation, Liver mass segmentation
Butterfly (data)	Multi-region	Region classification
CAMUS (article, data)	Heart	Image quality classification	Cardiac region segmentation
Dataset of B-mode fatty liver ultrasound images (article, data)	Liver	NFLD classification
Gallbladder Cancer Ultrasound (GBCU) (article, data)	Gallbladder	Gallbladder tumor classification
Multi-Modality Ovarian Tumor Ultrasound (MMOTU) (article, data)	Ovaries	Ovarian tumor classifcation	Ovarian tumor segmentation
Open Kidney Ultrasound (article, data)	Kidney		Kidney capsule segmentation
Point-of-care Ultrasound (POCUS) (article, data)	Lung	COVID-19 classification
PSFHS (article, data)	Fetus		Fetal head and pubic symphysis segmentation
Stanford Thyroid (data)	Thyroid		Thyroid nodule segmentation

Contributing

We would love for this benchmark to grow and flourish into a resource that anyone with an interest in machine learning for ultrasound analysis can pick up and use quickly and easily. Any help addressing bugs, contributing new datasets or tasks, or any other improvements are welcome and appreciated! I only ask that you respect the community guidelines laid out in the CODE_OF_CONDUCT.md. For more information on how to contribute, checkout out the Documentation.

UltraBench is built using uv. For the best experience, you can use the development container configuration (see DEVCONTAINER.md) to setup an isolated development environment with uv and all the necessary dependencies already installed. Otherwise, you can install uv and then following the commands below. To ensure that your code meets the style guidelines etc., this also installs and initializes the optional development dependencies.

uv sync
uv sync --group examples  # Install the dependencies for the examples, if required
source .venv/bin/activate
nbstripout --install
pre-commit install

Questions

If you have any questions, please open an issue or contact us via email.

How to Cite UltraBench

If you use UltraBench in your research, please cite our article Revisiting Data Augmentation for Ultrasound Images and the articles for each of the datasets. This ensures that the work of the authors of the original datasets is properly acknowledged, helps more people find the benchmark and encourages us to continue maintaining and improving it!

@article{
  tupper2025,
  title={Revisiting Data Augmentation for Ultrasound Images},
  author={Adam Tupper and Christian Gagn{\'e}},
  journal={Transactions on Machine Learning Research},
  issn={2835-8856},
  year={2025},
  url={https://openreview.net/forum?id=iGcxlTLIL5},
  note={}
}