Segmentation Metrics

Volumetric binary mask segmentation accuracy metrics

Scope

A small package for assessing the accuracy of binary and multi-label segmentations. There are lots of metrics that can be used to compare how close two segmentations are, here voxel overlap, surface and volume based metrics are all calculated at once and returned either as individual metrics, a dictionary or a Pandas DataFrame.

This package supports multi-label segmentation masks, in which case the metrics are calculated for each label and then averaged across labels to give a single score for each metric. The volume based metrics are summed across labels rather than averaged. By default, a saftey limit of 10 labels is applied and an error is raised if more than 10 distinct labels are found (e.g. in image is input rather than a labeled mask). If your images have more than 10 labels, set many_labels=True when initialising the SegmentationMetrics object to bypass this check and allow all labels to be included in the metrics calculation.

The surface based metrics in this package are calculated using code from deepmind's surface-distance repository, however as this is not available as a PyPI package, the code has been included as a submodule here.

Installation

Segmentation Metrics is available on PyPI and can be installed using pip:

pip install segmentationmetrics

or if you're a conda user:

conda install segmentationmetrics -c conda-forge

Alternatively, you can install from source:

git clone https://github.com/alexdaniel654/Segmentation_Metrics.git  
cd Segmentation_Metrics
pip install -e .

Calculated Metrics

Voxel overlap based metrics

• Dice Score/F1 Score
• Jaccard Score
• Sensitivity
• Specificity
• Precision
• Accuracy

Surface based metrics

• Mean Surface Distance (in mm) - The symmetric mean surface distance is returned by default i.e. the mean of the distance from surface A to surface B and surface B to surface A.
• Hausdorff Distance (in mm) - Computes the robust distance based on the percentile of distances rather than the maximum distance.

Volume based metrics

• Volume Difference (in millilitres)

Example Usage

import nibabel as nib # Package for reading MRI data
import segmentationmetrics as sm

img_manual = nib.load('mask_manually_segmented.nii.gz') # Load manually generated ground truth mask
img_automatic = nib.load('mask_automatically_segmented.nii.gz') # Load automatically generated mask

# Get voxel data from image object
mask_manual = img_manual.get_fdata()
mask_automatic = img_automatic.get_fdata()

# Get zoom from header
zoom = img_manual.header.get_zooms()

# Generate metrics
metrics = sm.SegmentationMetrics(mask_automatic, mask_manual, zoom)

# Print the dice score
print(f'The Dice score is {metrics.dice:.2f}')

The Dice score is 0.85

# Get and print a DataFrame containing all the scores for this mask pair
df = metrics.get_df()
print(df)

                                      Metric       Score
dice                                    Dice    0.844512
jaccard                              Jaccard    0.730870
sensitivity                      Sensitivity    0.732352
specificity                      Specificity    0.999926
precision                          Precision    0.997239
accuracy                            Accuracy    0.990492
mean_surface_distance  Mean Surface Distance    1.459697
hausdorff_distance        Hausdorff Distance    7.027224
volume_difference          Volume Difference -107.212906
true_volume                      True Volume  403.632624
predicted_volume            Predicted Volume  296.419718

# As above but with asymmetric mean surface distance and Hausdorff distance defined by the 99th percentil rather than the 95th percentile.
metrics = sm.SegmentationMetrics(mask_automatic, mask_manual, zoom, symmetric=False, percentile=99)
df = metrics.get_df()
print(df)

                                      Metric                                    Score
dice                                    Dice                                 0.844512
jaccard                              Jaccard                                  0.73087
sensitivity                      Sensitivity                                 0.732352
specificity                      Specificity                                 0.999926
precision                          Precision                                 0.997239
accuracy                            Accuracy                                 0.990492
mean_surface_distance  Mean Surface Distance  (1.6603182056644057, 1.259075931110695)
hausdorff_distance        Hausdorff Distance                                 9.335755
volume_difference          Volume Difference                              -107.212906
true_volume                      True Volume                               403.632624
predicted_volume            Predicted Volume                               296.419718