A collection of metrics for scene graph generation
Project description
SGBench: A Review and Efficient Implementation of Scene Graph Generation Metrics
Published at CVPR 2024, Scene Graphs and Graph Representation Learning Workshop.
For more information, visit our project page.
Installation
# for the full package (includes dependencies to image loading libraries)
pip install sgbench[all]
# for the light package (depends only on NumPy, but supports no IO)
pip install sgbench
We strive to make this package compatible with any scene graph generation codebase out there. If you should have any issues or version conflicts when installing SGBench, please file an issue on GitHub and we will look into it.
Dependencies
- NumPy - For faster array operations
- Pillow - To load ground truth PNG files
- tifffile - To open TIFF files
- imagecodecs - To support compression of TIFF files
If you choose to install SGBench without [all]
, only NumPy is required.
Usage
You can use SGBench from the command line or as a Python package.
Python
import sgbench
# for a single image
mapping = sgbench.match_instances(gt_masks, gt_labels, pred_masks, pred_labels)
matched_triplets = sgbench.remap_triplets(mapping, pred_triplets)
mR50 = sgbench.recall(
k=50,
mean=True,
gt_triplets=gt_triplets,
matched_triplets=matched_triplets,
num_predicates=56,
)
pr = sgbench.pair_recall(
k=20,
gt_triplets=gt_triplets,
matched_triplets=matched_triplets,
num_predicates=56,
)
Command Line Interface
The CLI only works if SGBench was installed with pip install sgbench[all]
.
sgbench --help
SGBench File Format
When evaluating using the command line interface, SGBench requires a specific file format. For the ground truth, SGBench uses the same file format as OpenPSG. If you are only using bounding boxes, just omit the PNG files and the segmentation mask related keys.
For the prediction file, SGBench uses its own format as described in the following.
SGG With Bounding Boxes
If your model returns bounding boxes, SGBench expects a JSON file in the following format:
{
"version": 1,
"images": [
{
// id of the ground truth image (must match image_id in ground truth file)
"id": "12345",
// bounding boxes are defined as pixels in xyxy format
"bboxes": [
[23.1, 50, 100, 120.0],
// ...
],
// categories are the class ids of the respective bounding boxes
"categories": [3, 51, 100, 23],
// triplets are encoded as subject id, object id, predicate
// ordering matters! the first triplet is the most confident one
"triplets": [
[3, 5, 2],
[12, 0, 1],
// ...
],
},
// ...
]
}
SGG With Segmentation Masks
If your model returns segmentation masks, SGBench expects a ZIP file that contains a JSON file and multiple TIFF files for the segmentation masks.
Each TIFF file stores the segmentation masks for a single image. Use a separate layer for every mask. You can check sgbench.io.write_layered_seg_mask
for an example of how to write the TIFF files.
The JSON file must be in the following format:
{
"version": 1,
"images": [
{
// id of the ground truth image (must match image_id in ground truth file)
"id": "12345",
// name of the corresponding TIFF file
"seg_filename": "seg12345.tiff",
// categories stores the class ids for each layer in the corresponding TIFF file
"categories": [3, 51, 100, 23],
// triplets are encoded as subject id, object id, predicate
// ordering matters! the first triplet is the most confident one
"triplets": [
[3, 5, 2],
[12, 0, 1],
// ...
],
},
// ...
]
}
Citation
If you find this work useful, please consider citing our paper:
@misc{lorenz2024sgbench,
title={A Review and Efficient Implementation of Scene Graph Generation Metrics},
author={Julian Lorenz and Robin Schön and Katja Ludwig and Rainer Lienhart},
year={2024},
eprint={2404.09616},
archivePrefix={arXiv},
primaryClass={cs.CV}
}
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.