Skip to main content

A collection of metrics for scene graph generation

Project description

SGBench: A Review and Efficient Implementation of Scene Graph Generation Metrics

Published at CVPR 2024, Scene Graphs and Graph Representation Learning Workshop.

For more information, visit our project page.

Installation

# for the full package (includes dependencies to image loading libraries)
pip install sgbench[all]
# for the light package (depends only on NumPy, but supports no IO)
pip install sgbench

We strive to make this package compatible with any scene graph generation codebase out there. If you should have any issues or version conflicts when installing SGBench, please file an issue on GitHub and we will look into it.

Dependencies

  • NumPy - For faster array operations
  • Pillow - To load ground truth PNG files
  • tifffile - To open TIFF files
  • imagecodecs - To support compression of TIFF files

If you choose to install SGBench without [all], only NumPy is required.

Usage

You can use SGBench from the command line or as a Python package.

Python

import sgbench
# for a single image
mapping = sgbench.match_instances(gt_masks, gt_labels, pred_masks, pred_labels)
matched_triplets = sgbench.remap_triplets(mapping, pred_triplets)
mR50 = sgbench.recall(
    k=50,
    mean=True,
    gt_triplets=gt_triplets,
    matched_triplets=matched_triplets,
    num_predicates=56,
)
pr = sgbench.pair_recall(
    k=20,
    gt_triplets=gt_triplets,
    matched_triplets=matched_triplets,
    num_predicates=56,
)

Command Line Interface

The CLI only works if SGBench was installed with pip install sgbench[all].

sgbench --help

SGBench File Format

When evaluating using the command line interface, SGBench requires a specific file format. For the ground truth, SGBench uses the same file format as OpenPSG. If you are only using bounding boxes, just omit the PNG files and the segmentation mask related keys.

For the prediction file, SGBench uses its own format as described in the following.

SGG With Bounding Boxes

If your model returns bounding boxes, SGBench expects a JSON file in the following format:

{
    "version": 1,
    "images": [
        {
            // id of the ground truth image (must match image_id in ground truth file)
            "id": "12345",
            // bounding boxes are defined as pixels in xyxy format
            "bboxes": [
                [23.1, 50, 100, 120.0],
                // ...
            ],
            // categories are the class ids of the respective bounding boxes
            "categories": [3, 51, 100, 23],
            // triplets are encoded as subject id, object id, predicate
            // ordering matters! the first triplet is the most confident one
            "triplets": [
                [3, 5, 2],
                [12, 0, 1],
                // ...
            ],
        },
        // ...
    ]
}

SGG With Segmentation Masks

If your model returns segmentation masks, SGBench expects a ZIP file that contains a JSON file and multiple TIFF files for the segmentation masks.

Each TIFF file stores the segmentation masks for a single image. Use a separate layer for every mask. You can check sgbench.io.write_layered_seg_mask for an example of how to write the TIFF files.

The JSON file must be in the following format:

{
    "version": 1,
    "images": [
        {
            // id of the ground truth image (must match image_id in ground truth file)
            "id": "12345",
            // name of the corresponding TIFF file
            "seg_filename": "seg12345.tiff",
            // categories stores the class ids for each layer in the corresponding TIFF file
            "categories": [3, 51, 100, 23],
            // triplets are encoded as subject id, object id, predicate
            // ordering matters! the first triplet is the most confident one
            "triplets": [
                [3, 5, 2],
                [12, 0, 1],
                // ...
            ],
        },
        // ...
    ]
}

Citation

If you find this work useful, please consider citing our paper:

@misc{lorenz2024sgbench,
      title={A Review and Efficient Implementation of Scene Graph Generation Metrics},
      author={Julian Lorenz and Robin Schön and Katja Ludwig and Rainer Lienhart},
      year={2024},
      eprint={2404.09616},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distributions

No source distribution files available for this release.See tutorial on generating distribution archives.

Built Distribution

sgbench-1.0.0-py3-none-any.whl (11.2 kB view details)

Uploaded Python 3

File details

Details for the file sgbench-1.0.0-py3-none-any.whl.

File metadata

  • Download URL: sgbench-1.0.0-py3-none-any.whl
  • Upload date:
  • Size: 11.2 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: poetry/1.8.2 CPython/3.11.9 Darwin/23.5.0

File hashes

Hashes for sgbench-1.0.0-py3-none-any.whl
Algorithm Hash digest
SHA256 d48275c9fef236b29efacdd6afb3b9ad20b0ec8245442015b23dea8de3f60190
MD5 65953cde86a395d93ffc8c0dbf65525c
BLAKE2b-256 a8408cba6478ad086c487a3b681d28473eb3288b84b3c190cfbb77bfac8bcf14

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page