A comprehensive benchmark for real-world Sentinel-2 imagery super-resolution

GitHub: https://github.com/ESAOpenSR/opensr-test

Documentation: https://opensr-test.readthedocs.io/

PyPI: https://pypi.org/project/opensr-test/

Paper: Coming soon!

Overview

In remote sensing, Super-Resolution goal is to artificially increase the ground sampling distance (GSD) of a low-resolution (LR) image. However, assessing the GSD is not a trivial task. In literature, we find two main points of concern.

Firstly, most models are tested on synthetic data, raising doubts about their real-world applicability and performance. Secondly, traditional evaluation metrics such as PSNR, LPIPS, and SSIM are not designed for assessing SR performance. These metrics fall short, especially in conditions involving changes in reflectance intensity or spatial misalignments - scenarios that are frequently encountered when working with real-world data.

To address these challenges, 'opensr-test' provides a fresh and fair approach for SR benchmark. On one front, we provide three datasets that were carefully crafted to minimize spatial and spectral misalignment. On the other front, we provide a comprehensive set of metrics grouped into three categories: spectral and spatial consistency, the distance between the SR, HR and LR images, and overall correctness. These metrics are designed to assess the performance of SR models in real-world scenarios.

How to use

The example below shows how to use opensr-test to benchmark your SR model.

import torch
import opensr_test

lr = torch.rand(4, 64, 64)
hr = torch.rand(4, 256, 256)
sr = torch.rand(4, 256, 256)

metrics = opensr_test.Metrics()
metrics.setup(lr=lr, sr=sr, hr=hr)
metrics.compute() # Compute the metrics
metrics.summary() # Print the metrics

Installation

Install the latest version from PyPI:

pip install opensr-test

Upgrade opensr-test by running:

pip install -U opensr-test

Install the latest dev version from GitHub by running:

pip install git+https://github.com/ESAOpenSR/opensr-test

Examples

The following examples show how to use opensr-test to benchmark your SR model.

• Use opensr-test with a custom SR TensorFlow model

• Use opensr-test with a custom PyTorch model

• Use opensr-test with a diffuser model

• Use opensr-test to test a multi-image SR model (Satlas Super Resolution)

• Use opensr-test to create a animated GIF of the SR correctness

Visualizations

The opensr-test package provides a set of visualizations to help you understand the performance of your SR model.

import torch
import opensr_test
from super_image import HanModel
import matplotlib.pyplot as plt

# Define the SR model
srmodel = HanModel.from_pretrained('eugenesiow/han', scale=4)
srmodel.eval()

# Load the data
lr, hr, landuse, parameters = opensr_test.load("spot").values()

# Define the benchmark experiment
metrics = opensr_test.Metrics()

# Define the image to be tested
idx = 0
lr_img = torch.from_numpy(lr[idx, 0:3])
hr_img = torch.from_numpy(hr[idx, 0:3])
land_img = torch.from_numpy(landuse[idx, 0])
with torch.no_grad():
    sr_img = srmodel(lr_img[None]).squeeze()

# Compute the metrics
metrics.setup(lr=lr_img, sr=sr_img, hr=hr_img, landuse=land_img)
metrics.compute()

Now, we can visualize the results using the opensr_test.visualize module. fDisplay the triplets LR, SR and HR images:

metrics.plot_triplets()

Display the quadruplets LR, SR, HR and landuse images:

metrics.plot_quadruplets()

Display the matching points between the LR and SR images:

metrics.plot_spatial_matches()

Display a summary of all the metrics:

metrics.plot_summary()

Display the correctness of the SR image:

metrics.plot_tc()

Deeper understanding

Explore the API section for more details about personalizing your benchmark experiments.

Citation

If you use opensr-test in your research, please cite our paper:

Coming soon!

Acknowledgements

This work was make with the support of the European Space Agency (ESA) under the project “Explainable AI: application to trustworthy super-resolution (OpenSR)”. Cesar Aybar acknowledges support by the National Council of Science, Technology, and Technological Innovation (CONCYTEC, Peru) through the “PROYECTOS DE INVESTIGACIÓN BÁSICA – 2023-01” program with contract number PE501083135-2023-PROCIENCIA.