A comprehensive benchmark for real-world Sentinel-2 imagery super-resolution
Project description
A comprehensive benchmark for real-world Sentinel-2 imagery super-resolution
GitHub: https://github.com/ESAOpenSR/opensr-test
Documentation: https://esaopensr.github.io/opensr-test
PyPI: https://pypi.org/project/opensr-test/
Paper: Coming soon!
Overview
In remote sensing, Image Super-Resolution (ISR) goal is to improve the ground sampling distance. However, two problems are common in the literature. First, most models are tested on synthetic data, raising doubts about their real-world applicability and performance. Second, traditional evaluation metrics such as PSNR, LPIPS, and SSIM are not designed for assessing ISR performance. These metrics fall short, especially in conditions involving changes in luminance or spatial misalignments - scenarios that are frequently encountered in remote sensing imagery.
To address these challenges, 'opensr-test' provides a fair approach for ISR benchmark. We provide three datasets that were carefully crafted to minimize spatial and spectral misalignment. Besides, 'opensr-test' precisely assesses ISR algorithm performance across three independent metrics groups that measure consistency, synthesis, and correctness.
How to use
The example below shows how to use opensr-test
to benchmark your SR model.
import torch
import opensr_test
lr = torch.rand(4, 64, 64)
hr = torch.rand(4, 256, 256)
sr = torch.rand(4, 256, 256)
metrics = opensr_test.Metrics()
metrics.compute(lr=lr, sr=sr, hr=hr)
Benchmark
Benchmark comparison of SR models. Downward arrows (↓) denote metrics in which lower values are preferable, and upward arrows (↑) indicate metrics in which higher values reflect better performance.
Consistency | Synthesis | Correctness | ||||||
---|---|---|---|---|---|---|---|---|
reflectance ↓ | spectral ↓ | spatial ↓ | high-frequency ↑ | ha ↓ | om ↓ | im ↑ | ||
NAIP | SuperImage | 0.008 | 7.286 | 0.131 | 0.003 | 0.117 | 0.784 | 0.098 |
SR4RS | 0.016 | 3.471 | 1.156 | 0.010 | 0.869 | 0.077 | 0.054 | |
diffusers | 0.463 | 12.437 | 2.88 | 0.013 | 0.905 | 0.055 | 0.040 | |
SPOT | SuperImage | 0.009 | 3.512 | 0.062 | 0.006 | 0.160 | 0.794 | 0.046 |
SR4RS | 0.039 | 3.232 | 1.151 | 0.023 | 0.834 | 0.115 | 0.051 | |
diffusers | 0.417 | 11.730 | 0.817 | 0.014 | 0.686 | 0.251 | 0.063 | |
VENµS | SuperImage | 0.009 | 8.687 | 0.099 | 0.003 | 0.403 | 0.380 | 0.217 |
SR4RS | 0.014 | 3.394 | 1.122 | 0.012 | 0.971 | 0.017 | 0.012 | |
diffusers | 0.467 | 13.303 | 0.806 | 0.009 | 0.933 | 0.043 | 0.024 |
Installation
Install the latest version from PyPI:
pip install opensr-test
Upgrade opensr-test
by running:
pip install -U opensr-test
Install the latest dev version from GitHub by running:
pip install git+https://github.com/ESAOpenSR/opensr-test
Examples
The following examples show how to use opensr-test
to benchmark your SR model.
Visualizations
The opensr-test
package provides a set of visualizations to help you understand the performance of your SR model.
import torch
import opensr_test
import matplotlib.pyplot as plt
from super_image import HanModel
# Define the SR model
srmodel = HanModel.from_pretrained('eugenesiow/han', scale=4)
# Load the data
lr, hr, landuse, parameters = opensr_test.load("spot").values()
# Define the benchmark experiment
metrics = opensr_test.Metrics()
# Define the image to be tested
idx = 0
lr_img = torch.from_numpy(lr[idx, 0:3])
hr_img = torch.from_numpy(hr[idx, 0:3])
sr_img = srmodel(lr_img[None]).squeeze().detach()
# Compute the metrics
metrics.compute(
lr=lr_img, sr=sr_img, hr=hr_img,
stability_threshold = parameters.stability_threshold[idx],
im_score = parameters.correctness_params[0],
om_score = parameters.correctness_params[1],
ha_score = parameters.correctness_params[2]
)
Now, we can visualize the results using the opensr_test.visualize
module.
fDisplay the triplets LR, SR and HR images:
metrics.plot_triplets()
Display the quadruplets LR, SR, HR and landuse images:
metrics.plot_quadruplets()
Display the matching points between the LR and SR images:
metrics.plot_spatial_matches()
Display a summary of all the metrics:
metrics.plot_summary()
Display the correctness of the SR image:
metrics.plot_tc()
Deeper understanding
Explore the API section for more details about personalizing your benchmark experiments.
Citation
If you use opensr-test
in your research, please cite our paper:
Coming soon!
Acknowledgements
This work was make with the support of the European Space Agency (ESA) under the project “Explainable AI: application to trustworthy super-resolution (OpenSR)”. Cesar Aybar acknowledges support by the National Council of Science, Technology, and Technological Innovation (CONCYTEC, Peru) through the “PROYECTOS DE INVESTIGACIÓN BÁSICA – 2023-01” program with contract number PE501083135-2023-PROCIENCIA. Luis Gómez-Chova acknowledges support from the Spanish Ministry of Science and Innovation (project PID2019-109026RB-I00 funded by MCIN/AEI/10.13039/501100011033).
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for opensr_test-0.2.1-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 5c21678a6c56cd327112f4bff63815fefef265f8add6e57acb51c7db226676d4 |
|
MD5 | 5b6d8234e35b2f4a1c68b870e6726b8b |
|
BLAKE2b-256 | deee76d1a5cf8f276ecbfc7e896e5f3db1787fb799308115aeb0abcda13f56ef |