pytorchfwd 0.0.4.dev0

Compute frecet wavelet distances

Lokesh Veeramacheneni¹, Moritz Wolter¹, Hilde Kuehne^1,2, and Juergen Gall¹

1. University of Bonn,

2. University of Tübingen, MIT-IBM Watson AI Lab.

[Archive] [Project Page]

Keywords: Frechet Distance, Wavelet Packet Transform, Frechet Inception Distance, Diffusion, GAN, ImageNet, Image generation metrics.

Abstract: Modern metrics for generative learning like Fréchet Inception Distance (FID) demonstrate impressive performance. However, they suffer from various shortcomings, like a bias towards specific generators and datasets. To address this problem, we propose the Fréchet Wavelet Distance (FWD) as a domain-agnostic metric based on Wavelet Packet Transform (\(W_p\)). FWD provides a sight across a broad spectrum of frequencies in images with a high resolution, along with preserving both spatial and textural aspects. Specifically, we use \(W_p\) to project generated and dataset images to packet coefficient space. Further, we compute Fréchet distance with the resultant coefficients to evaluate the quality of a generator. This metric is general-purpose and dataset-domain agnostic, as it does not rely on any pre-trained network while being more interpretable because of frequency band transparency. We conclude with an extensive evaluation of a wide variety of generators across various datasets that the proposed FWD is able to generalize and improve robustness to domain shift and various corruptions compared to other metrics.

Installation

Install via pip

pip install pytorchfwd

Usage

python -m pytorchfwd <path to dataset> <path to generated images>

Here are the other arguments and defaults used.

python -m pytorchfwd --help

usage: pytorchfwd.py [-h] [--batch-size BATCH_SIZE] [--num-processes NUM_PROCESSES] [--save-packets] [--wavelet WAVELET] [--max_level MAX_LEVEL] [--log_scale] path path

positional arguments:
  path                  Path to the generated images or path to .npz statistics file.

options:
  -h, --help            show this help message and exit
  --batch-size          Batch size for wavelet packet transform. (default: 128)
  --num-processes       Number of multiprocess. (default: None)
  --save-packets        Save the packets as npz file. (default: False)
  --wavelet             Choice of wavelet. (default: sym5)
  --max_level           wavelet decomposition level (default: 4)
  --log_scale           Use log scaling for wavelets. (default: False)

We conduct all the experiments with Haar wavelet with transformation/decomposition level of 4 for 256x256 image. In future, we plan to release the jax-version of this code.

Citation

If you use this work, please cite using following bibtex entry

@misc{veeramacheneni2024fwd,
   title={Fr\'echet Wavelet Distance: A Domain-Agnostic Metric for Image Generation},
   author={Lokesh Veeramacheneni and Moritz Wolter and Hildegard Kuehne and Juergen Gall},
   year={2024},
   eprint={2312.15289},
   archivePrefix={arXiv},
   primaryClass={cs.CV},
   url={https://arxiv.org/abs/2312.15289},
}

Acknowledgments

The code is built with inspiration from Pytorch-FID. We use PyTorch Wavelet Toolbox for Wavelet Packet Transform implementation. We recommend to have a look at these repositories.

Testing

The tests folder contains tests to conduct independent verification of FWD. Github workflow executes all these tests. To run tests on your local system install nox, as well as this package via pip install ., and run

nox -s test