pytorch-fxd 0.3.1

A package containing methods to compute FXD score and other popular metrics for Chest X-Rays.

FXD score for PyTorch

An Improved Metric for Evaluating Chest X-Ray Generation Methods

This is a port of the official implementation of FXD Score to PyTorch.

Generative models are widely used to augment data to improve machine learning models trained on insufficient data. The quality of generated data plays a significant role in the model’s performance. Evaluating a quality of the generative models used a challenging problem. Many sample-based evaluation metrics like Fréchet Inception Distance (FID) compare the distributions of real and generated images in a latent space of models trained on the ImageNet dataset. Such metrics work well in evaluating quality of images of common entities but generally fail in case of images from niche domains not present in the ImageNet dataset like chest x-rays. In this work we propose the FXD score which computes Fréchet distance over the latent space of the pretrained TorchXrayVision model. We perform a series of carefully designed experiments to test for necessary conditions for a meaningful metric. Through these experiments we show that the proposed metric shows more discriminability robustness to transformations and is more sensitive to, mode dropping, mode collapsing and overfitting compared to metrics based on ImageNet weights. Which also proves the importance of choosing domain specific representations in evaluating the quality of generative models. We believe that the proposed FXD score will enable researchers in building generation models efficiently which will in turn aid in building better models to analyze patients more accurately.

FXD Score is calculated by computing the Fréchet distance between two Gaussians fitted to feature representations of the Inception network.

Installation

All the codes have been tested on Linux (ubuntu)

Install from pip:

pip install pytorch-fxd

Requirements:

• python3
• pytorch
• torchvision
• pillow
• numpy
• scipy
• pytorch-fid
• torchxrayvision
• scikit-image

Usage

To compute the FXD score between two datasets, where images of each dataset are contained in an individual folder:

pytorch-fxd --dataroot path/to/dataset

path/to/dataset must contain two folders: setA and setB

To run the evaluation on GPU, use the flag --cuda

To compute other metric scores, use the flag --all

The scores are by default calculated using Torchxrayvision's pretrained model. To calculate using Inception v3 instead, use the flag --metric fid

To view other optional arguments:

pytorch-fxd --help

License

This implementation is licensed under the MIT License.