pytorch-fxd 0.2

A package containing methods to compute FXD score and other popular metrics for Chest X-Rays.

FXD Score: An Improved Metric for Evaluating Chest X-Ray Generation Methods

Generative models are widely used to augment data to improve machine learning models trained on insufficient data. The quality of generated data plays a significant role in the model’s performance. Evaluating a quality of the generative models used a challenging problem. Many sample-based evaluation metrics like Fréchet Inception Distance (FID) compare the distributions of real and generated images in a latent space of models trained on the ImageNet dataset. Such metrics work well in evaluating quality of images of common entities but generally fail in case of images from niche domains not present in the ImageNet dataset like chest x-rays. In this work we propose the FXD score which computes Fréchet distance over the latent space of the pretrained TorchXrayVision model. We perform a series of carefully designed experiments to test for necessary conditions for a meaningful metric. Through these experiments we show that the proposed metric shows more discriminability robustness to transformations and is more sensitive to, mode dropping, mode collapsing and overfitting compared to metrics based on ImageNet weights. Which also proves the importance of choosing domain specific representations in evaluating the quality of generative models. We believe that the proposed FXD score will enable researchers in building generation models efficiently which will in turn aid in building better models to analyze patients more accurately.