clip-mmd 0.0.2

CLIP Maximum Mean Discrepancy (CMMD) for evaluating generative models

CLIP-MMD

An unofficial implementation of Rethinking FID: Towards a Better Evaluation Metric for Image Generation. Uses Transformers on PyTorch.

Features

• support multiple GPUs with DDP.
• acquire less memory; support larger dataset (like 1M each).
• uses resizer of PIL for more accurate results, recommended by Clean-FID.
• support multiple input: folder, Dataset(for calculation during training), pre-computed stastics.

Comparison

Measure time (second) of calculation. Cuda is on single V100.

n, m, dim	original	efficient	(by row)	(cuda)	(cuda + by row)
2048, 16384, 768	1.379	1.142	0.421	0.460	0.164
16384, 16384, 768	3.312	2.764	1.137	0.567	0.391
50000, 50000, 768	29.74	26.64	12.85	1.413	1.383
1280000, 50000, 768	OOM	OOM	98.21	OOM	4.446

Thanks

To cmmd-pytorch. Some logics came from it.

Usage

pip install clip-mmd

CLI interface

usage: clip-mmd [-h] [--no-cuda] [--gpus GPUS] [--no-mem-save] [--batch-size BATCH_SIZE] [--calculator-bs CALCULATOR_BS] [--model MODEL] [--num-workers NUM_WORKERS] [--extract-mode] [--size SIZE]
                [--interpolation {nearest,bilinear,bicubic,lanczos,hamming,box}]
                data_path_1 [data_path_2]

Command Line Interface for CMMD calculations and feature extraction.

positional arguments:
  data_path_1           Path to the first data folder or file
  data_path_2           Path to the second data folder or file, or output path of pre-extracted features

options:
  -h, --help            show this help message and exit
  --no-cuda             only use cpu
  --gpus GPUS           Comma-separated list of GPUs to use (e.g., 0,1,2,3). Use it if you want do on multi gpus.
  --no-mem-save         Flag to disable memory-saving features
  --batch-size BATCH_SIZE
                        Batch size for processing
  --calculator-bs CALCULATOR_BS
                        Batch size for the CMMD calculator
  --model MODEL         Model to use for feature extraction. Default: openai/clip-vit-large-patch14-336,
  --num-workers NUM_WORKERS
                        Numbers of dataloader workers
  --extract-mode        If enabled, only extract reatures from data_path_1, and save to data_path_2.
  --size SIZE           Image patch size for model input
  --interpolation {nearest,bilinear,bicubic,lanczos,hamming,box}
                        Interpolation algorithm for resampling an image. Default: bicubic.

example

clip-mmd /path/to/generated_samples.npz /path/to/reference/imgs/
clip-mmd /path/to/images/ features/for/them.pth --gpus 3,4,5,6 --batch-size 128 --extract-mode
clip-mmd /path/to/features1.pth /path/to/features2.pth
clip-mmd /folder/1  /folder/2 --interpolation bilinear
clip-mmd /pre/extracted/features.pth /sampled/images 

OR directly use it:

from clip_mmd import logic
prep = logic.CMMD(data_parallel=True, device=[2,3,4,5])
distance = prep.execute('path/to/folder/1', 'path/to/folder/2')
print(f'CMMD: {distance:5f}')

You can also use path to sampled npz or pth, pre-caculated statics, or even customized Dataset.