FVD calculation in PyTorch with I3D or VideoMAE models
Project description
Content-Debiased FVD for Evaluating Video Generation Models
Project Page | Documentation | Paper
FVD is observed to favor the quality of individual frames over realistic motions. We verify this with quantitative measurement. We show that the bias can be attributed to the features extracted from a supervised video classifier trained on the content-biased dataset and using features from large-scale unsupervised models can mitigate the bias. This repo contains code tookit for easily computing FVDs with different pre-trained models. Please refer to out project page or paper for more details about the analysis.
On the Content Bias in Fréchet Video Distance
Songwei Ge, Aniruddha Mahapatra, Gaurav Parmar, Jun-Yan Zhu, Jia-Bin Huang
UMD, CMU
CVPR 2024
Quickstart
We provide a simple interface to compute FVD scores between two sets of videos that can be adapted to different scenarios. The following code snippet demonstrates how to compute FVD scores between two folders of videos.
from fvd import cd_fvd
fvd = cd_fvd('videomae', ckpt_path=None)
fvd.compute_real_stats(fvd.load_videos('path/to/real videos/'))
fvd.compute_fake_stats(fvd.load_videos('path/to/fake videos/'))
fvd = fvd.compute_fvd_from_stats()
Please refer to the documentation for more detailed instructions on the usage.
Precomputed Datasets
We provide precomputed statistics for the following datasets.
Dataset | Video Length | Resolution | Reference Split | # Reference Videos | Model | Skip Frame # | Seed |
---|---|---|---|---|---|---|---|
UCF101 | 16, 128 | 128, 256 | train |
2048, full | I3D , VideoMAE-v2-SSv2 |
1 | 0 |
Sky | 16, 128 | 128, 256 | train |
2048, full | I3D , VideoMAE-v2-SSv2 |
1 | 0 |
Taichi | 16, 128 | 128, 256 | train |
2048, full | I3D , VideoMAE-v2-SSv2 |
1 | 0 |
Kinetics | 16 | 128, 256 | train |
2048, full | I3D , VideoMAE-v2-SSv2 |
1 | 0 |
Kinetics | 128 | 128, 256 | train |
2048 | I3D , VideoMAE-v2-SSv2 |
1 | 0 |
FFS | 16, 128 | 128, 256 | train |
2048, full | I3D , VideoMAE-v2-SSv2 |
1 | 0 |
Citation
@inproceedings{ge2024content,
title={On the Content Bias in Fréchet Video Distance},
author={Ge, Songwei and Mahapatra, Aniruddha and Parmar, Gaurav and Zhu, Jun-Yan and Huang, Jia-Bin},
booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
year={2024}
}
Acknowledgement
We thank Angjoo Kanazawa, Aleksander Holynski, Devi Parikh, and Yogesh Balaji for their early feedback and discussion. We thank Or Patashnik, Richard Zhang, and Hadi Alzayer for their helpful comments and paper proofreading. We thank Ivan Skorokhodov for his help with reproducing the StyleGAN-v ablation experiments. Part of the evaluation code is built on StyleGAN-v.
Licenses
All material in this repository is made available under the MIT License.
metric_utils.py is adapted from the stylegan-v metric_utils.py, which was built on top of StyleGAN2-ADA and restricted by the NVidia Source Code license .
VideoMAE-v2 checkpoint is publicly available. Please consider filling this questionaire to help improve the future works.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.