vbench2 0.1.0

Video generation benchmark

VBench-2.0 is a comprehensive benchmark suite for video generative models. You can use VBench-2.0 to evaluate video generation models from 18 different ability aspects.

This project is the PyPI implementation of the following research:

VBench-2.0: Advancing Video Generation Benchmark Suite for Intrinsic Faithfulness
Dian Zheng^∗, Ziqi Huang^∗, Hongbo Liu, Kai Zou, Yinan He, Fan Zhang, Yuanhan Zhang, Jingwen He, Wei-Shi Zheng⁺, Yu Qiao⁺, Ziwei Liu⁺

Installation

pip install torch==2.5.1 torchvision==0.20.1 torchaudio==2.5.1 --index-url https://download.pytorch.org/whl/cu118
pip install vbench2
pip install mmcv==2.2.0 -f https://download.openmmlab.com/mmcv/dist/cu118/torch2.4/index.html --no-cache-dir
pip install retinaface_pytorch==0.0.8 --no-deps

Usage

Evaluate Your Own Videos

We support evaluating any video. Simply provide the path to the video file, or the path to the folder that contains your videos. There is no requirement on the videos' names.

• Note: We support customized videos / prompts for the following dimensions: 'subject_consistency', 'background_consistency', 'motion_smoothness', 'dynamic_degree', 'aesthetic_quality', 'imaging_quality'

Evaluation on the Standard Prompt Suite of VBench-2.0

command line

    vbench evaluate --videos_path $VIDEO_PATH --dimension $DIMENSION

For example:

    vbench evaluate --videos_path "sampled_videos/lavie/human_action" --dimension "human_action"

python

    from vbench import VBench
    my_VBench = VBench(device, <path/to/VBench_full_info.json>, <path/to/save/dir>)
    my_VBench.evaluate(
        videos_path = <video_path>,
        name = <name>,
        dimension_list = [<dimension>, <dimension>, ...],
    )

For example:

    from vbench import VBench
    my_VBench = VBench(device, "vbench/VBench_full_info.json", "evaluation_results")
    my_VBench.evaluate(
        videos_path = "sampled_videos/lavie/human_action",
        name = "lavie_human_action",
        dimension_list = ["human_action"],
    )

Prompt Suite

We provide prompt lists are at prompts/.

Check out details of prompt suites, and instructions for how to sample videos for evaluation.

Citation

If you find this package useful for your reports or publications, please consider citing the VBench-2.0 paper:

 @article{zheng2025vbench2,
     title={{VBench-2.0}: Advancing Video Generation Benchmark Suite for Intrinsic Faithfulness},
     author={Zheng, Dian and Huang, Ziqi and Liu, Hongbo and Zou, Kai and He, Yinan and Zhang, Fan and Zhang, Yuanhan and He, Jingwen and Zheng, Wei-Shi and Qiao, Yu and Liu, Ziwei},
     journal={arXiv preprint arXiv:2503.21755},
     year={2025}
 }