vbench2 0.1.1

VBench-2.0 is a comprehensive benchmark suite for video generative models. You can use VBench-2.0 to evaluate video generation models from 18 different ability aspects.

This project is the PyPI implementation of the following research:

VBench-2.0: Advancing Video Generation Benchmark Suite for Intrinsic Faithfulness
Dian Zheng^∗, Ziqi Huang^∗, Hongbo Liu, Kai Zou, Yinan He, Fan Zhang, Yuanhan Zhang, Jingwen He, Wei-Shi Zheng⁺, Yu Qiao⁺, Ziwei Liu⁺

Installation

conda install psutil
pip install torch==2.5.1 torchvision==0.20.1 torchaudio==2.5.1 --index-url https://download.pytorch.org/whl/cu118
python -m pip install ninja
python -m pip install git+https://github.com/Dao-AILab/flash-attention.git@v2.7.2.post1
pip install vbench2
pip install mmcv==2.2.0 -f https://download.openmmlab.com/mmcv/dist/cu118/torch2.4/index.html --no-cache-dir
pip install retinaface_pytorch==0.0.8 --no-deps

Usage

Evaluation on the Standard Prompt Suite of VBench-2.0

command line

    vbench2 evaluate --videos_path $VIDEO_PATH --dimension $DIMENSION

For example:

    vbench2 evaluate --videos_path "sampled_videos/HunyuanVideo/Human_Interaction" --dimension "Human_Interaction"
    srun -p video-aigc-3 --gres=gpu:1 vbench2 evaluate --videos_path "/mnt/petrelfs/zhengdian/zhengdian/VBench2.0/sample_video/HunyuanVideo/Human_Interaction" --dimension "Human_Interaction"

python

    from vbench2 import VBench2
    my_VBench = VBench2(device, <path/to/VBench_full_info.json>, <path/to/save/dir>)
    my_VBench.evaluate(
        videos_path = <video_path>,
        name = <name>,
        dimension_list = [<dimension>, <dimension>, ...],
    )

For example:

    from vbench2 import VBench2
    my_VBench = VBench2(device, "vbench2/VBench2_full_info.json", "evaluation_results")
    my_VBench.evaluate(
        videos_path = "sampled_videos/HunyuanVideo/Human_Interaction",
        name = "HunyuanVideo_Human_Interaction",
        dimension_list = ["Human_Interaction"],
    )

Prompt Suite

We provide prompt lists are at prompts/.

Check out details of prompt suites, and instructions for how to sample videos for evaluation.

Citation

If you find this package useful for your reports or publications, please consider citing the VBench-2.0 paper:

 @article{zheng2025vbench2,
     title={{VBench-2.0}: Advancing Video Generation Benchmark Suite for Intrinsic Faithfulness},
     author={Zheng, Dian and Huang, Ziqi and Liu, Hongbo and Zou, Kai and He, Yinan and Zhang, Fan and Zhang, Yuanhan and He, Jingwen and Zheng, Wei-Shi and Qiao, Yu and Liu, Ziwei},
     journal={arXiv preprint arXiv:2503.21755},
     year={2025}
 }