AIGVE-Tool

AI Generated Video Evaluation toolkit

Implemented:

Models:

Video-Only Neural Network-Based evaluation metrics:

1. GSTVQA
2. SimpleVQA
3. LightVQA_Plus

Distribuition Comparison-Based evaluation metrics:

These metrics primarily assess the quality of generated samples by comparing distributions of real and generated data:

1. FID
2. FVD
3. IS

Vision-Language Similarity-Based evaluation metrics:

These metrics primarily evaluate alignment, similarity, and coherence between visual and textual representations. They focus on how well images and text match, often using embeddings from models like CLIP and BLIP:

1. CLIPSim
2. CLIPTemp
3. BLIP
4. Pickscore

Vision-Language Understanding-Based evaluation metrics:

These metrics assess higher-level understanding, reasoning, and factual consistency in vision-language models. They go beyond similarity, evaluating semantic correctness, factual alignment, and structured comprehension:

1. VIEScore
2. TIFA
3. DSG

Multi-Faceted evaluation metrics

These metrics are structured, multi-dimensional evaluation metrics designed to assess AI models across diverse sub-evaluation dimensions. They provide a comprehensive benchmarking framework that integrates aspects like video understanding, physics-based reasoning, and modular evaluation, enabling a more holistic assessment of model performance.

1. VideoPhy
2. VideoScore

Dataset:

1. Toy dataset
2. AIGVE-Bench

Environment

conda env remove --name aigve

conda env create -f environment.yml
conda activate aigve
conda install pytorch==2.1.0 torchvision==0.16.0 torchaudio==2.1.0 pytorch-cuda=11.8 -c pytorch -c nvidia

(MMCV from v1.7.2 support PyTorch 2.1.0 and 2.0.0)

Run:

python main.py {metric_config_file}.py

Take Examples:

rm -rf ~/.cache

For GSTVQA: cd VQA_Toolkit/aigve python main_aigve.py AIGVE_Tool/aigve/configs/gstvqa.py --work-dir ./output

For SimpleVQA: cd VQA_Toolkit/aigve python main_aigve.py AIGVE_Tool/aigve/configs/simplevqa.py --work-dir ./output

For LightVQAPlus: `` cd VQA_Toolkit/aigve python main_aigve.py AIGVE_Tool/aigve/configs/lightvqa_plus.py --work-dir ./output

``

For GSTVQACrossData: cd VQA_Toolkit/aigve python main_aigve.py AIGVE_Tool/aigve/configs/gstvqa_crossdata.py --work-dir ./output

For CLIPSim: cd VQA_Toolkit/aigve python main_aigve.py AIGVE_Tool/aigve/configs/clipsim.py --work-dir ./output

For VideoPhy: cd VQA_Toolkit/aigve python main_aigve.py AIGVE_Tool/aigve/configs/clipsim.py --work-dir ./output

Acknowledge

The Toolkit is build top the top of MMEngine

We acknowledge original repositories of various VQA methods: GSTVQA, CLIPSim,