videogen-hub 0.1.4a0

VideoGenHub is a one-stop library to standardize the inference and evaluation of all the conditional video generation models.

VideoGenHub

VideoGenHub is a one-stop library to standardize the inference and evaluation of all the conditional video generation models.

• We define 2 prominent generation tasks (Text-to-Video and Image-to-Video).
• We built a unified inference pipeline to ensure fair comparison. We currently support around 10 models.

📰 News

• 2024 Jun 10: GenAI-Arena Paper is out. It is featured on Huggingface Daily Papers.

📄 Table of Contents

• 🛠️ Installation
• 👨‍🏫 Get Started
• 🎫 License
• 🖊️ Citation

🛠️ Installation 🔝

To install from pypi:

pip install videogen-hub

To install from github:

git clone https://github.com/TIGER-AI-Lab/VideoGenHub.git
cd VideoGenHub
cd env_cfg
pip install -r requirements.txt
cd ..
pip install -e .

The requirement of opensora is in env_cfg/opensora.txt

For some models like show one, you need to login through huggingface-cli.

huggingface-cli login

👨‍🏫 Get Started 🔝

Benchmarking

To reproduce our experiment using benchmark.

For text-to-video generation:

./t2v_inference.sh --<model_name> --<device>

Infering one model

import videogen_hub

model = videogen_hub.load('VideoCrafter2')
video = model.infer_one_video(prompt="A child excitedly swings on a rusty swing set, laughter filling the air.")

# Here video is a torch tensor of shape torch.Size([16, 3, 320, 512])

See Google Colab here: https://colab.research.google.com/drive/145UMsBOe5JLqZ2m0LKqvvqsyRJA1IeaE?usp=sharing

🧠 Philosophy 🔝

By streamlining research and collaboration, VideoGenHub plays a pivotal role in propelling the field of Video Generation.

• Purity of Evaluation: We ensure a fair and consistent evaluation for all models, eliminating biases.
• Research Roadmap: By defining tasks and curating datasets, we provide clear direction for researchers.
• Open Collaboration: Our platform fosters the exchange and cooperation of related technologies, bringing together minds and innovations.

Implemented Models

We included more than 10 Models in video generation.

Method	Venue	Type
LaVie	-	Text-To-Video Generation
VideoCrafter2	-	Text-To-Video Generation
ModelScope	-	Text-To-Video Generation
StreamingT2V	-	Text-To-Video Generation
Show 1	-	Text-To-Video Generation
OpenSora	-	Text-To-Video Generation
OpenSora-Plan	-	Text-To-Video Generation
T2V-Turbo	-	Text-To-Video Generation
DynamiCrafter2	-	Image-To-Video Generation
SEINE	ICLR'24	Image-To-Video Generation
Consisti2v	-	Image-To_Video Generation
I2VGenXL	-	Image-To_Video Generation

TODO

• Add ComfyUI Support
• Add Metrics Support
• Add Visualization Support (Similar to ImagenHub)
• Add Video Editing Task

🎫 License 🔝

This project is released under the License.

🖊️ Citation 🔝

This work is a part of GenAI-Arena work.

Please kindly cite our paper if you use our code, data, models or results:

@misc{jiang2024genai,
      title={GenAI Arena: An Open Evaluation Platform for Generative Models}, 
      author={Dongfu Jiang and Max Ku and Tianle Li and Yuansheng Ni and Shizhuo Sun and Rongqi Fan and Wenhu Chen},
      year={2024},
      eprint={2406.04485},
      archivePrefix={arXiv},
      primaryClass={cs.AI}
}