visionzip 0.1

VisionZip: Longer is Better but Not Necessary in Vision Language Models

VisionZip: Longer is Better but Not Necessary in Vision Language Models

TABLE OF CONTENTS

1. News
2. Highlights
3. Video
4. Demo
5. Installation
6. Quick Start
7. Evaluation
8. Examples
9. Citation
10. Acknowledgement
11. License

News

• [2024.11.30] We release Paper and this GitHub repo, including code for LLaVA.

VisionZip: Longer is Better but Not Necessary in Vision Language Models [Paper]
Senqiao Yang, Yukang Chen, Zhuotao Tian, Chengyao Wang, Jingyao Li, Bei Yu, Jiaya Jia

Highlights

1. Our VisionZip is a text-agnostic method that outperforms the current state-of-the-art Efficient VLM methods. By retaining only 10% of visual tokens, it achieves nearly 95% performance.
2. VisionZip significantly reduces the prefilling time and the total inference time (with KV cache enabled).
3. Why does this simple, text-agnostic method outperform text-relevant methods? We conduct an in-depth analysis in the paper and provide a demo to visualize these findings.
4. Since VisionZip is a text-agnostic method that reduces visual tokens before input into the LLM, it can adapt to any existing LLM acceleration algorithms and is applicable to any task that a vanilla VLM can perform, such as multi-turn conversations.
5. VisionZip can be applied during the inference stage (without incurring any additional training cost), the efficient tuning stage (to achieve better results), and the training stage (saving 2× memory and 2× time).

Video

Demo

Speed Improvement

The input video is about the Titanic, and the question is, "What’s the video talking about?"

It is important to note that the left side shows the vanilla model, which encodes only 16 frames, while the right side shows our VisionZip, which, despite **encoding 32 frames, is still twice as fast as the vanilla model.**

Visualize Redundancy and Misalignment

Explore the visual redundancy and feature misalignment in the above Demo. To run it locally, use the following command:

python gradio_demo.py 

Installation

Our code is easy to use.

1. Install the LLaVA environment.

2. For formal usage, you can install the package from PyPI by running the following command:

pip install visionzip

For development, you can install the package by cloning the repository and running the following command:

git clone https://github.com/dvlab-research/VisionZip
cd VisionZip
pip install -e .

Quick Start

from llava.model.builder import load_pretrained_model
from llava.mm_utils import get_model_name_from_path
from llava.eval.run_llava import eval_model
from visionzip import visionzip
model_path = "liuhaotian/llava-v1.5-7b"

tokenizer, model, image_processor, context_len = load_pretrained_model(
    model_path=model_path,
    model_base=None,
    model_name=get_model_name_from_path(model_path)
)
## VisoinZip retains 54 dominant tokens and 10 contextual tokens
model = visionzip(model, dominant=54, contextual=10)

Evaluation

The evaluation code follows the structure of LLaVA or Lmms-Eval. After loading the model, simply add two lines as shown below:

## Load LLaVA Model (code from llava.eval.model_vqa_loader)
tokenizer, model, image_processor, context_len = load_pretrained_model(model_path, args.model_base, model_name)
## add VisionZip
from visionzip import visionzip
model = visionzip(model, dominant=54, contextual=10)

Examples

Multi-turn Conversations

VisionZip, which extracts text-agnostic tokens, is better suited for multi-turn dialogue.

Longer Videos with More Frames

VisionZip reduces the number of visual tokens per frame, allowing more frames to be processed. This improves the model's ability to understand longer videos.

Citation

If you find this project useful in your research, please consider citing:

@inproceedings{None,
  author       = {},
  title        = {},
  booktitle    = {},
  year         = {2024},
}

Acknowledgement

• This work is built upon LLaVA, mini-Gemini, Lmms-Eval, and Video-LLaVA. We thank them for their excellent open-source contributions.

• We also thank StreamingLLM, FastV, SparseVLM, and others for their contributions, which have provided valuable insights.

License

• VisionZip is licensed under the Apache License 2.0.