NeVA: NVIDIA's Visual Question Answering Transformer

NeVA is a powerful and versatile Visual Question Answering model by NVIDIA. Neva builds upon the open-source LLaMA model, integrating it with an NVIDIA-trained GPT model to offer state-of-the-art performance.

Appreciation

• All the creators in Agora, Join Agora the community of AI engineers changing the world with their creations.
• LucidRains for inspiring me to devote myself to open source AI

Installation

To integrate NeVA into your Python environment, you can install it via pip:

pip install nevax

Usage

import torch
from nevax import Neva

#usage
img = torch.randn(1, 3, 256, 256)
caption_tokens = torch.randint(0, 4)

model = Neva()
output = model(img, caption_tokens)

Description

At a high level, NeVA utilizes a frozen Hugging Face CLIP model to encode images. These encoded images are projected to text embedding dimensions, concatenated with the embeddings of the given prompt, and subsequently passed through the language model. The training process comprises two main stages:

1. Pretraining: Only the projection layer is trained with the language model kept frozen. This stage uses image-caption pairs for training.
2. Finetuning: Both the language model and the projection layer are trained. This stage utilizes synthetic instruction data generated with GPT4.

Model Specifications

• Architecture Type: Transformer
• Network Architecture: GPT + CLIP
• Model versions: 8B, 22B, 43B

Input & Output

• Input Format: RGB Image + Text
• Input Parameters: temperature, max output tokens, quality, toxicity, humor, creativity, violence, helpfulness, not_appropriate
• Output Format: Text

Integration and Compatibility

• Supported Hardware Platforms: Hopper, Ampere/Turing
• Supported Operating Systems: Linux
• Runtime: N/A

Training & Fine-tuning Data

Pretraining Dataset:

• Link: CC-3M
• Description: The dataset comprises CC3M images and captions, refined to 595,000 samples.
• License: COCO, CC-3M, BLIP

Finetuning Dataset:

• Link: Synthetic data produced by GPT4
• Description: The dataset, with 158,000 samples, was synthetically generated by GPT4. It encompasses a blend of short question answers, detailed image descriptions, and higher-level reasoning questions.
• License: CC-BY-NC 4.0 License

Inference

• Engine: Triton
• Test Hardware: Other

References

• Visual Instruction Tuning paper
• Blog
• Codebase
• Demo

Licensing

This project is licensed under the MIT License license.

Citation