NVIDIA Plugin for Vision Agents

NVIDIA VLM integration for Vision Agents. Supports vision language models through NVIDIA's Chat Completions API with NVCF asset management.

Features

• Video understanding: Automatically buffers and forwards video frames to NVIDIA VLM models
• Streaming responses: Supports streaming text responses with real-time chunk events
• Asset management: Automatically uploads frames as assets and cleans them up after use
• Configurable frame rate and buffer duration for optimal performance

Installation

uv add "vision-agents[nvidia]"
# or directly
uv add vision-agents-plugins-nvidia

Configuration

Set your NVIDIA API key:

export NVIDIA_API_KEY=your_nvidia_api_key

Usage

Basic VLM Usage

from vision_agents.plugins import nvidia

vlm = nvidia.VLM(
    model="meta/llama-3.2-11b-vision-instruct",
    fps=1,
    frame_buffer_seconds=10,
)

# VLM automatically buffers video frames when used with an Agent
response = await vlm.simple_response("What do you see?")
print(response.text)

With Custom Configuration

from vision_agents.plugins import nvidia

vlm = nvidia.VLM(
    model="meta/llama-3.2-11b-vision-instruct",
    api_key="your-api-key",
    fps=2,
    frame_buffer_seconds=15,
    frame_width=1280,
    frame_height=720,
    max_tokens=2048,
    temperature=0.3,
    top_p=0.8,
)

Configuration

Dependencies

• vision-agents: Core Vision Agents framework
• aiohttp>=3.9.0: Async HTTP client for API requests