OpenAI Plugin for Vision Agents
OpenAI LLM integration for Vision Agents framework with support for both standard and realtime interactions.
It enables features such as:
- Real-time transcription and language processing using OpenAI models
- Easy integration with other Vision Agents plugins and services
- Function calling capabilities for dynamic interactions
Installation
uv add "vision-agents[openai]"
# or directly
uv add vision-agents-plugins-openai
Usage
Standard LLM
This example shows how to use "gpt-4.1" model with TTS and STT services for audio communication via openai.LLM() API.
The openai.LLM() class uses OpenAI's Responses API under the hood.
To work with models via legacy Chat Completions API, see the Chat Completions models section.
from vision_agents.core import User, Agent
from vision_agents.core.agents import AgentLauncher
from vision_agents.plugins import deepgram, getstream, cartesia, smart_turn, openai
agent = Agent(
edge=getstream.Edge(),
agent_user=User(name="Friendly AI"),
instructions="Be nice to the user",
llm=openai.LLM("gpt-4.1"),
tts=cartesia.TTS(),
stt=deepgram.STT(),
turn_detection=smart_turn.TurnDetection(),
)
Realtime LLM
Realtime audio and video communication is also supported via Realtime class.
In this mode, the model handles audio and video processing directly without the need for TTS and STT services.
from vision_agents.core import User, Agent
from vision_agents.plugins import getstream, openai
agent = Agent(
edge=getstream.Edge(),
agent_user=User(name="Friendly AI"),
instructions="Be nice to the user",
llm=openai.Realtime(),
)
Chat Completions models
The openai.ChatCompletionsLLM and
openai.ChatCompletionsVLM classes provide APIs for text and vision models that use the Chat Completions API.
They are compatible with popular inference backends such as vLLM, TGI, and Ollama.
For example, you can use them to interact with Qwen 3 VL visual model hosted on Baseten:
from vision_agents.core import User, Agent
from vision_agents.plugins import deepgram, getstream, elevenlabs, vogent, openai
# Instantiate the visual model wrapper
llm = openai.ChatCompletionsVLM(model="qwen3vl")
# Create an agent with video understanding capabilities
agent = Agent(
edge=getstream.Edge(),
agent_user=User(name="Video Assistant", id="agent"),
instructions="You're a helpful video AI assistant. Analyze the video frames and respond to user questions about what you see.",
llm=llm,
stt=deepgram.STT(),
tts=elevenlabs.TTS(),
turn_detection=vogent.TurnDetection(),
processors=[],
)
For full code, see examples/qwen_vl_example.
Function Calling
The LLM and Realtime APIs support function calling, allowing the assistant to invoke custom functions you define.
This enables dynamic interactions like:
- Database queries
- API calls to external services
- File operations
- Custom business logic
from vision_agents.plugins import openai
llm = openai.LLM("gpt-4.1")
# Or use openai.Realtime() for realtime model
@llm.register_function(
name="get_weather",
description="Get the current weather for a given city"
)
async def get_weather(city: str) -> dict:
"""Get weather information for a city."""
return {
"city": city,
"temperature": 72,
"condition": "Sunny"
}
# The function will be automatically called when the model decides to use it
Requirements
- Python 3.10+
- GetStream account for video calls
- Open AI API key
Links
License
MIT
2 maintainers
Metadata
2 maintainers
Release files for vision-agents-plugins-openai 0.6.9
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| vision_agents_plugins_openai-0.6.9.tar.gz | 34.6 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| vision_agents_plugins_openai-0.6.9-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 62.3 kB
Release files / vision_agents_plugins_openai-0.6.9.tar.gz
| Download URL | vision_agents_plugins_openai-0.6.9.tar.gz |
|---|---|
| Size | 34.6 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
108195e0ad4e4af40030d79309d3ccbe675fa75404ccb599f7c785eeee72064f
|
|
BLAKE2b-256 checksum How to use checksums |
aef4d4b0eb560075e5ffd20ec092b9234de1ee52a6fd908509bdfb5023f1a0da
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
uv/0.10.10 {"installer":{"name":"uv","version":"0.10.10","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"macOS","version":null,"id":null,"libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}
|
Release files / vision_agents_plugins_openai-0.6.9-py3-none-any.whl
| Download URL | vision_agents_plugins_openai-0.6.9-py3-none-any.whl |
|---|---|
| Size | 27.7 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
bbeb6b0f6ff1781700ff3787d1dac665b070c7ea9132b567afb0e2fb3d3558dd
|
|
BLAKE2b-256 checksum How to use checksums |
541d83c095730a9c2232b9a247c02eb8f8d75305df98cd45c1c293dee2d9a382
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
uv/0.10.10 {"installer":{"name":"uv","version":"0.10.10","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"macOS","version":null,"id":null,"libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}
|
