Inworld AI Plugin
Inworld AI integration for Vision Agents. Provides:
- LLM / VLM — text and vision chat completions through the Inworld Realtime Router, which proxies upstream across OpenAI / Anthropic / Google / etc. with auto-selection, fallbacks, and traffic splitting.
- TTS — high-quality streaming text-to-speech.
- Realtime — WebRTC speech-to-speech for low-latency voice agents.
Installation
uv add "vision-agents[inworld]"
# or directly
uv add vision-agents-plugins-inworld
Get your API key from the Inworld Portal and set
INWORLD_API_KEY in your environment (or pass api_key= explicitly).
LLM / VLM (router)
inworld.LLM and inworld.VLM hit Inworld's OpenAI-compatible
/v1/chat/completions endpoint. The model argument accepts:
"inworld/<router-id>"— a router defined in the Inworld portal"<provider>/<model-id>"— e.g."openai/gpt-4o-mini""auto"— let Inworld pick (combine withsort_by)
from vision_agents.plugins import inworld
# Lowest-latency routing with a small fallback chain
llm = inworld.LLM(
model="auto",
sort_by=["latency"],
ttft_timeout="500ms",
fallback_models=["openai/gpt-4o-mini", "google-ai-studio/gemini-2.5-flash"],
)
# Vision over the router (frames sent as image_url content).
# Tuned for low-latency video Q&A: small frames, short buffer, fast fallback.
vlm = inworld.VLM(
model="auto",
sort_by=["latency"],
ttft_timeout="500ms",
fallback_models=["google-ai-studio/gemini-2.5-flash", "openai/gpt-4o-mini"],
fps=1,
frame_buffer_seconds=3,
frame_width=512,
frame_height=384,
)
See example/inworld_llm_example.py and example/inworld_vlm_example.py
for end-to-end voice and video agents respectively.
Tuning the VLM for latency
Video Q&A latency is dominated by input tokens (frames cost a lot more than text) and upstream choice. The example values above are the right starting point:
frame_width=512, frame_height=384— 4× fewer bytes than the 800×600 default, with negligible accuracy loss for typical Q&A.frame_buffer_seconds=3withfps=1— 3 frames/request. Longer buffers inflate input tokens quadratically without helping short-horizon questions.sort_by=["latency"]+ttft_timeout="500ms"+ a fallback chain to fast vision models keeps TTFT predictable when one provider degrades.
When to use which
- Voice agents →
inworld.Realtime(WebRTC, full-duplex, lowest latency). The text router cannot beat full-duplex audio for STT→LLM→TTS pipelines. - Text agents, STT→LLM→TTS pipelines, video Q&A →
inworld.LLM/inworld.VLM.
Routing kwargs
fallback_models: ordered list, tried on failure.ignore_models: excluded fromauto.sort_by: any of"price","latency","throughput","intelligence","math","coding". Multiple metrics rank with tiebreakers.ttft_timeout: switch to fallback if first token doesn't arrive in time (Inworld minimum"300ms").metadata: free-form dict consumed by router CEL expressions for conditional routing.web_search/web_search_options: opt-in upstream web grounding.compression_aggressiveness(0–1): Inworld's prompt compression applied to the system message — cuts input tokens, lowers TTFT for long prompts.extra_body: raw escape hatch merged in last.
Caching
Implicit prompt caching is automatic on OpenAI / DeepSeek / Gemini-2.5
upstreams — no code needed. Explicit caching (Anthropic / Google) is a
per-message thing: add a cache_control block to the message content
yourself, e.g. {"type": "text", "text": "...", "cache_control": {"type": "ephemeral"}}.
Router definitions themselves (router IDs, A/B variants, traffic weights) are configured in the Inworld portal — out of scope for this plugin.
TTS
High-quality text-to-speech with streaming support. The plugin now defaults
to Inworld's TTS-2 model (currently in research preview), which adds
natural-language steering, 100+ languages (15 GA, 90+ experimental), and
high-quality instant voice cloning over the previous inworld-tts-1.5-*
generation.
from vision_agents.plugins import inworld
# Defaults to model_id="inworld-tts-2", voice_id="Sarah"
tts = inworld.TTS()
# Or specify explicitly
tts = inworld.TTS(
api_key="your_inworld_api_key",
voice_id="Ashley",
model_id="inworld-tts-2",
temperature=1.1,
)
TTS options
api_key: Inworld AI API key (default: reads fromINWORLD_API_KEY)voice_id: Voice to use (default:"Sarah";"Dennis","Ashley","Olivia","Clive"and custom/cloned voices also supported)model_id:"inworld-tts-2"(default),"inworld-tts-1.5-max","inworld-tts-1.5-mini"."inworld-tts-1"and"inworld-tts-1-max"are deprecated by Inworld — migrate toinworld-tts-2orinworld-tts-1.5-*.temperature: 0–2 (default: 1.1)
The plugin requests LINEAR16 (16-bit PCM WAV) chunks from Inworld so each
streamed chunk is self-contained and decodes cleanly under streaming TTS;
no extra configuration needed.
Steering (TTS-2)
TTS-2 takes natural-language stage directions inline with your text. Place the instruction in square brackets before the segment it should apply to:
text = (
"[whisper in a hushed style] I have to tell you something. "
"[laugh] Just kidding! [say with force] Now let's get to work."
)
async for chunk in await tts.stream_audio(text):
...
Steering covers articulation, intonation, volume, pitch, range, speed, and
vocal style — and supports non-verbal sounds like [laugh], [breathe],
[clear throat], [sigh], [cough], [yawn]. Combining dimensions
([whisper in a hushed style], [say playfully and very fast]) produces
better results than bare single-word tags. See Inworld's
steering docs and
prompting guide
for the full reference.
Agent example
A complete example wiring inworld.TTS() into a Stream-edge agent with
Deepgram STT, Gemini LLM, and smart-turn detection lives at
example/inworld_tts_example.py. The
companion example/inworld-audio-guide.md
is loaded as the agent's system prompt and teaches the LLM how to emit
TTS-2 steering tags so replies sound expressive out of the box.
Realtime (WebRTC)
Low-latency speech-to-speech via Inworld's Realtime API. This transport uses
WebRTC (UDP, native Opus) for lower latency than the WebSocket alternative.
Requires a WebRTC-capable edge transport — pair with getstream.Edge() as
shown below.
from vision_agents.core import Agent, User
from vision_agents.plugins import getstream, inworld, smart_turn
agent = Agent(
edge=getstream.Edge(),
agent_user=User(name="My Agent", id="agent"),
llm=inworld.Realtime(
model="openai/gpt-4o-mini",
voice="Dennis",
instructions="You are a friendly voice assistant.",
),
turn_detection=smart_turn.TurnDetection(),
)
Realtime options
model: provider-prefixed model ID. Examples:"openai/gpt-4o-mini"(default),"google-ai-studio/gemini-2.5-flash","inworld/<router-id>"for an Inworld routervoice: voice for audio responses (default:"Dennis";"Clive","Olivia"and custom voices also supported)api_key: Inworld AI API key (default: reads fromINWORLD_API_KEY)instructions: system promptrealtime_session: advanced — pass a fullRealtimeSessionCreateRequestParamfor session fields not exposed by the primary args (custom turn-detection,tool_choice, etc.)
Registering tools
realtime = inworld.Realtime()
@realtime.register_function(description="Get the current weather for a city.")
async def get_weather(city: str) -> str:
return f"It's sunny in {city}."
Tools follow the OpenAI function-calling schema. Inworld's Realtime API is
protocol-compatible with OpenAI's Realtime API, so registered functions flow
through the same response.function_call_arguments.done path.
Notes
- v1 is WebRTC only; a WebSocket transport may be added later.
- Video input is not currently supported by Inworld's Realtime API.
Requirements
- Python 3.10+
httpx>=0.28,av>=10,aiortc>=1.9,openai[realtime]>=2.26,<3
Metadata
Release files for vision-agents-plugins-inworld 0.6.9
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| vision_agents_plugins_inworld-0.6.9.tar.gz | 19.4 kB | Without provenance | Details for vision_agents_plugins_inworld-0.6.9.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| vision_agents_plugins_inworld-0.6.9-py3-none-any.whl | Python 3 | none | any | Details for vision_agents_plugins_inworld-0.6.9-py3-none-any.whl |
Total release size: 42.6 kB
Release files / vision_agents_plugins_inworld-0.6.9.tar.gz
| Download URL | vision_agents_plugins_inworld-0.6.9.tar.gz |
|---|---|
| Size | 19.4 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
4f84fe82586b51a62d59aeaf1a24d4f97316fc225c8f84362ec27eff9d37acf2
|
|
BLAKE2b-256 checksum How to use checksums |
fdc242598d1c356422189f253c716b9fae49a708c433ae77dd0f1fd39ac382c2
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
uv/0.10.10 {"installer":{"name":"uv","version":"0.10.10","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"macOS","version":null,"id":null,"libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}
|
Release files / vision_agents_plugins_inworld-0.6.9-py3-none-any.whl
| Download URL | vision_agents_plugins_inworld-0.6.9-py3-none-any.whl |
|---|---|
| Size | 23.2 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
01464fb59686743a99a8d3693c95a28ba9f37be23dc79b64c07e30fce04c59b2
|
|
BLAKE2b-256 checksum How to use checksums |
befb7456b6b65a27086efcd8d4edd40956078b5eb0b5871703ddac817f08b95b
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
uv/0.10.10 {"installer":{"name":"uv","version":"0.10.10","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"macOS","version":null,"id":null,"libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}
|
Security for vision-agents-plugins-inworld 0.6.9
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.