AssemblyAI Plugin

Streaming Speech-to-Text (STT) plugin for Vision Agents using AssemblyAI's Universal-3 Pro model.

Features

• Real-time streaming transcription via async WebSocket
• Built-in punctuation-based turn detection with configurable silence thresholds
• Streaming diarization — identify speakers in real time
• Native SpeechStarted event support
• Custom prompt and keyterms boosting support
• Sub-300ms time to complete transcript latency
• Built-in reconnection with exponential backoff

Installation

uv add "vision-agents[assemblyai]"
# or directly
uv add vision-agents-plugins-assemblyai

Usage

from vision_agents.plugins import assemblyai

stt = assemblyai.STT(
    speech_model="u3-rt-pro",  # Default model
    sample_rate=16000,
)

With streaming diarization

Enable speaker_labels to identify speakers in a mixed audio stream. Each transcript event will carry a distinct participant per speaker and the raw label in response.other["speaker_label"].

stt = assemblyai.STT(
    speaker_labels=True,
    max_speakers=2,  # optional hint (1-10)
)

With keyterms boosting

stt = assemblyai.STT(
    keyterms_prompt=["AssemblyAI", "Vision Agents"],
)

With custom turn silence thresholds

stt = assemblyai.STT(
    min_turn_silence=100,   # ms before speculative EOT check
    max_turn_silence=1200,  # ms before forcing turn end
)

Configuration

Environment Variables

Set ASSEMBLYAI_API_KEY in your environment or pass api_key to the constructor.

Dependencies

• aiohttp>=3.9.0
• vision-agents

Docs

• https://www.assemblyai.com/docs/streaming/universal-3-pro
• https://www.assemblyai.com/docs/streaming/streaming-diarization-and-multichannel