Microsoft Teams Bridge for ElevenLabs Agents (Python)

Put an ElevenLabs Agent on a real Microsoft Teams call. The hosted StandIn media bridge (standin.komaa.com) joins the Teams call and dials into this bridge over an HMAC-authenticated WebSocket; the bridge opens one ElevenLabs Agent conversation per call and relays between them. You run no Teams media stack yourself.

Microsoft Teams call
       |
       v
StandIn media bridge      (hosted; joins the call)
       |   HMAC WebSocket, PCM 16 kHz
       v
this bridge               (you run it)
       |   WebSocket
       v
ElevenLabs Agent          (STT + LLM + TTS + turn-taking)

The hot path is copy-only: both sides speak base64 PCM 16 kHz mono (pcm_16000), so caller audio and agent audio are relayed verbatim in both directions. No resampling, no re-encoding, no transcoding.

Features

• Realtime voice, end to end - the caller talks to your ElevenLabs agent and hears it reply. Turn-taking, VAD and interruption are the agent's own (server-side); the bridge adds nothing to the latency budget beyond a relay hop.
• Barge-in done right - when the caller interrupts, the bridge cancels playback on the Teams side and drops stale in-flight agent audio by event_id, so no "audio ghosts" play after the cut.
• Per-call personalization - caller name, tenant and call direction are injected as dynamic_variables at conversation start; an optional localized greeting or spoken disclosure rides first_message; per-caller memory uses the caller's AAD id as user_id (guests get none, never a shared identity).
• Vision on demand - a look client tool lets the agent see the caller's camera or screen-share: describe-then-answer via any OpenAI-compatible vision endpoint, or native multimodal upload (recording-gated). See Vision and recording.
• Agent client tools - end_call, express (avatar emotion), show_image (image on the bot's video tile, SSRF-guarded), look.
• Two call governors - a StandIn-side cutoff the bridge speaks a goodbye for, and a bridge-side MAX_CALL_MINUTES hard cap with a deterministic TTS goodbye.
• Observability - GET /healthz for liveness and GET /metrics (Prometheus text format): calls, durations, rejects, relay/drop counters.
• Hardened transport - replay-proof HMAC upgrade, single-use handshake guard, connection caps, payload caps, pre-start timeout, dead-peer detection, graceful SIGTERM drain, and an EL_HOST allowlist so your API key can only be sent to ElevenLabs.

Not yet at parity with the Node.js sibling. Five of its features are not implemented here: the group-call gate (GROUP_CALL_REQUIRE_ADDRESS / GROUP_CALL_WAKE_PHRASES, "speak only when addressed" in a meeting), ambient vision (AMBIENT_VISION_ENABLED, continuous visual awareness), the per-call vision spend cap (MAX_VISION_PER_MINUTE), the no-answer reaper (STALE_CALL_REAPER_SECONDS, ending a call that never went live), and viseme lip-sync. All are on the roadmap below. One environment variable is also named differently: the pairing secret is WORKER_SHARED_SECRET here and BRIDGE_SECRET there. Everything else, including the wire protocol and the remaining variable names, is the same.

Install

pip install elevenlabs-msteams-bridge

Requires Python 3.10+.

Run

This is the whole configuration - three values, all required, no optional keys. Everything else has a default that is already correct. Of the four backends this is the least to stand up: the agent itself lives in the ElevenLabs dashboard, so there is no gateway to run beside the bridge. Put them in a .env file in the working directory, which is loaded automatically (an existing environment variable always wins):

# Your ElevenLabs account key.
ELEVENLABS_API_KEY=sk_...

# The Conversational AI agent this bridge speaks for. Create it at elevenlabs.io first -
# its prompt, voice and tools are configured there, not here.
ELEVENLABS_AGENT_ID=agent_...

# The connection secret from the StandIn portal. Must byte-match, or the HMAC handshake is
# rejected with 401 - which looks, from the caller's side, like the bot simply never answering.
WORKER_SHARED_SECRET=paste-the-value-from-the-StandIn-portal

Then run it:

elevenlabs-msteams-bridge

The bridge listens on :8080 (override with PORT) and binds 0.0.0.0. It takes the last path segment as the call id, so it accepts whatever path you register in the portal - StandIn appends /{callId} per call. Expose the port with a tunnel and register the public wss:// URL as your identity's Agent voice URL in the StandIn dashboard, never the local ws:// bind.

Your ElevenLabs agent's audio input and output format must be PCM 16000 Hz - the bridge ends the call with a clear error if the agent negotiates anything else.

Embed

import asyncio
from elevenlabs_msteams_bridge import load_config, start_server

async def main():
    server = await start_server(load_config())
    await asyncio.Event().wait()  # run until cancelled

asyncio.run(main())

Pass your own async vision callable to answer the agent's look tool with any model you like - the raw frame never leaves your process:

async def describe(frame: dict, question: str) -> str:
    ...  # call your vision model with frame["dataBase64"] / frame["mime"]
    return "a person holding a badge"

server = await start_server(load_config(), vision=describe)

A complete runnable project lives in examples/basic-bridge/.

Configuration

Every setting is an environment variable, and .env.example ships fully commented with the package.

Configuration reference documents all of them: what each does, its default, and when to change it.

Three that catch people out:

• WORKER_SHARED_SECRET must byte-match the pairing secret from the StandIn portal. A mismatch is rejected with 401, which from the caller's side looks like the bot simply never answering.
• The agent's audio in/out format must be pcm_16000 (agent settings). The bridge validates the conversation metadata at call start and ends the call on mismatch - anything else means garbled audio.
• conversation_config_override fields (first message, prompt, voice) are rejected by ElevenLabs unless allowlisted in the agent's security settings.

Endpoints

• GET /healthz - liveness.
• GET /metrics - Prometheus counters (calls, rejections, relayed/dropped frames).
• GET /{...}/{callId} + WebSocket upgrade - the worker wire, HMAC-signed with X-StandIn-Timestamp / X-StandIn-Signature over "{timestampMs}.{callId}".

Notes for operators:

• /healthz and /metrics are unauthenticated (only the WebSocket upgrade is HMAC-gated). They expose no call content, just liveness and counters, but if you would rather not leak call volumes, keep the port behind your ingress/tunnel rules.
• One bridge process serves one agent id (ELEVENLABS_AGENT_ID). Run one process per agent if you route multiple agents.

Vision and recording

The look tool prefers your VISION_API_URL endpoint: the frame is described transiently and only the text enters the conversation. Without one, the bridge falls back to uploading the frame to ElevenLabs (multimodal) - that persists the frame with a third party, so it is only allowed while Teams recording is active. Note that even path-2 descriptions become ElevenLabs conversation content, which ElevenLabs retains per your agent's settings; enable the agent's zero-retention mode if callers' surroundings must not be stored.

Roadmap

Where the bridge stands today, and what each of these needs to move:

• Group-call gate: the Node.js sibling can withhold the agent's audio in a meeting until a caller addresses it by name (GROUP_CALL_REQUIRE_ADDRESS, GROUP_CALL_WAKE_PHRASES). Not implemented here yet, so in a group call this bridge relays the agent's audio the whole time.
• Ambient vision: look is on demand, so the agent has to decide to look. Continuous awareness (each scene change described once and injected as non-interrupting context) is not implemented here yet.
• Vision spend cap: there is no per-call sliding-window ceiling on paid vision calls (MAX_VISION_PER_MINUTE). Budget at your vision endpoint instead until it lands.
• No-answer reaper: a worker can authenticate, send session.start, and then never get a live relay. The Node.js sibling ends such a call after STALE_CALL_REAPER_SECONDS; here the dead-peer timer only fires on silence, so a worker that keeps heartbeating holds the call id and a connection slot.
• Viseme lip-sync: the bridge relays audio without estimating a viseme timeline, so the Teams tile shows no mouth movement synchronized to the agent's speech.

License

MIT (c) Komaa DigiTech