Agora Conversational AI Python SDK

The Agora Conversational AI SDK provides convenient access to the Agora Conversational AI APIs, enabling you to build voice-powered AI agents with support for both cascading flows (ASR -> LLM -> TTS) and multimodal flows (MLLM) for real-time audio processing.

Install

pip install agora-agents

Requirements

• Python 3.8+

Quick Start

Start with the Agent builder: create a client with app credentials, pass it to Agent(client=client, ...), choose your ASR, LLM, and TTS providers, then start a session. Omit vendor API keys for supported Agora-managed global models, or provide keys when you want BYOK. Set Agora interaction language with turn_detection.language; provider-specific STT language values remain under asr.params. Ares uses only the REST asr.language value sourced from turn_detection.language.

import os
import time

from agora_agent import (
    Agent,
    Agora,
    Area,
    DeepgramSTT,
    OpenAI,
    MiniMaxTTS,
    expires_in_hours,
)

AGENT_PROMPT = (
    "You are a concise, technically credible voice assistant. "
    "Keep replies short unless the user asks for detail."
)

GREETING = "Hi there! I am your Agora voice assistant. How can I help?"


def start_conversation() -> str:
    app_id = os.environ["AGORA_APP_ID"]
    app_certificate = os.environ["AGORA_APP_CERTIFICATE"]

    client = Agora(
        area=Area.US,
        app_id=app_id,
        app_certificate=app_certificate,
    )

    agent = Agent(client=client, turn_detection={"language": "en-US"}).with_stt(
        DeepgramSTT(
            model="nova-3",
            language="en",
        )
    ).with_llm(
        OpenAI(
            model="gpt-4o-mini",
            system_messages=[{"role": "system", "content": AGENT_PROMPT}],
            greeting_message=GREETING,
            failure_message="Please wait a moment.",
            max_history=50,
            params={
                "max_tokens": 1024,
                "temperature": 0.7,
                "top_p": 0.95,
            },
        )
    ).with_tts(
        MiniMaxTTS(
            model="speech_2_6_turbo",
            voice_id="English_captivating_female1",
        )
    )

    session = agent.create_session(
        channel=f"demo-channel-{int(time.time())}",
        agent_uid="123456",
        remote_uids=["*"],
        name=f"conversation-{int(time.time())}",
        idle_timeout=30,
        expires_in=expires_in_hours(1),
        debug=False,
    )

    return session.start()

Why no token or vendor key in the example?

Agora generates the required ConvoAI REST auth and RTC join tokens automatically when you provide app_id and app_certificate. For supported Agora-managed global models, leave vendor API keys unset; provide keys when you want BYOK. CN MiniMax TTS is not Agora-managed in the same way and typically includes key.

Regional agent builders

Bind the client once with Agent(client=client, ...) and pass vendor classes directly such as OpenAI(...) or MiniMaxTTS(...). The bound client selects the API routing region and provides IDE hints via CNAgent / GlobalAgent, but does not restrict which vendor classes you can use. See [docs/guides/regional-routing.md](./docs/guides/regional-routing.md) for regional examples.

AI Studio pipeline IDs

Use pipeline_id when you want a published AI Studio pipeline to provide the base agent configuration:

import time
# client = Agora(area=Area.US, app_id="...", app_certificate="...")
agent = Agent(
    client=client,
    pipeline_id="studio-pipeline-id",
)

session = agent.create_session(
    channel=f"demo-channel-{int(time.time())}",
    agent_uid="1",
    remote_uids=["100"],
    name=f"conversation-{int(time.time())}",
)

You can override it per session:

import time
session = agent.create_session(
    channel=f"demo-channel-{int(time.time())}",
    agent_uid="1",
    remote_uids=["100"],
    pipeline_id="session-pipeline-id",
    name=f"conversation-{int(time.time())}",
)

AgentKit sends the resolved value as the top-level /join field pipeline_id, not inside properties. Explicit Agent config such as with_llm(), with_tts(), with_stt(), with_mllm(), and advanced_features may send properties fields that override the saved pipeline settings.

BYOK version

Use the same Agent builder shape, but provide credentials explicitly when you want vendor-managed billing and routing instead of Agora-managed models.

agent = Agent(client=client, turn_detection={"language": "en-US"}).with_stt(
    DeepgramSTT(
        api_key=os.environ["DEEPGRAM_API_KEY"],
        model="nova-3",
        language="en",
    )
).with_llm(
    OpenAI(
        api_key=os.environ["OPENAI_API_KEY"],
        base_url="https://api.openai.com/v1/chat/completions",
        model="gpt-4o-mini",
        system_messages=[{"role": "system", "content": AGENT_PROMPT}],
        greeting_message=GREETING,
        max_tokens=1024,
        temperature=0.7,
        top_p=0.95,
    )
).with_tts(
    ElevenLabsTTS(
        key=os.environ["ELEVENLABS_API_KEY"],
        model_id="eleven_flash_v2_5",
        voice_id=os.environ["ELEVENLABS_VOICE_ID"],
        base_url="wss://api.elevenlabs.io/v1",
    )
)

Migrating from agora-agent-server-sdk on PyPI? Use pip install agora-agents; imports stay agora_agent — see changelog migration notes or installation guide.

BYOK

If you want to bring your own vendor credentials instead of using Agora-managed models, use the BYOK guide:

• BYOK Guide

MLLM (Realtime / Multimodal)

Use with_mllm() for OpenAI Realtime, Gemini Live, Vertex AI, or xAI Grok. No STT, LLM, or TTS vendor is needed when MLLM mode is enabled.

from agora_agent import Agent, Agora, Area, OpenAIRealtime
import time

client = Agora(
    area=Area.US,
    app_id=os.environ["AGORA_APP_ID"],
    app_certificate=os.environ["AGORA_APP_CERTIFICATE"],
)

agent = Agent(client=client).with_mllm(
    OpenAIRealtime(
        api_key=os.environ["OPENAI_API_KEY"],
        model="gpt-4o-realtime-preview",
        greeting_message="Hello! Ready to chat.",
    )
)

session = agent.create_session(
    channel=f"demo-channel-{int(time.time())}",
    agent_uid="1",
    remote_uids=["*"],
    name=f"conversation-{int(time.time())}",
)
session.start()

See the MLLM Flow guide for full examples with Gemini Live and Vertex AI.

Documentation

API reference documentation is available here.

Reference

A full reference for this library is available here.

Exception Handling

When the API returns a non-success status code (4xx or 5xx response), a subclass of the following error will be thrown.

from agora_agent.core.api_error import ApiError

try:
    client.agents.start(...)
except ApiError as e:
    print(e.status_code)
    print(e.body)

Pagination

Paginated requests will return a SyncPager or AsyncPager, which can be used as generators for the returned object.

from agora_agent import Agora, Area

client = Agora(
    area=Area.US,
    app_id="your-app-id",
    app_certificate="your-app-certificate",
)
response = client.agents.list(
    appid=client.app_id,
)
for item in response:
    yield item
# alternatively, you can paginate page-by-page
for page in response.iter_pages():
    yield page

# You can also iterate through pages and access the typed response per page
pager = client.agents.list(...)
for page in pager.iter_pages():
    print(page.response)  # access the typed response for each page
    for item in page:
        print(item)

Advanced

Access Raw Response Data

The SDK provides access to raw response data, including headers, through the .with_raw_response property. The .with_raw_response property returns a "raw" client that can be used to access the .headers and .data attributes.

from agora_agent import Agora

client = Agora(
    ...,
)
response = client.agents.with_raw_response.start(...)
print(response.headers)  # access the response headers
print(response.data)  # access the returned object
pager = client.agents.list(...)
print(pager.response)  # access the typed response for the first page
for item in pager:
    print(item)  # access the returned object(s)
for page in pager.iter_pages():
    print(page.response)  # access the typed response for each page
    for item in page:
        print(item)  # access the returned object(s)

Retries

The SDK is instrumented with automatic retries with exponential backoff. A request will be retried as long as the request is deemed retryable and the number of retry attempts has not grown larger than the configured retry limit (default: 2).

A request is deemed retryable when any of the following HTTP status codes is returned:

• 408 (Timeout)
• 429 (Too Many Requests)
• 5XX (Internal Server Errors)

Use the max_retries request option to configure this behavior.

client.agents.start(..., request_options={
    "max_retries": 1
})

Timeouts

The SDK defaults to a 60 second timeout. You can configure this with a timeout option at the client or request level.

from agora_agent import Agora

client = Agora(
    ...,
    timeout=20.0,
)


# Override timeout for a specific method
client.agents.start(..., request_options={
    "timeout_in_seconds": 1
})

Custom Client

You can override the httpx client to customize it for your use-case. Some common use-cases include support for proxies and transports.

import httpx
from agora_agent import Agora

client = Agora(
    ...,
    httpx_client=httpx.Client(
        proxy="http://my.test.proxy.example.com",
        transport=httpx.HTTPTransport(local_address="0.0.0.0"),
    ),
)

Contributing

While we value open-source contributions to this SDK, this library is generated programmatically. Additions made directly to this library would have to be moved over to our generation code, otherwise they would be overwritten upon the next generated release. Feel free to open a PR as a proof of concept, but know that we will not be able to merge it as-is. We suggest opening an issue first to discuss with us!

On the other hand, contributions to the README are always very welcome!