openrtc 0.10.0

Docs · Quick start · Isolation · Routing · API

from openrtc import AgentPool
from my_agents import RestaurantAgent, DentalAgent, SupportAgent

pool = AgentPool()                       # one worker, prewarm once
pool.add("restaurant", RestaurantAgent)  # standard livekit.agents.Agent subclasses
pool.add("dental", DentalAgent)
pool.add("support", SupportAgent)
pool.run()                               # N agents share one Silero VAD + turn detector

A thin multi-agent layer for LiveKit Agents. Register many standard livekit.agents.Agent subclasses on one AgentPool and host them in a single worker: shared prewarm (Silero VAD, turn detector) loads once instead of once per worker, and every incoming call still gets its own AgentSession. OpenRTC never introduces a base class and never sits between you and @function_tool, RunContext, on_enter, on_exit, or the *_node hooks. You change how many workers you run, not how you write an agent.

Why OpenRTC

The default one-worker-per-agent model in livekit-agents reloads the same stack (Python runtime, Silero VAD, turn detector) in every process. OpenRTC answers the questions an operator actually asks:

• How many agents per box? One worker hosts every registered agent, each session an asyncio.Task over a shared JobProcess. The density benchmark clears 50+ concurrent sessions per worker under a 4 GB peak-RSS budget with headroom (see Density).
• Do I rewrite my agents? No. Your Agent subclasses, tools, and provider objects are unchanged; you delete per-worker boilerplate (entrypoint, AgentSession wiring, cli.run_app) and register classes on one pool.
• What does it cost in RAM? Prewarm loads once per worker, not once per agent, so you stop paying resident set for copies you do not need.
• What if I need hard isolation? Pass isolation="process" for the one-subprocess-per-session model with independent crashes and livekit's per-session memory caps.

Features

Full release history: docs/changelog.md.

Quick start

pip install openrtc            # or: uv add openrtc
pip install "openrtc[cli]"     # adds the openrtc CLI (rich + typer)

Requires Python 3.11 to 3.13 (>=3.11,<3.14; the transitive onnxruntime behind Silero and the turn detector has no 3.10 wheels). The base install pulls livekit-agents[openai,silero,turn-detector]>=1.5,<1.7 and watchfiles. Ships a PEP 561 py.typed marker. Set LIVEKIT_URL / LIVEKIT_API_KEY / LIVEKIT_API_SECRET as for any LiveKit worker.

Explicit registration with add() when you want every agent named and configured in one place:

from livekit.agents import Agent
from livekit.plugins import openai
from openrtc import AgentPool


class RestaurantAgent(Agent):
    def __init__(self) -> None:
        super().__init__(instructions="You help callers make restaurant bookings.")


pool = AgentPool(default_llm=openai.responses.LLM(model="gpt-4.1-mini"))
pool.add(
    "restaurant",
    RestaurantAgent,
    stt=openai.STT(model="gpt-4o-mini-transcribe"),
    tts=openai.TTS(model="gpt-4o-mini-tts"),
    greeting="Welcome to reservations.",
)
pool.run()

One file per agent with discover() when you prefer a module per agent and optional @agent_config(...):

from pathlib import Path
from livekit.plugins import openai
from openrtc import AgentPool

pool = AgentPool(
    default_stt=openai.STT(model="gpt-4o-mini-transcribe"),
    default_llm=openai.responses.LLM(model="gpt-4.1-mini"),
    default_tts=openai.TTS(model="gpt-4o-mini-tts"),
)
pool.discover(Path("./agents"))
pool.run()

# agents/restaurant.py
from livekit.agents import Agent
from openrtc import agent_config


@agent_config(name="restaurant", greeting="Welcome to reservations.")
class RestaurantAgent(Agent):
    def __init__(self) -> None:
        super().__init__(instructions="You help callers make restaurant bookings.")

Without @agent_config the agent name defaults to the filename stem, and STT/LLM/TTS/greeting fall back to the pool defaults. Provider slots accept either instantiated plugin objects (openai.STT(...)) or shorthand strings ("openai/gpt-4o-mini-transcribe"), which the LiveKit runtime resolves at session construction. OpenRTC installs a sensible default turn_handling (multilingual turn detector with VAD interruption); override it per agent via session_kwargs. Define classes at module scope so spawn-based worker reload can import them. Depth: openrtc.mintlify.app.

Isolation modes

AgentPool(isolation=...) picks how each session runs inside the worker. Coroutine is the default; pass isolation="process" to opt into the one-subprocess-per-session model.

pool = AgentPool(
    isolation="coroutine",         # default
    max_concurrent_sessions=50,    # advisory backpressure (coroutine only)
    consecutive_failure_limit=5,   # supervisor threshold (coroutine only)
    drain_timeout=30,              # seconds to wait for in-flight sessions on SIGTERM
)

max_concurrent_sessions (50), consecutive_failure_limit (5), and drain_timeout (30) are validated as positive integers. On SIGTERM the worker drains: it stops accepting jobs and waits up to drain_timeout seconds for in-flight sessions before cancelling.

Density

From the stub-workload benchmark (tests/benchmarks/density.py, results in docs/benchmarks/density-v0.1.md; macOS arm64, Python 3.13, single worker). The gate: 50+ concurrent sessions per worker at 4 GB peak RSS with no errors. It passes with headroom.

Process mode instead loads the runtime plus models per session (~3 GB each, see docs/audit-2026-05-02.md), so the same 50 sessions would need ~150 GB; coroutine mode shares one process.

Throughput is the other half. N sessions share one event loop and one GIL, so tests/benchmarks/throughput.py drives real Silero VAD over synthetic 16 kHz PCM and measures steady-state loop p99: it stays well under a 100 ms budget to 100 sessions with flat RSS. Read both as an on-loop-CPU ceiling, not a full-pipeline guarantee: the harness stubs the WebRTC/STT/LLM/TTS path and the ~250 to 400 MB model footprint. Measure on your own hardware before quoting a sessions-per-worker number.

Prove it on your machine (no LiveKit server, no API keys, no model download):

uv run python examples/density_demo.py                # 16 sessions
uv run python examples/density_demo.py --sessions 50  # the gap widens with N

livekit-agents (process per session):   1861 MB total  (116.3 MB/session)
OpenRTC coroutine pool (one process):     195 MB total  ( 12.2 MB/session)
OpenRTC uses 9.5x less memory for the same 16 sessions.

Routing

One worker hosts several agent classes, so each session resolves to one registered name. The chain is evaluated in order, and the first match wins:

1. ctx.job.metadata["agent"]
2. ctx.job.metadata["demo"]
3. room metadata ["agent"]
4. room metadata ["demo"]
5. room-name prefix match (agent name followed by a literal hyphen, e.g. restaurant-call-123)
6. first registered agent (fallback)

Within a source, agent outranks demo. Metadata may be a JSON object string or a mapping; blank strings, non-JSON strings, and JSON scalars are ignored and defer to the next strategy. The room-metadata strategies read ctx.job.room.metadata first (authoritative before ctx.connect(), when ctx.room.metadata is still empty).

A value naming an unregistered agent raises eagerly instead of falling through: ValueError("Unknown agent '<name>' requested via <job metadata|room metadata>."). An empty pool raises RuntimeError("No agents are registered in the pool."). Routing never falls back silently. Full rules: routing docs.

Session observers

Attach external telemetry to every session without subclassing or touching internals. Any object with two async methods satisfies the SessionObserver protocol (structural typing, no base class):

from openrtc import AgentPool, SessionInfo, SessionOutcome


class LoggingObserver:
    async def on_session_start(self, info: SessionInfo, session: object) -> None:
        print(f"live: {info.agent_name} in {info.room_name}")

    async def on_session_end(self, info: SessionInfo, outcome: SessionOutcome) -> None:
        print(f"done: {info.agent_name} -> {outcome.status.value}")


pool = AgentPool(observers=[LoggingObserver()])   # or pool.add_observer(...)

on_session_start receives the live AgentSession (subscribe to its metrics there). on_session_end receives a SessionOutcome with status SUCCESS, FAILED, or CANCELLED, and may fire without a matching start if a session dies before going live. Observer calls are isolated: a slow or raising observer is logged and skipped, never crashing the session. Register before run(); under process isolation an observer must be picklable, so build live resources lazily inside on_session_start.

Hot reload

Edit an agent file while calls are in flight, and OpenRTC swaps every live session to the new class on its next turn. This is something livekit-agents cannot do (each session is its own process); OpenRTC can because the agent class is a shared-memory object.

openrtc dev ./agents        # coroutine mode watches your files; on by default
openrtc dev ./agents --no-watch          # opt out
openrtc dev ./agents --watch-path ./lib  # watch extra paths

On save, the module is re-imported into a fresh namespace and validated (compile + import) before any swap. livekit's update_agent blocks new turns and drains the in-flight one, so the current turn finishes on the old class and the next runs the new, with no dropped audio. Guarantees:

• Rollback-safe. A SyntaxError, ImportError, or missing Agent subclass keeps the running class and logs the error with file:line. A bad save never poisons the pool.

• Loud feedback. Each reload logs [reload] agent.py changed -> swapped N sessions in Xms.

• Opt-out for critical flows. Wrap a block that must not change class mid-flight:

  from openrtc import pin_reload
  
  with pin_reload(ctx.session):
      ...  # payment confirmation, multi-step auth: no swap until this exits
  

Hot reload is coroutine-mode only (process mode runs one subprocess per session). openrtc start never hot reloads. Enable it programmatically with AgentPool(enable_hot_reload=True).

CLI

Install openrtc[cli] to put openrtc on your PATH. Five subcommands mirror the LiveKit Agents shape (start, dev, console, connect, download-files), plus an OpenRTC-only list. Pass the agents directory as the first positional path instead of --agents-dir.

openrtc list ./agents \
  --default-stt openai/gpt-4o-mini-transcribe \
  --default-llm openai/gpt-4.1-mini \
  --default-tts openai/gpt-4o-mini-tts

openrtc start ./agents                           # production worker (after exporting LIVEKIT_*)
openrtc dev   ./agents ./openrtc-metrics.jsonl   # 2nd positional path = --metrics-jsonl

Flags are scoped per command: --json / --plain / --resources on list; --isolation / --max-concurrent-sessions on the worker commands; --no-watch / --watch-path control hot reload on dev; the metrics and dashboard flags on the worker commands and connect. --metrics-jsonl appends one JSON object per line (an envelope of schema_version, kind (snapshot or event), seq, wall_time_unix, and payload), interleaving pool snapshots with session_started / session_finished / session_failed events for tail -f or jq. OpenRTC-only flags are stripped before the handoff to LiveKit's CLI parser. Full flag lists: docs/cli.md.

Architecture

flowchart TB
    LK[LiveKit dispatch] --> POOL[AgentPool: one worker process]
    POOL --> PW[Shared prewarm: Silero VAD plus turn detector, loaded once]
    POOL --> ROUTE{Routing chain}
    ROUTE -->|job metadata| REG[Registered Agent subclasses]
    ROUTE -->|room metadata| REG
    ROUTE -->|room-name prefix| REG
    ROUTE -->|first-registered fallback| REG
    REG --> SESS[Per-session AgentSession: one asyncio.Task each]
    PW --> SESS
    SESS --> OBS[SessionObservers]
    SESS --> SINK[runtime_snapshot plus JSONL metrics sink]

Prewarm runs once as the worker's setup function and caches VAD and turn detector in proc.userdata. For each job the universal entrypoint runs the routing chain, instantiates the chosen Agent subclass, builds an AgentSession from cached defaults plus per-agent overrides, and starts it as a task on the shared loop. Registration data is spawn-safe, so it survives serialization to worker subprocesses. Architecture deep dive.

Public API at a glance

The public surface is exactly openrtc.__all__, 14 names. Everything else is internal and not treated as stable.

AgentPool(...) (all keyword-only, all optional):

Methods: add, discover, list_agents, get, remove, add_observer, run, runtime_snapshot, drain_metrics_stream_events. Read-only properties: isolation, max_concurrent_sessions, consecutive_failure_limit, drain_timeout, enable_hot_reload, server. add() raises on an empty or duplicate name and on an agent_cls that is not a livekit.agents.Agent subclass; direct **session_options override the same keys in session_kwargs.

src/openrtc/
├── __init__.py
├── py.typed
├── core/                  # foundational, flat (pool, config, discovery, wiring)
│   ├── pool.py            # AgentPool facade
│   ├── config.py          # AgentConfig, AgentDiscoveryConfig, agent_config
│   ├── discovery.py       # file-system discovery helpers
│   ├── serialization.py   # spawn-safe config serialization
│   ├── turn_handling.py   # turn-detector integration
│   └── wiring.py          # AgentSession assembly helpers
├── routing/               # base_routing.py + variant siblings + resolver
│   ├── base_routing.py    # RoutingStrategy protocol
│   ├── metadata_routing.py
│   ├── room_prefix_routing.py
│   ├── default_routing.py
│   └── resolver.py        # ordered strategy chain
├── runtime/               # base_runtime.py + variant siblings + registry
│   ├── base_runtime.py    # RuntimeBackend protocol
│   ├── coroutine_runtime.py
│   ├── process_runtime.py
│   ├── coroutine_server.py
│   ├── prewarm.py         # shared prewarm helpers
│   ├── resources.py       # shared resource cache
│   ├── file_watcher.py    # FileWatcher / FileChange, hot reload
│   └── registry.py        # selects active runtime
├── observability/         # base_observer.py + base_sink.py + concretes
│   ├── base_observer.py   # SessionObserver protocol
│   ├── base_sink.py       # metrics sink protocol
│   ├── jsonl_sink.py      # JSONL metrics schema and writer
│   ├── metrics.py         # RuntimeMetricsStore, footprint helpers
│   ├── snapshot.py        # PoolRuntimeSnapshot dataclass
│   ├── resident_set.py    # RSS memory helpers
│   ├── savings.py         # cost-savings estimator
│   └── footprint.py       # per-session memory footprint
├── cli/                   # base_cli.py + variant siblings
│   ├── base_cli.py        # shared Typer args and parameter bundles
│   ├── main_cli.py        # top-level Typer app and subcommands
│   ├── dashboard_cli.py   # Rich dashboard and list output
│   ├── entry_cli.py       # lazy console entry / missing-extra hint
│   ├── livekit_cli.py     # LiveKit argv/env handoff, pool run
│   └── reporter_cli.py    # background metrics reporter thread
└── utils/                 # foundational, flat
    ├── types.py           # ProviderValue and related typing
    └── validation.py      # input validation helpers

Contributing

git clone https://github.com/mahimailabs/openrtc-runtime
cd openrtc-runtime
uv sync --group dev
uv run pytest

Read CONTRIBUTING.md before opening a PR. CI runs Ruff and mypy (strict) alongside the suite, with a combined line + branch coverage gate at 99%.

Community

License

MIT. Fork it, ship it.

Built by Mahimai Raja, founder of Mahimai AI, a voice AI company, in public. Standing on LiveKit Agents.