rtpbench

Wire-level latency and conformance tests for voice agents over SIP/RTP.

rtpbench calls a voice agent over SIP, captures the RTP on both ends, and measures turn latency, barge-in and SIP behaviour from the packets themselves, on clean and impaired networks (tc netem). Every measurement carries an error bar calibrated against a reference agent that logs the truth, and every case is proven against that agent, passing and failing: rtpbench cases runs that proof on your host.

Status: early; commands and result formats can still change between releases.

What it found

Three open-source ways to put the OpenAI Realtime API on a SIP call, through 24 cases, with and without a carrier in front, on a mock engine that holds the AI engine's share of each turn fixed:

The counts take every profile, direct and through the carrier. Run direct only, as target NAME --mock does by default, LiveKit fails 4: late-offer fails only through the carrier. All three pass the 3 cases for lost SIP messages, run direct on a clean network. Unsupported is an optional feature declined as its RFC allows, with the call unharmed: LiveKit refuses DTMF by SIP INFO (405), AI Voice Connector CE refuses it (501) and negotiates no RFC 4733 events, and neither takes session timers. LiveKit does negotiate RFC 4733 events, but the digit never reaches the engine: a failure.

• LiveKit keeps sending audio to a held caller in every kind of hold, AI Voice Connector CE when held with c=0.0.0.0, and both answer a re-INVITE that changes codec with the old codec. AVA sends RTP only while it speaks, so whether its media resumes after a hold cannot be judged.
• LiveKit's extra latency is its own media path: 326 ms per turn more than AI Voice Connector CE's, against 322 ms between their turn latencies.
• LiveKit's RTP falls 20 ms behind its own timestamps 4 s into every call, so a receiver needs a 21 ms jitter buffer for it on a clean network; AI Voice Connector CE's needs 1.1 ms.
• AI Voice Connector CE delivers the engine's speech sample for sample. LiveKit re-encodes it on the way, through its WebRTC room, and on a clean network stretches or cuts a pause by 20-40 ms in 6% of answers (10% in a second wave).
• AVA, as its golden configuration ships, cuts the last 0.8 s of every answer, and it never stops for an interruption: it keeps the caller's audio from the engine while it speaks. Over AudioSocket under packet loss, the engine sometimes got silence instead of the caller, so 5 more cases fail now and then.
• AI Voice Connector CE's main branch fixed 4 of the 10 cases it failed in the baseline (a failed transfer no longer hangs up on the caller; compact headers, a bare a=ssrc and SIP over TCP are taken), and now declines a DTMF INFO with 501 where it left it unanswered. It still refuses an INVITE without SDP and answers no re-INVITE without one.

The details, commands, conditions and raw runs are in the baseline, AI Voice Connector CE at b95f5af, Asterisk + AVA and lost SIP messages, and every call's timeline on the results site.

Common ways of measuring turn latency have errors of their own: measured inside the agent, a turn misses the network (80 ms on an LTE path); an energy rule on a recording reads turns long, by 87-110 ms at the median. Each method against the truth: metrology.

Quick start

Needs Linux, Docker 28 or later with Compose, and uv. WSL2 runs it, but its clock can drift and step by far more than the error bars: read what calibrate warns about first (clock steps and rate).

uvx rtpbench calibrate --profiles clean --turns 5     # how exactly this host times the wire
uvx 'rtpbench[vad]' cases                             # prove every case against the reference agent
uvx 'rtpbench[vad]' target livekit --mock             # a bundled target, no AI account needed
uvx 'rtpbench[vad]' target sip:agent@host             # an agent of your own, the same way

The first three take about 1, 17 and 11 minutes, after pulling about 0.4 GB of Docker images (compressed) on first use; every target and the carrier take 1.5 GB (images). The last needs an agent that answers SIP; your own agent says what it needs, such as --network for one in Docker on this host.

Each run writes results/<timestamp>/ with the captures, summary.md and, for cases, report.html: a timeline of every call's SIP, speech and RTP. rtpbench index links every run under results/ from results/index.html; rtpbench COMMAND --help lists a command's options, defaults and valid names.

Documentation

• Calibration: error bars, clock steps, network profiles and their sources
• Conformance cases: what each case checks
• Targets, your own agent, the mock engine and the carrier
• CI: pass or fail your agent's run against a policy

Development

From a clone, uv run rtpbench (uv run --extra vad rtpbench for [vad]) runs the checkout's code, as the docs show it:

uv sync
uv run ruff format && uv run ruff check --fix
uv run mypy
uv run pytest

Licence

Apache-2.0, except targets/aivc/opensips.cfg, derived from AI Voice Connector CE's example and under its GPL-3.0-or-later, and targets/ava/ai-agent.yaml, derived from AVA's golden OpenAI configuration and under its MIT licence.