Skip to main content

LiveKit Evals

PyPI version Python 3.9+ License: MIT

Track and evaluate your LiveKit voice AI agents with just 3 lines of code.

Automatically capture transcripts, usage metrics, latency data, and session analytics from your LiveKit agents. Perfect for monitoring, debugging, and optimizing your voice AI applications.

✨ Features

  • 🎯 3-Line Integration - Add to any LiveKit agent in seconds
  • 📝 Precise Transcripts - Accurate timing using VAD state change events
  • 📊 Usage Metrics - Track LLM tokens, STT duration, TTS characters
  • ⚡ Latency Tracking - Monitor LLM, STT, and TTS performance
  • 🔍 Auto-Detection - Automatically extracts models, providers, and configuration
  • 📞 SIP Support - Detects SIP trunking and phone numbers
  • 🎥 Recording URLs - Captures egress recording links
  • 🎙️ Call Recordings - Automatic call recording to S3 (MP3 format, enabled by default, no S3 config needed)
  • 🔊 Stereo Recording - Dual-channel recording with agent on left, caller on right (one param)
  • 🗂️ Custom Data - Attach arbitrary JSON to every webhook payload via custom_data
  • 🔐 Secure - API key authentication; temporary S3 credentials fetched per-session
  • ⏱️ Turn-Detection Latency - End-of-utterance (EOU) + transcription delays per turn, the hidden part of response latency
  • 🌐 Network Telemetry - Connection quality, reconnects, and WebRTC stats (jitter, RTT, packet loss, audio dropouts)
  • 🚨 Provider Errors - Typed STT/LLM/TTS failures with recoverability, model, and provider
  • 🗣️ Conversation Analytics - Talk ratio, silence, interruptions, backchannels, DTMF keypresses, full SIP attributes

🚀 Quick Start

Prerequisites

  1. Get your API key from https://app.superbryn.com/api-keys
  2. Set environment variable:
    export SUPERBRYN_API_KEY=your_api_key_here
    

Installation

pip install -U "livekit-evals>=0.3.0"

Integration (3 Lines)

Add these lines to your LiveKit agent:

from livekit_evals import create_webhook_handler

async def entrypoint(ctx: JobContext):
    # ... your existing setup code ...
    
    # 1. Create webhook handler (recording enabled by default)
    webhook_handler = create_webhook_handler(
        room=ctx.room,
        is_deployed_on_lk_cloud=True,  # Set to False if self-hosting
        # disable_recording=True  # Uncomment to disable call recording
    )
    
    # ... create your session ...
    session = AgentSession(
        llm=openai.LLM(model="gpt-4o-mini"),
        stt=deepgram.STT(model="nova-3"),
        tts=cartesia.TTS(voice="..."),
    )
    
    # ... your session setup ...
    await session.start(agent=YourAgent(), room=ctx.room)
    
    # 2. Attach to session (MUST be after session.start)
    if webhook_handler:
        webhook_handler.attach_to_session(session)
        # 3. Send webhook on shutdown
        ctx.add_shutdown_callback(webhook_handler.send_webhook)
    
    await ctx.connect()

That's it! 🎉 Your agent will now automatically track all session data and send it to your webhook endpoint.

📖 Full Example

Here's a complete working example:

import logging
from dotenv import load_dotenv
from livekit.agents import (
    Agent,
    AgentSession,
    JobContext,
    WorkerOptions,
    cli,
)
from livekit.plugins import cartesia, deepgram, openai, silero

# Import livekit-evals
from livekit_evals import create_webhook_handler

logger = logging.getLogger("agent")
load_dotenv()


class Assistant(Agent):
    def __init__(self) -> None:
        super().__init__(
            instructions="""You are a helpful voice AI assistant.
            You eagerly assist users with their questions.""",
        )


async def entrypoint(ctx: JobContext):
    # Logging setup
    ctx.log_context_fields = {"room": ctx.room.name}

    # Initialize webhook handler (auto-detects all metadata)
    webhook_handler = create_webhook_handler(
        room=ctx.room,
        is_deployed_on_lk_cloud=True  # Set to False if self-hosting
    )

    # Set up voice AI pipeline
    session = AgentSession(
        llm=openai.LLM(model="gpt-4o-mini"),
        stt=deepgram.STT(model="nova-3", language="en"),
        tts=cartesia.TTS(voice="your-voice-id"),
        vad=silero.VAD.load(),
    )

    # Start the session
    await session.start(agent=Assistant(), room=ctx.room)

    # Attach webhook handler to capture events
    # IMPORTANT: Must be after session.start()
    if webhook_handler:
        webhook_handler.attach_to_session(session)
        ctx.add_shutdown_callback(webhook_handler.send_webhook)

    # Connect to room
    await ctx.connect()


if __name__ == "__main__":
    cli.run_app(WorkerOptions(entrypoint_fnc=entrypoint))

🔧 Configuration

Environment Variables

Variable Required Description Default
SUPERBRYN_API_KEY ✅ Yes API key for webhook authentication and call recording -
LIVEKIT_PROJECT_ID ⚪ Optional LiveKit project ID Auto-detected from LIVEKIT_URL
AGENT_ID ⚪ Optional Unique agent identifier "livekit-agent"
VERSION_ID ⚪ Optional Agent version identifier "v1"

Note: Call recording is enabled by default. Temporary S3 credentials are fetched automatically using your SUPERBRYN_API_KEY -- no S3 configuration needed.

Setting Environment Variables

Linux/Mac:

export SUPERBRYN_API_KEY=your_api_key_here

Windows (CMD):

set SUPERBRYN_API_KEY=your_api_key_here

Windows (PowerShell):

$env:SUPERBRYN_API_KEY="your_api_key_here"

Docker:

docker run -e SUPERBRYN_API_KEY=your_api_key_here ...

.env file:

SUPERBRYN_API_KEY=your_api_key_here
LIVEKIT_PROJECT_ID=my-project-id
AGENT_ID=my-agent
VERSION_ID=v1.0.0

📊 What Gets Tracked

Transcript Data

  • Precise timing using VAD state change events
  • Speaker turns (user/assistant)
  • Start/end timestamps (ISO 8601)
  • Start/end times in milliseconds (relative to call start)
  • Response delays between turns
  • Interruption detection
  • Confidence scores (when available)
  • Language detection
  • Speaker IDs

Tool Calls

  • Function name and raw JSON arguments for every tool invoked by the agent
  • Tool result (output string) paired with each call
  • Error flag (is_error) when the tool execution raised an exception
  • Timing — start_ms (call dispatched) and end_ms (result received), both as ms offsets from call start

Usage Metrics

  • LLM: Input tokens, output tokens, total tokens, model, provider
  • STT: Audio duration, model, provider
  • TTS: Character count, audio duration, model, provider, voice ID

Latency Metrics

  • LLM: Time to first token (TTFT), total duration
  • STT: Processing duration
  • TTS: Time to first byte (TTFB), total duration
  • Aggregated: Average latencies per component

Session Metadata

  • Agent ID and version
  • LiveKit project ID
  • System prompt
  • Call duration
  • Phone number (if SIP call)
  • SIP trunking detection
  • Egress recording URLs
  • LiveKit Cloud deployment status

Extended Telemetry (on by default)

Everything below ships as additive sections in the same payload — all previously existing fields are unchanged. Disable with extended_capture=False for the legacy shape. Older livekit-agents versions simply produce fewer fields (each capture is feature-detected; nothing breaks).

Section What's inside
call.turn_detection Per-turn end-of-utterance delay, transcription delay, end-of-turn model inference stats
call.latency (new keys) eou_ms, transcription_ms, e2e_ms (EOU + LLM TTFT + TTS TTFB — true perceived latency)
call.speech_stats User/agent talk seconds, turn counts, talk ratio, silence, longest gap, response delays; realtime models also speech_created_count and cancelled_responses
call.transcript.turns[] (realtime models) Agent turns carry provider_request_id, and turn_latency.llm_ms is that reply's own time to first token (0 when not measured)
call.interruptions Interrupted agent turns, false interruptions, detected interruptions + backchannels (SDK ≥1.6)
call.network Connection quality timeline, reconnects, disconnect reasons, WebRTC stats polled every 10s (jitter, RTT, packet loss %, concealed samples, NACKs, bytes)
call.errors Typed provider failures (stt_error/llm_error/tts_error), message, recoverable, model, provider
call.close LiveKit's own close reason + terminal error (separate from your call_end_reason)
call.sip Full sip.* attribute snapshot + DTMF keypresses with timestamps
call.vad VAD inference count/latency and idle time
call.environment livekit-agents / livekit / livekit-evals / Python versions, RTC stats support flag
call.usage (new keys) LLM cached prompt tokens + tokens/sec, STT/TTS connection acquire times, realtime-model audio/text/cached token splits

Example of the extended sections:

{
  "call": {
    "latency": { "llm_ms": 450.5, "stt_ms": 120.3, "tts_ms": 180.7, "total_ms": 751.5,
                 "eou_ms": 420.0, "transcription_ms": 95.0, "e2e_ms": 1051.2 },
    "turn_detection": {
      "avg_eou_delay_ms": 420.0, "max_eou_delay_ms": 610.2, "avg_transcription_delay_ms": 95.0,
      "eot_inference_count": 14, "avg_eot_prediction_ms": 81.5,
      "eou_events": [ { "timestamp_ms": 5210, "eou_delay_ms": 402.1, "transcription_delay_ms": 88.0, "on_user_turn_completed_delay_ms": 1.2 } ]
    },
    "speech_stats": {
      "user_talk_seconds": 41.2, "agent_talk_seconds": 88.6, "user_turn_count": 12, "agent_turn_count": 13,
      "avg_user_turn_seconds": 3.4, "avg_agent_turn_seconds": 6.8, "agent_talk_ratio": 0.683,
      "silence_seconds": 20.2, "longest_silence_ms": 4100,
      "avg_response_delay_ms": 910.4, "max_response_delay_ms": 2210.0
    },
    "interruptions": { "agent_turns_interrupted": 2, "false_interruptions": 0,
                       "detected_interruptions": 2, "backchannels": 5, "detection_delay_ms": 150.0 },
    "network": {
      "worst_connection_quality": "poor",
      "connection_quality_events": [ { "timestamp_ms": 61200, "participant": "+12025551234", "quality": "poor" } ],
      "reconnect_count": 1,
      "connection_events": [ { "type": "reconnecting", "timestamp_ms": 63400 }, { "type": "reconnected", "timestamp_ms": 64100 } ],
      "track_subscription_failures": 0,
      "room_disconnect_reason": "client_initiated",
      "rtc": {
        "summary": { "samples_collected": 15, "avg_jitter_ms": 12.0, "max_jitter_ms": 41.5,
                     "avg_rtt_ms": 70.0, "max_rtt_ms": 180.2, "packets_received": 14200, "packets_lost": 36,
                     "packet_loss_pct": 0.25, "concealed_samples": 4800, "silent_concealed_samples": 1200,
                     "nack_count": 6, "bytes_received": 2400000, "bytes_sent": 2300000,
                     "avg_jitter_buffer_delay_ms": 42.1, "remote_fraction_lost_max": 0.02 },
        "samples": [ { "t_ms": 10000, "jitter_ms": 9.5, "rtt_ms": 65.0, "packets_lost": 4 } ]
      }
    },
    "errors": [ { "timestamp_ms": 84200, "error_type": "stt_error", "message": "deepgram websocket closed: 1011",
                  "recoverable": true, "label": "deepgram.STT", "source_model": "nova-3", "source_provider": "deepgram" } ],
    "close": { "reason": "participant_disconnected", "error_type": null, "error_message": null },
    "sip": { "attributes": { "sip.callID": "abc-123", "sip.callStatus": "hangup", "sip.phoneNumber": "+12025551234" },
             "dtmf": [ { "timestamp_ms": 32000, "digit": "1", "code": 1 } ] },
    "vad": { "inference_count": 4210, "avg_inference_ms": 1.9, "total_inference_seconds": 8.1, "idle_time_seconds": 3.2 },
    "environment": { "livekit_evals_version": "0.2.14", "livekit_agents_version": "1.6.5",
                     "livekit_rtc_version": "1.1.13", "python_version": "3.12.12", "rtc_stats_supported": true }
  }
}

🔍 How It Works

  1. Event Listening: Attaches to LiveKit session events (user_state_changed, agent_state_changed, conversation_item_added) and to the per-plugin metrics_collected events on STT/LLM/TTS (the non-deprecated metrics surface), with session_usage_updated as a fallback for realtime models. With extended capture (default) it also listens to the session error / agent_false_interruption events, the session-level metrics_collected re-emit (the only surface carrying VAD/EOU/interruption metrics — a one-line deprecation warning in logs is expected), room events (connection_quality_changed, reconnecting/reconnected, sip_dtmf_received, participant_attributes_changed, ...), and polls room.get_rtc_stats() every 10s
  2. Data Aggregation: Collects and processes events during the session
  3. Auto-Detection: Extracts configuration from session objects
  4. Webhook Delivery: Sends comprehensive payload to webhook endpoint when session ends

Webhook Payload Format

{
  "event": "call.ended",
  "call": {
    "id": "room-name",
    "room_name": "room-name",
    "participant_identity": "user-123",
    "started_at": "2025-10-19T12:00:00.000Z",
    "ended_at": "2025-10-19T12:05:30.000Z",
    "duration_seconds": 330,
    "transcript": {
      "turns": [
        {
          "speaker": "user",
          "text": "Hello, how are you?",
          "timestamp": "2025-10-19T12:00:05.000Z",
          "start_timestamp": "2025-10-19T12:00:05.000Z",
          "end_timestamp": "2025-10-19T12:00:07.000Z",
          "start_time_ms": 5000,
          "end_time_ms": 7000,
          "interrupted": false,
          "confidence_score": 0.98,
          "language": "en"
        },
        {
          "speaker": "assistant",
          "text": "I'm doing great, thanks for asking!",
          "timestamp": "2025-10-19T12:00:08.000Z",
          "start_timestamp": "2025-10-19T12:00:08.000Z",
          "end_timestamp": "2025-10-19T12:00:11.000Z",
          "start_time_ms": 8000,
          "end_time_ms": 11000,
          "response_delay_ms": 1000,
          "interrupted": false
        }
      ]
    },
    "tool_calls": [
      {
        "id": "call_abc123",
        "function_name": "book_appointment",
        "arguments": "{\"date\": \"2026-06-20\", \"time\": \"10:00\"}",
        "result": "{\"confirmation_id\": \"APT-9001\", \"status\": \"confirmed\"}",
        "is_error": false,
        "start_ms": 12500,
        "end_ms": 13800,
        "timestamp_ms": 12500
      }
    ],
    "recording_url": "https://...",
    "stereo_recording_url": "https://...",
    "metadata": {
      "agent_id": "my-agent",
      "livekit_project_id": "my-project",
      "llm_model": "gpt-4o-mini",
      "llm_provider": "openai",
      "stt_model": "nova-3",
      "stt_provider": "deepgram",
      "tts_model": "sonic-english",
      "tts_provider": "cartesia",
      "tts_voice_id": "...",
      "system_prompt": "You are a helpful assistant...",
      "sip_trunking_enabled": false,
      "egress_enabled": true,
      "lk_agent_enabled": true,
      "phone_number": null
    },
    "usage": {
      "llm_model": "gpt-4o-mini",
      "llm_provider": "openai",
      "llm_input_tokens": 1250,
      "llm_output_tokens": 850,
      "llm_total_tokens": 2100,
      "stt_provider": "deepgram",
      "stt_model": "nova-3",
      "stt_duration_seconds": 45.2,
      "audio_duration_seconds": 45.2,
      "tts_provider": "cartesia",
      "tts_model": "sonic-english",
      "tts_characters": 1200,
      "tts_audio_duration_seconds": 42.5
    },
    "latency": {
      "llm_ms": 450.5,
      "stt_ms": 120.3,
      "tts_ms": 180.7,
      "total_ms": 751.5
    },
    "custom_data": {
      "ticket_id": "TKT-9001",
      "customer_tier": "enterprise"
    }
  }
}

🛠️ Advanced Usage

Prompt Sync

Keeps SuperBryn in step with the prompt your agent runs. Pass the prompt template (placeholders unfilled):

PROMPT = "You are a helpful voice AI assistant for {{company}}."

webhook_handler = create_webhook_handler(
    room=ctx.room,
    is_deployed_on_lk_cloud=True,
    prompt=PROMPT,            # the template your agent runs
    prompt_version="v13",     # optional: your label for this version
)
  • Every call carries metadata.prompt_ref = {"hash": "sha256:...", "version": "v13"}, and the template's text is pushed to SuperBryn once per process per version (POST /public-api/v1/prompts).
  • When you deploy a changed prompt, the agent's settings in SuperBryn show the change and offer to make it a new version. A prompt saved but never deployed never shows up, because no call names it.
  • Prompt sync never holds the webhook back: if the push fails, the call is still sent and is matched once the prompt text arrives.
  • The API key needs the obs:write scope. Keys generated in Monitor > Configure have it; with an older key a SUPERBRYN_PROMPT_SYNC_FORBIDDEN warning is logged once.
  • Without prompt=, the agent's instructions are sent as metadata.system_prompt (as before) and read as rendered text, so a change shows up after a few calls instead of one.
  • first_message= and prompt_tools=[{"name", "description", "parameters"}] are pushed with the prompt. sync_prompt=False turns it off.

Disabling Extended Capture

Extended telemetry (turn detection, network stats, errors, SIP detail, ...) is on by default and adds no meaningful overhead — network stats are polled once every 10 seconds and everything else is passive event listening. To emit the exact legacy payload instead:

webhook_handler = create_webhook_handler(
    room=ctx.room,
    is_deployed_on_lk_cloud=True,
    extended_capture=False,  # legacy payload shape only
)

Custom Data

Attach any JSON-serializable fields to the webhook payload using custom_data. These are forwarded verbatim in payload["call"]["custom_data"] and are never interpreted by the package — they're purely for your own downstream use.

At creation time (data known at session startup):

webhook_handler = create_webhook_handler(
    room=ctx.room,
    is_deployed_on_lk_cloud=True,
    custom_data={
        "ticket_id": "TKT-9001",
        "customer_tier": "enterprise",
        "lead_source": "website",
    },
)

During the session (data discovered at runtime, e.g. after a tool call):

# Replace the entire dict
webhook_handler.set_custom_data({"resolved": True, "resolution_code": "answered"})

# Or merge additional keys while keeping existing ones
webhook_handler.update_custom_data({"appointment_booked": True, "slot": "2026-06-05T10:00"})

Both methods can be called at any point before send_webhook() fires on shutdown.

Custom API Key

Pass API key directly instead of using environment variable:

webhook_handler = create_webhook_handler(
    room=ctx.room,
    is_deployed_on_lk_cloud=True,
    api_key="your_api_key_here"
)

Custom LiveKit Project ID

webhook_handler = create_webhook_handler(
    room=ctx.room,
    is_deployed_on_lk_cloud=True,
    livekit_project_id="my-custom-project-id"
)

Self-Hosted Agents

If you're self-hosting your LiveKit agents (not using LiveKit Cloud):

webhook_handler = create_webhook_handler(
    room=ctx.room,
    is_deployed_on_lk_cloud=False  # Important for cost calculation
)

Custom Telephony Rates

If you're using custom telephony providers (Twilio, Vonage, etc.) with specific per-minute rates:

webhook_handler = create_webhook_handler(
    room=ctx.room,
    is_deployed_on_lk_cloud=True,
    call_rate_usd=0.015  # Your custom rate per minute ($/min)
)

This overrides default provider costs and ensures accurate cost tracking for your telephony usage.

Call Recording (Enabled by Default)

Call recording is automatically enabled. Recordings are:

  • ✅ MP3 format (universal compatibility)
  • ✅ Publicly accessible via direct URL
  • ✅ Secured with short-lived credentials (30-minute expiry, scoped per session)
  • ✅ Automatically included in webhook payload

No S3 keys, buckets, or regions need to be configured -- the package fetches temporary upload credentials from SuperBryn's credentials service using your SUPERBRYN_API_KEY.

Recording URLs are included in the webhook payload:

{
  "call": {
    "recording_url": "https://superbryn-call-recordings.s3.ap-south-1.amazonaws.com/call_recordings/+12025551234/20250106-153045/call.mp3"
  }
}

To disable recording:

webhook_handler = create_webhook_handler(
    room=ctx.room,
    is_deployed_on_lk_cloud=True,
    disable_recording=True  # Disable call recording
)

Bring Your Own Egress (External Recording URL)

If you already run your own egress, you can disable SuperBryn's recording and supply your own recording URL instead. Disable our egress with disable_recording=True, then call set_external_recording_url() once your recording is ready (any time before the webhook fires on shutdown):

webhook_handler = create_webhook_handler(
    room=ctx.room,
    is_deployed_on_lk_cloud=True,
    disable_recording=True,  # don't start SuperBryn's egress
)

# ... later, once your own egress has produced a file ...
reachable = await webhook_handler.set_external_recording_url(
    "https://my-bucket.s3.amazonaws.com/calls/room-123/call.mp3?X-Amz-Signature=...",
    # probe=False,  # skip the call-time reachability check
)

Only public or pre-signed URLs are supported. When you call this method the URL is validated at call time with a lightweight ranged GET (Range: bytes=0-0):

  • 200/206 → reachable; the URL is marked usable for mirroring.
  • 401/403 → private object or bad/expired signature — a clear error is logged and mirroring should be skipped.
  • 404 → object not uploaded yet or wrong path (a timing/path issue).

A ranged GET is used instead of HEAD because S3 pre-signed URLs are signed for a single HTTP method — a HEAD on a GET-signed URL returns 403 and would falsely look private. Note the check reflects reachability at call time; a pre-signed URL can still expire before a later (e.g. server-side) mirror runs, so sign for a long-enough TTL or mirror promptly.

The webhook payload gains two fields so the consumer can decide whether to mirror:

{
  "call": {
    "recording_url": "https://my-bucket.s3.amazonaws.com/.../call.mp3?...",
    "recording_url_source": "external",     // "superbryn_s3" for the managed flow
    "recording_url_reachable": true          // call-time probe result (null if not probed)
  }
}

For the default managed flow these are "superbryn_s3" / true — the file is already in SuperBryn's bucket and needs no mirroring.

Stereo Recording (Dual-Channel)

Record in dual-channel stereo where the agent is on the left channel and all other participants (caller/SIP) are on the right channel. This is useful for separate-speaker transcription and analysis.

Stereo is opt-in — the default is mono. Enabling it is a one-line change: add stereo_recording=True. There is nothing else to configure or install — it reuses the exact same recording pipeline (egress + S3 credentials), just switching the egress to DUAL_CHANNEL_AGENT mixing instead of a mono mix.

webhook_handler = create_webhook_handler(
    room=ctx.room,
    is_deployed_on_lk_cloud=True,
    stereo_recording=True  # L=agent, R=caller
)

Prerequisites: the only requirement is that recording already works for you (i.e. you currently get a recording_url in your webhook payload). That requires:

  • SUPERBRYN_API_KEY set (used to fetch temporary S3 credentials — no S3 config needed)
  • LiveKit Egress available on your deployment (enabled by default on LiveKit Cloud; the egress service must be running if self-hosted)
  • Standard LiveKit env vars (LIVEKIT_URL, LIVEKIT_API_KEY, LIVEKIT_API_SECRET)

If recording works today, stereo_recording=True is the only change needed — no version bump, no new dependency. stereo_recording=True also implies recording is enabled (it overrides disable_recording).

When stereo is enabled, both recording_url and stereo_recording_url are populated in the webhook payload. Note: a single MP3 file is written — the stereo separation lives in the file's two channels, so both fields point to the same URL:

{
  "call": {
    "recording_url": "https://...call.mp3",
    "stereo_recording_url": "https://...call.mp3"
  }
}

If you need to stop the recording before deleting the room (e.g. in a graceful shutdown), call stop_egress() to ensure the file is finalized on S3:

await webhook_handler.stop_egress()  # finalize recording before room deletion

Semantic Call End Reasons

If your agent knows the business reason for ending a call, set it explicitly on the WebhookHandler before closing the room. This helps preserve reasons such as transfer_to_human, conversation_complete, caller_hung_up, or no_answer_timeout in the final webhook payload.

Without this, LiveKit may only emit a generic close reason like "participant left" or "session closed".

# Example: transfer to human
webhook_handler.set_call_end_reason("transfer_to_human")
await webhook_handler.stop_egress()
await ctx.api.room.delete_room(...)

You can use any short snake_case reason string that fits your application.

Common examples:

  • conversation_complete
  • purpose_achieved
  • transfer_to_human
  • caller_hung_up
  • main_agent_hung_up
  • no_answer_timeout
  • silence_timeout
  • duration_limit

set_call_end_reason() is most useful when your application logic decides why the call is ending. For example:

  • an end_call tool is invoked by the agent
  • your app triggers a transfer to a human
  • you enforce a silence timeout or no-answer timeout
  • you intentionally delete the room during graceful shutdown

🧪 Testing data (eval mode)

When SuperBryn tests your agent, eval mode sends your agent's own record of each test call to SuperBryn, so you can see it next to the test: its transcript and tool calls, the latency of each reply (transcript, end of turn, first LLM token, first TTS audio, end to end), its models, turn-taking and interruption settings, tools, token and audio usage, every LLM request with what became of its answer (spoken, cut off, not spoken, stopped or only a tool call; not for models such as Nova Sonic that keep one request ID for the whole session), and every STT, LLM or TTS failure. It runs only for SuperBryn test calls, in the background: it adds no latency and records no audio. Every other call sends nothing about the call. To spot phone tests, a SIP call without a test ID checks SuperBryn's caller numbers; they are cached for an hour in a file in the temp dir that every job on the host shares (a failed fetch is retried after a minute).

It needs livekit-evals 0.3.0 or later:

pip install -U "livekit-evals>=0.3.0"

Add two lines to your entrypoint, once per job, before session.start():

from livekit_evals import attach_test_data

async def entrypoint(ctx: JobContext):
    session = AgentSession(...)
    attach_test_data(ctx, session)
    await session.start(agent=YourAgent(), room=ctx.room)
Variable Required Description
SUPERBRYN_TEST_API_KEY ✅ Yes Organization API key with the test_data:write scope. Falls back to SUPERBRYN_API_KEY
SUPERBRYN_AGENT_ID ✅ Yes Your SuperBryn agent ID (a UUID); without it nothing is sent. Leaves AGENT_ID, which Observability uses, untouched
SUPERBRYN_BASE_URL ⚪ Optional Defaults to https://api.superbryn.com

attach_test_data also takes api_key=, agent_id= and base_url=.

How a test call is recognised:

  • Phone tests need nothing more. They are matched by the number SuperBryn calls from.
  • SIP endpoint tests carry an X-SuperBryn-Call-Id header. Set your inbound trunk's include_headers to X headers (SIP_X_HEADERS), or map X-SuperBryn-Call-Id with headers_to_attributes to an attribute named superbryn_call_id (any name ending in superbryn_call_id or x-superbryn-call-id works; other names are not read).
  • Webhook and Custom cURL triggers: copy superbryn_call_id from SuperBryn's trigger request into the job (dispatch) or room metadata as {"superbryn_call_id": "..."}.

Its info, warning and error log lines (logger test_data) start with SUPERBRYN_TEST_DATA_: SUPERBRYN_TEST_DATA_SENT when SuperBryn accepted an event, SUPERBRYN_TEST_DATA_DISABLED when the API key or agent ID is missing, SUPERBRYN_TEST_DATA_RETRY, SUPERBRYN_TEST_DATA_REJECTED and SUPERBRYN_TEST_DATA_FAILED for delivery problems, and SUPERBRYN_TEST_DATA_ERROR for anything else. None of them affects the call.

🐛 Troubleshooting

Webhook Not Sending

Check API Key:

echo $SUPERBRYN_API_KEY

Enable Debug Logging:

import logging
logging.basicConfig(level=logging.DEBUG)

Look for these log messages:

  • SUPERBRYN_WEBHOOK_HANDLER_CREATED - Handler initialized
  • SUPERBRYN_WEBHOOK_SENT - Webhook delivered successfully
  • SUPERBRYN_WEBHOOK_UNAUTHORIZED - Invalid API key
  • SUPERBRYN_WEBHOOK_FAILED - Delivery failed
  • SUPERBRYN_WEBHOOK_ERROR - Exception occurred

Common Errors

Error Cause Solution
SUPERBRYN_API_KEY not configured Missing API key Set SUPERBRYN_API_KEY environment variable
SUPERBRYN_WEBHOOK_UNAUTHORIZED Invalid API key Verify your API key is correct
SUPERBRYN_WEBHOOK_FORBIDDEN Expired/disabled key Generate a new API key
No empty turn found to fill State change timing issue Usually harmless, check logs for patterns

Missing Transcript Data

Ensure webhook_handler.attach_to_session(session) is called:

  • ✅ After await session.start()
  • ✅ At the end of your entrypoint (no early returns)

Provider Detection Issues

The package auto-detects providers from model names. Supported providers (25+):

LLM Providers:

  • OpenAI (gpt, whisper, tts-1, o1, o3)
  • Anthropic (claude)
  • Google (gemini, palm, bard, gemma)
  • Meta (llama, meta-llama)
  • Mistral (mistral, mixtral)
  • Cohere (cohere, command)
  • Perplexity (perplexity, pplx)
  • Groq
  • Together AI (together, togethercomputer)
  • Replicate
  • Hugging Face (huggingface, hf-)

TTS Providers:

  • ElevenLabs (eleven, elevenlabs)
  • Cartesia (cartesia, sonic)
  • PlayHT (playht, play.ht)
  • Resemble AI (resemble, resembleai)
  • Murf (murf, murf.ai)
  • WellSaid Labs (wellsaid, wellsaidlabs)
  • Speechify
  • Sarvam (saarika, sarvam, bulbul)
  • Azure/Microsoft (azure, microsoft)
  • AWS Polly (aws, polly, amazon)
  • Google Cloud (gcloud, google-cloud)

STT Providers:

  • Deepgram (deepgram, nova, aura)
  • AssemblyAI (assemblyai, assembly)
  • Rev.ai (rev.ai, revai)
  • Speechmatics
  • Gladia

Realtime/Multi-modal:

  • LiveKit
  • Twilio
  • Vonage

If your provider isn't detected, it will show as "unknown" but won't affect functionality.

📝 Migration Guide

If you're currently using the standalone webhook_handler.py:

Before:

from webhook_handler import create_webhook_handler

After:

from livekit_evals import create_webhook_handler

Everything else stays the same! The API is identical.

🤝 Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.

💡 Support


Made with ❤️ by SuperBryn

Metadata

Release files for livekit-evals 0.3.4

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for livekit-evals 0.3.4
File Size Uploaded Provenance Actions
livekit_evals-0.3.4.tar.gz 91.5 kB Without provenance Details for livekit_evals-0.3.4.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for livekit-evals 0.3.4
File Interpreter ABI Platform Actions
livekit_evals-0.3.4-py3-none-any.whl Python 3 none any Details for livekit_evals-0.3.4-py3-none-any.whl

Total release size: 162.8 kB

Release files / livekit_evals-0.3.4.tar.gz

Download URL livekit_evals-0.3.4.tar.gz
Size 91.5 kB
Tags Source
SHA-256 checksum
How to use checksums
681a78d1fdae9eedb0443ed8001fae518f110b7fd76d83dec39be0e6ab819f36
BLAKE2b-256 checksum
How to use checksums
fb65d4ef2a9fd805fcbb42150141c9ea7003f0824f02b5ad4f4d2150e34bda65
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.12.15

Release files / livekit_evals-0.3.4-py3-none-any.whl

Download URL livekit_evals-0.3.4-py3-none-any.whl
Size 71.3 kB
Tags Python 3
SHA-256 checksum
How to use checksums
8e0c428974b6946731eba5892908cbc47b23eab07e21226dee9ac6118aeb89e7
BLAKE2b-256 checksum
How to use checksums
edda150521e8dfbfb96ffa445dfd0d8c55eca0af1204fc5867972e46e3b2af8f
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.12.15

Security for livekit-evals 0.3.4

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page