voice-evals

Reproducible evaluation for voice agents. Score recorded calls, run scripted conversations over WebSocket, and turn transcription, latency, interruptions and task outcomes into CI gates.

Watch the demo · Quick start · Kaggle notebooks · Live probe · Documentation

A voice agent can produce the right words and still respond too slowly, talk over a caller, or miss a correction. voice-evals makes those failures inspectable: a scripted caller talks to your agent, interrupts it, records the session, and exports a dataset that the offline evaluator can score again.

Start without API keys. Replay evaluation and the mock probe run offline. Real calls need your agent's endpoint and a caller TTS provider.

Watch the demo

The demo loops automatically without sound. The scores in it come from synthetic calls. Transcript · Download MP4 with sound · Media credits.

Choose your starting point

Quick start

Python 3.10+. Clone the repository to get the example datasets, then install the package:

git clone https://github.com/Gjusev/voice-evals.git
cd voice-evals
python -m pip install voice-evals
voice-eval run evals/data/demo_calls.jsonl

If you already have a dataset, installing from PyPI is enough: voice-eval run calls.jsonl.

samples=3 failures=0
wer mean=0.0303 max=0.0909
task_completion=0.6667
fact_coverage=0.8333
hallucination_rate=0.3333
e2e_ms p50=950.0 p95=1103.0 p99=1116.6
stage means: e2e_ms=963.3 llm_ttft_ms=330.0 stt_ms=200.0 tts_ttfa_ms=236.7
interruptions=1 median_barge_in_stop_ms=210.0
gate: PASSED

Make quality a CI gate

voice-eval run calls.jsonl --max-wer 0.05 --min-task-completion 0.90 --max-e2e-p95-ms 1000 --json --output result.json

Pick thresholds that fit your use case. Replay exits with 0 when the gates pass, 1 when one fails, and 2 on a dataset error. See the repository's CI workflow for executable examples.

Reproduce in Kaggle

Both notebooks are published and verified. The offline kernel passed its full check suite with internet disabled on 2026-10-04, and the live kernel completed its no-secrets mock path. If a notebook is unavailable to you, its local source and the Kaggle reproduction guide describe the same steps. The offline bundle supplies pinned wheels and checksums.

The offline kernel runs with internet disabled. The live kernel needs an externally reachable WSS endpoint for a live call; localhost on your machine is not reachable from Kaggle. Its latency numbers include the Kaggle datacenter's network path, and its mock timings are simulated.

What it measures

The output field hallucination_rate measures calls containing a configured forbidden phrase. It does not detect every possible hallucination. Read the metric definitions.

How it works

View full-size diagram · Mermaid source

One evaluator, two entry points. A scored probe exports the same replay format used for offline calls. Replay reproduces the legacy scores from that recording; a new live call can vary with the agent, provider and network.

Live probe

From the cloned repository, try the four-turn appointment scenario with a conditional branch and deliberate interruption:

python -m pip install "voice-evals[probe]"
voice-eval probe src/voice_evals/resources/scenarios/appointment-v2.json --mock --output-dir out/probe-demo --max-wer 0.05 --max-barge-in-stop-ms 500
voice-eval run out/probe-demo/calls.jsonl

The mock demo takes about 25 seconds and needs no credentials or network. Its timing is simulated.

For a real call, configure ELEVENLABS_API_KEY, ELEVENLABS_VOICE_ID and PROBE_TRANSPORT_URL, plus PROBE_AGENT_API_KEY if your endpoint requires authentication:

voice-eval probe src/voice_evals/resources/scenarios/appointment-v2.json --caller elevenlabs --output-dir out/probe-live --json

Your endpoint must match a protocol map. The current transport supports mono s16le PCM at 16/24 kHz; arbitrary telephony or binary protocols need an adapter. Start with the local reference agent to exercise real sockets.

Caller and integration options

Open-model examples include Qwen3-TTS, Kokoro and Chatterbox caller profiles, plus a Voxtral STT gateway example. Model services run separately. Logos identify technologies and integrations, not endorsements.

Inspect the evidence

out/probe-demo/
  manifest.json       sanitized config, provenance and file hashes
  events.jsonl        normalized event journal
  calls.jsonl         replay dataset for eligible sessions
  result.json         scores, probe observations and gate result
  audio/caller/       sent audio, per utterance
  audio/agent/        received audio, per response

Ineligible sessions report NOT SCORED with reasons. Manifests record credential environment-variable names, not secret values. Missing stage observations stay unavailable; a timed-out interruption is never assigned an invented stop duration.

Recorded live validation: one ElevenLabs TTS → real WSS → reference-agent Scribe STT session completed with WER 0.1379 and barge-in stop 232.6 ms. It failed its WER gate of 0.10. The reference dialogue policy was scripted, and this single run is transport evidence rather than a model ranking. Method, latency aggregation and reproduction tests.

Documentation

Development

uv sync --extra probe
uv run pytest -q -m "not live"
uv run ruff check src tests scripts kaggle-kernel examples
uv build

The Makefile also provides make demo-probe, make test-integration and make kaggle-bundle. Offline tests never call real services. Live integration tests require VOICE_EVALS_LIVE_TESTS=1 and relevant credentials, and are blocked under CI.

Found an integration gap? Open an issue with the protocol, a minimal sanitized example, and the expected behavior.

Roadmap

• Twilio/Telnyx media-stream adapters.
• A dedicated OpenAI Realtime adapter.
• An optional LLM judge for graded outcomes.
• A GitHub Action for comparison against a committed baseline.

These are planned capabilities, not current integrations.

License

Apache 2.0. Built by Gjusev.