voxprobe

Voice agents fail in ways text evals never see — voxprobe sends an adaptive simulated caller, records both sides, and measures what actually happened on the line.

_{One run of voxprobe simulate --scenario 02 --target local-clinic-buggy --mode audio: left channel = the agent under test, right = the simulated caller. Both findings above were produced by the framework — the dead air by measurement, the Saturday booking by the judge with the clinic's ground truth.}

What it is

voxprobe is a persona-driven QA framework for voice agents. A scenario describes a person — what they know, what they must never invent, what they want, and how they talk. voxprobe plays that person through a real voice pipeline against the agent under test — the bundled sample agents in-process, or any Pipecat websocket agent — records the call as stereo audio (agent left, caller right), rebuilds a transcript from the audio, measures turn-taking and response latency for both parties, and produces PASS/FAIL with findings that each cite a timestamp and a quote.

• Adaptive caller, not a script. Persona + ordered plan + a per-turn "director" ("answer their question first", "wrap up now") — it accepts offers, pushes back, changes its mind, says goodbye.
• Two full voice pipelines in one process. Built on Pipecat 1.7 with an in-process loopback transport pair voxprobe adds (paced output, virtual mic with silence, interruption flush) — see ADR-002. A deliberate barge-in driver interrupts the agent on purpose and records whether/when it yields (Pipecat's interrupted flag).
• Websocket targets. The same caller connects to any agent speaking Pipecat's websocket protocol (voxprobe serve-agent exposes the sample agent for a smoke test). Live-validated end to end; known issue: audio reaches the remote agent unpaced, and in the smoke run its STT missed a few words — tracked in the roadmap.
• Timing from audio, words from ASR. Speech regions per channel come from ffmpeg silencedetect; Whisper transcribes each region. Dead air and talk-over are measured, not judged — see ADR-001.
• A deliberately broken agent ships in the box — and the detector is benchmarked against it. targets/local-clinic-buggy.yaml toggles bugs (weekend_booking, fabricated_dob, phi_leak, …); voxprobe bench measures precision/recall/F1 per bug class against a clean control, and voxprobe calibrate measures human agreement with the judge.
• PASS/FAIL without an LLM in the loop. The judge returns structured per-criterion / per-hypothesis verdicts with evidence; dead air and talk-over are measured; decide() combines them deterministically.
• $0 by default. Groq + Gemini free tiers for the LLMs, Deepgram's signup credit for speech. No telephony required.

New in 0.2.0 — the other end of the line (CALL-E)

You cannot program an outbound AI caller like CALL-E — its API takes a task paragraph and returns a self-report — so voxprobe programs the person it calls. voxprobe line up puts our receptionist (with switchable planted behaviors) on a real phone number; voxprobe otherend run --profile <p> lets CALL-E dial it, then grades what CALL-E reported against what our receptionist actually said — deterministically, from our stereo recording.

Every row's adversity is proven by a manifest regex over our own lines; the two headline findings plus an ASR error carried into structured_result ("with Doctor Chen" → "without") are in FEEDBACK.md. Measured from the audio across 7 real calls: CALL-E caller response gap p50 median 2.63 s (2.49–3.01 s), one talk-over event, 12.9 min. Full method and caveats (the hold is nominal; the judge below scores our receptionist, not CALL-E) in docs/DEVLOG.md.

Results (v0.1.0 core)

Does the detector work? — planted-bug benchmark (full table + method)

114 text-mode runs: 8 bug classes planted one at a time in the bundled sample agent, the same scenarios against the clean agent as control, k = 3 repeats per cell, strict judge detector (the injected hypothesis marked observed, resolved by nearest text — no keyword guessing).

Per class: weekend_booking, ignore_constraints, medical_advice, promise_refill, transfer_dead_end, fabricated_dob all P = R = 1.0; no_verification R 0.667 and phi_leak R 0.833 — every one of those misses is a run where the planted bug never manifested (the capable LLM agent overrode the planted instruction; the sheet marks manifested per run) — recall given manifestation is 1.0 for every class we can check. Transparent symptom-rule detectors (regexes over the agent's lines) are scored alongside. Reproduce: uv run voxprobe bench --name mine -k 3.

Does a human agree with the judge? — calibration sheet

25 judged claims (12 judge-positive / 13 negative, stratified), hand-labelled by the author: 25/25 agree, Cohen's κ = 1.0, judge-positive precision 1.0. Small n, single rater — a sanity check, not a certificate; the sheet and the scorer are in the repo.

Real audio runs (in examples/: MP3 · live transcript · audio-derived transcript · analysis)

Quick start

git clone https://github.com/Divya1S/voxprobe.git && cd voxprobe && cp .env.example .env   # add GROQ_API_KEY, GOOGLE_API_KEY (free); DEEPGRAM_API_KEY for audio
uv sync --extra dev --extra arena && uv run pytest -q
uv run voxprobe simulate --scenario 02 --target local-clinic-buggy --mode audio               # or --mode text for an LLM↔LLM dry run
uv run voxprobe bench --name mine -k 3 --bugs weekend_booking                                   # detector benchmark (text mode)

Requires Python ≥ 3.11, uv, and ffmpeg on PATH. Output lands in recordings/, transcripts/, reports/.

How it works

flowchart LR
  S[(scenarios/*.yaml<br/>persona · plan · bug hypotheses)] --> B[caller brain<br/>persona + director<br/>Groq → Gemini failover]
  T[(targets/*.yaml<br/>business ground truth<br/>planted bugs)] --> A[agent under test<br/>bundled sample pipeline]
  B <-->|virtual phone line<br/>16 kHz, paced, silence-ticked| A
  B --> R[stereo recording<br/>L agent · R caller]
  R --> X[silencedetect regions<br/>→ Whisper per region]
  X --> M[metrics: response gaps p50/p95<br/>dead air · overlaps · talk share]
  M --> J[measured issues + LLM-judge draft<br/>with ground truth, timestamps, quotes]

Full write-up: docs/ARCHITECTURE.md. Engineering log with the numbers behind each change: docs/DEVLOG.md.

How it differs from existing tools

Concessions: pipecat.evals has negative assertions, function-call and DTMF assertions and a suite runner voxprobe lacks; EVA has 213 scenarios and a paper; scenario has more adapters. voxprobe's wedge is the adaptive caller plus audio-derived, both-party measurement — and a self-benchmark.

Metrics glossary (what is actually computed)

Thresholds are one SegmentationPolicy object, tunable and documented.

Repository layout

Status and roadmap

v0.2.0 (pip install voxprobe). Done: core, text arena, audio arena, benchmark, human calibration, golden tests, barge-in driver, websocket adapter, CALL-E adapter + inbound line + otherend grader with a 6-profile matrix on real calls. Next: a timed hold via mid-call control, n≥2 on every profile, a Gemini Live target, then a write-up. Details and honest limitations: docs/ROADMAP.md, CHANGELOG.md. The Vapi phone adapter is experimental (bring your own paid telephony; every outbound number must be on ALLOWED_NUMBERS_E164).

License

MIT