hotato 0.2.0

Hotato: the open turn-taking eval for voice agents. Does your agent drop the turn, or hog it? Score barge-in / talk-over / backchannel / endpointing from a call recording, offline, with an honest fix map. MIT.

hotato

Find interruption bugs in your voice agent before users do.

Hotato scores turn-taking from a call recording, on your machine, so call audio stays with you. It catches the three failures callers feel most: the agent talks over a real interruption, stops for a backchannel ("mhm"), or is slow to yield. Every failing event returns three measured signals (did_yield, seconds_to_yield, talk_over_sec) and a fix class that names the knob to turn.

See it fail a bad agent

uvx hotato demo

The demo battery is intentionally bad and fully synthetic: it exists to show what a catch looks like.

Score your own call

uvx hotato doctor --stereo your_call.wav   # two-channel WAV: caller ch0, agent ch1

doctor scores the call, writes the visual HTML report, and opens it. Or pull a recording straight from your stack:

• Vapi: uvx hotato capture --stack vapi --call-id <id> with VAPI_API_KEY set.
• Twilio: uvx hotato capture --stack twilio --recording-sid RE... with TWILIO_ACCOUNT_SID / TWILIO_AUTH_TOKEN set.
• Retell: uvx hotato capture --stack retell --call-id <id> with RETELL_API_KEY set; fetches the call's multichannel recording.
• LiveKit / Pipecat: uvx hotato setup --stack livekit (or pipecat) scaffolds recording in your infra, then hotato capture scores the files.

Scope, stated once: Hotato scores separated caller/agent tracks, as one two-channel WAV or two aligned mono WAVs. Per-stack details and verification dates: adapters/README.md, docs/ADAPTER-STATUS.md.

What Hotato measures

Three objective timing signals per event, measured frame by frame and reproducible with --dump-frames:

Signal	What it answers
did_yield	did the agent stop talking when the caller took the floor?
seconds_to_yield	how long did that take?
talk_over_sec	how many seconds it kept talking over the caller first

Every failing event carries exactly one fix, from a taxonomy of two classes. Pass --stack livekit, pipecat, vapi, or generic to get the knob in your stack's vocabulary:

Failure	fix_class	The knob it names
Missed a real interruption	config	interruption sensitivity: LiveKit turn_handling interruption.min_duration / interruption.min_words, Pipecat VADParams(start_secs, stop_secs, confidence), Vapi stopSpeakingPlan.numWords
Slow yield	config	endpointing latency: LiveKit endpointing.min_delay / endpointing.max_delay, Pipecat SpeechTimeoutUserTurnStopStrategy(user_speech_timeout=...), Vapi stopSpeakingPlan.backoffSeconds
Excess talk-over	config	overlap debounce: LiveKit interruption.min_duration, Pipecat VADParams(stop_secs), Vapi stopSpeakingPlan.voiceSeconds
Yielded to its own echo	config	audio routing: echo cancellation and separate caller/agent channels
Yielded to a backchannel	engagement-control	a vendor-neutral pointer. No single timing threshold separates a backchannel from a one-word interruption. Where your stack provides an interruption/backchannel classifier, use it; the general case calls for a learned engagement-control / addressee-detection layer

Each config knob ships with a direction and its honest trade-off. When a battery fails on both axes at once (missed a real interruption AND false-triggered on a backchannel), a battery-level funnel pointer fires: that pattern is the signal a discriminating layer is needed, since one threshold trades the two cases against each other.

CI

uvx hotato run --suite barge-in --format json   # text is the human default; json is the machine envelope

Exit codes: 0 all pass (or --no-fail), 1 a regression, 2 usage or IO error. Two ready-made gates: copy .github/workflows/hotato.yml for a PR check with a sticky results comment (docs/CI.md), or add --hotato-suite to your existing pytest run; the plugin auto-registers on install and fails the session on a regression (docs/PYTEST.md).

What you get

• doctor: score a recording (or the bundled self-test), write the visual report, open it.
• report: self-contained HTML with per-event SVG timelines, a per-frame inspector, print CSS for PDF, and --base base.json regression deltas.
• team: aggregate a directory of runs into pass rate over time plus mean/median/p90 talk-over and time-to-yield.
• export: research-grade CSVs (events.csv, frames.csv, envelope.json), columns documented in-file.
• benchmark: score your captured battery per stack, then compare result files side by side.
• Pytest plugin: a hotato_score fixture plus a session gate (pytest --hotato-suite).
• MCP server: one tool, voice_eval_run; pass report_path to also get the HTML report. uvx --from "hotato[mcp]" hotato-mcp
• Tiered corpus suites: 112 deterministic scenarios across silver and gold tiers, plus defect suites that must fail.

Optional neural cross-check (verified, non-reference)

pip install 'hotato[neural]', then hotato run --stereo call.wav --backend neural re-runs the same turn-taking timing math over a Silero VAD speech track. The ONNX weights ship inside the package and inference runs offline on CPU. Verified properties:

• Same contract. The neural track comes back in the identical result shape as the energy reference, on the same hop grid, end to end.
• Deterministic. Repeated runs on the same audio are byte-identical.
• The energy reference is untouched. Installing the extra changes no energy number, and --suite always scores with energy (a note says so if you pass --backend neural there).
• Built for real recordings. The bundled fixtures are synthetic shaped noise rendered for the energy reference, and a speech-trained model marks no speech in them, so the cross-check is informative on your own calls, where both tracks carry real speech.
• Clear errors. Without the extra, --backend neural exits with an explicit error, never a silent energy fallback. Silero accepts 8000 Hz, 16000 Hz, and integer multiples of 16000 Hz; other rates get an actionable resample message.

Full method and the measured cross-check properties: METHODOLOGY.md.

Install

uvx hotato runs every command with zero install. To add it to a project:

pip install hotato                 # core: stdlib-only, zero dependencies
pip install 'hotato[neural]'       # optional Silero VAD cross-check
pip install 'hotato[livekit]'      # LiveKit live capture
pip install 'hotato[pipecat]'      # Pipecat live capture

More

• Reports and analytics: docs/REPORTS.md · Pytest gate: docs/PYTEST.md · Suites: docs/SUITES.md · CI recipes: docs/CI.md
• Python API: docs/API.md · Stack benchmarks: docs/BENCHMARK-STACKS.md
• Adapters: adapters/README.md · Status and verification dates: docs/ADAPTER-STATUS.md
• Why this exists: docs/WHY.md · Method: METHODOLOGY.md · Security: SECURITY.md
• Contributing: the highest-value PR is a real, labelled call fixture. Start at docs/SUBMITTING.md.

Why "hotato": good turn-taking is a game of hot potato. Take your turn, then pass it, fast and clean. MIT licensed (LICENSE); the open core stays open.