hotato 0.5.0

Find where your voice agent talks over callers, and keep it from coming back. Offline regression tests from your own call recordings, MIT.

hotato

Find where your voice agent talks over callers, and keep it from coming back.

Offline regression tests from your own call recordings. MIT.

Turn-taking regression monitoring for voice agents.

Connect your stack once. Hotato pulls recent recordings, scans for candidate talk-over and false-stop moments, and lets you turn confirmed bugs into CI tests.

Your audio stays on your machine unless you explicitly pull it from your stack.

hotato connect vapi
hotato sweep --stack vapi --since 7d --out hotato-sweep.html

That opens one offline dashboard ranking every candidate moment across your last 7 days of calls. Run sweep on a schedule (cron, CI, wherever) and it becomes a passive monitor; promote a confirmed bug into a permanent fixture with hotato fixture promote, then gate CI so it never comes back. Full guide: docs/SET-AND-FORGET.md · runnable examples/set-and-forget/.

No stack connected yet? hotato sweep --demo runs the identical flow over two bundled real calls: no credentials, no network.

Have one call recording and nothing else? One command surfaces its candidate moments:

uvx hotato scan --stereo your-call.wav   # two-channel WAV: caller ch0, agent ch1

From there: run scores a call to a PASS/FAIL verdict and an HTML report, compare measures a before/after delta for one fixed moment (fixed/regressed/improved/worse), verify proves it at battery scale across every fixture (reporting coincidence, not causation), and fixture create saves any moment as a permanent regression test that fails CI if it comes back. Everything runs on your machine; the audio never leaves it.

Hotato catches the three talk-over failures callers feel: the agent talking over the caller, false-stopping on a backchannel ("mhm"), or yielding too slowly. You label the expected behavior (yield = stop for the caller, hold = keep talking through a backchannel); Hotato measures whether the timing matched. It reports what it measured, never a guess at intent.

Try it with no audio of your own:

uvx hotato demo   # scores two real recorded calls a provider's default agent failed, so you see the FAILs, timelines, and fix cards

MCP

uvx --from "hotato[mcp]" hotato-mcp   # one tool, voice_eval_run; client configs in docs/MCP.md

Install

uvx hotato runs any command with zero install. To add it to a project:

pip install hotato                 # core: stdlib-only, zero dependencies
pip install 'hotato[neural]'       # optional Silero VAD cross-check
pip install 'hotato[livekit]'      # LiveKit live capture
pip install 'hotato[pipecat]'      # Pipecat live capture

Depth

• Set-and-forget monitoring (connect once, sweep on a schedule, promote confirmed bugs into fixtures): docs/SET-AND-FORGET.md · runnable examples/set-and-forget/
• Bad call to CI regression test, step by step: docs/BAD-CALL-TO-CI.md · runnable examples/bad-call-to-ci/
• What it measures (the three timing signals, re-derivable by hand): METHODOLOGY.md · Python API docs/API.md
• The fix ladder (each failure names a likely fix class; when the evidence maps cleanly to stack config, Hotato names the setting family and direction): docs/FIX-PLANS.md
• Rule out the non-turn-taking bugs first (STT, buffering, verbosity, refusals, wrong-language): docs/WHY.md
• Pull a call from your stack (Vapi, Twilio, Retell, LiveKit, Pipecat): adapters/README.md · status docs/ADAPTER-STATUS.md
• CI gates: GitHub Action docs/CI.md · pytest plugin docs/PYTEST.md
• Reports and analytics: docs/REPORTS.md · Suites docs/SUITES.md · Stack benchmarks docs/BENCHMARK-STACKS.md
• Real battery: 12 scripted calls against a live voice agent on its provider's default settings, where a missed interruption and a false stop on a backchannel fail in the same run, so diagnose refuses to name one threshold: corpus/vapi-defaults/README.md
• For agents: llms.txt · llms-full.txt · MCP server docs/MCP.md · Security SECURITY.md
• Contributing: the highest-value PR is a real, labelled call fixture: docs/SUBMITTING.md

Why "hotato": good turn-taking is a game of hot potato. Speak, then pass the turn the moment the caller wants it. MIT licensed (LICENSE); the open core stays open.

mcp-name: io.github.attenlabs/hotato