hotato 1.5.0

Conversation QA for voice agents. Catch the calls that pass every text check but talk over the caller, skip a required disclosure, or claim a task that never happened. Self-hosted, offline, MIT.

        ) )   ( (
       ( (     ) )
██╗  ██╗ ██████╗ ████████╗ █████╗ ████████╗ ██████╗ 
██║  ██║██╔═══██╗╚══██╔══╝██╔══██╗╚══██╔══╝██╔═══██╗
███████║██║   ██║   ██║   ███████║   ██║   ██║   ██║
██╔══██║██║   ██║   ██║   ██╔══██║   ██║   ██║   ██║
██║  ██║╚██████╔╝   ██║   ██║  ██║   ██║   ╚██████╔╝
╚═╝  ╚═╝ ╚═════╝    ╚═╝   ╚═╝  ╚═╝   ╚═╝    ╚═════╝ 

hotato

Open-source, self-hosted conversation QA for voice agents.

Score every call across five dimensions reported on their own (outcome, policy, conversation, speech, reliability), with the evidence behind every result. Deterministic checks and the model-judged rubric each score on their own lane.
Runs offline · MIT · zero dependencies.

Your voice agent passes every text assertion and still loses the call. It talks over the caller, skips a required disclosure, or confirms a refund that never posted. hotato scores the conversation and the outcome, with the timing evidence behind every flag, then turns a caught bug into a CI contract so it stays caught.

Quickstart: one command, on your machine

pip install hotato && hotato start --demo

Already have uv? Zero-install, same command:

uvx hotato start --demo

One command sweeps two bundled calls a provider's default agent failed, writes the candidate dashboard, and turns one missed-interruption moment into a demo failure contract it verifies on the spot, all offline:

[start] demo: swept 2 bundled calls, 5 candidate moments; wrote hotato-sweep.json, hotato-sweep.html, hotato-no-single-threshold.svg, contracts/demo-missed-interruption.hotato/contract.json
hotato start: swept the 2 bundled demo calls offline.
  sweep dashboard: hotato-sweep.html
  demo contract:   contracts/demo-missed-interruption.hotato
  verified contract: FAIL as expected -- the demo call missed the interruption
  [ ... then the exact next commands: promote a candidate, gate it in CI, re-check it ... ]

Open hotato-sweep.html. The candidate moments come ranked by how far the timing missed, each with a hear-the-bug playhead. This is the artifact you screenshot into a PR:

_{hotato-sweep.html · candidate moments ranked by salience, each with a playhead that sweeps the timeline in sync with the embedded audio. Candidates you review and label into your verdict.}

Point it at your own recording to walk the same loop end to end:

hotato start --stereo my-call.wav          # trust -> scan -> review -> label -> contract

Every command takes a two-channel recording (caller on one channel, agent on the other). A mono file or a bad export is marked NOT SCORABLE, so every verdict rests on inputs that carry the timing evidence.

What hotato catches: five dimensions, reported on their own

A voice agent can pass every text assertion and still lose the call: the refund never fires, the recording disclosure gets skipped, the caller gets talked over. hotato test run grades one call against a conversation-test file across five dimensions, each kept apart in its own count:

• Outcome · did the job get done, graded on tool-call and state evidence, not the transcript's say-so.
• Policy · required disclosures, PII handling, and the compliance phrases your team owns.
• Conversation · the deterministic turn-taking core: did the agent yield when the caller took the floor, and how fast.
• Speech · response latency and the timing around each turn.
• Reliability · pass@1 / pass@k / pass^k across repeated runs with a Wilson interval, so a flaky check reads as flaky.

Two lanes stay structurally separate. Deterministic checks (phrase, PII, policy, tool-call, sequence, latency, outcome, all pure regex, checksum, and span-lookup) live behind a wall from the model-judged rubric lane. A rubric verdict is deterministic: false and advisory, kept in its own count; the deterministic lane sets the gate. Each dimension keeps its own line in the output, including --format json, and the scored-output schemas reject an overall_score key outright.

hotato scenario init refund-check --out conversation-test.yaml   # a starter you edit for your own call
hotato test run conversation-test.yaml --agent support-bot

success: FAIL  (required: all_deterministic_assertions_pass, no_rubric_failure)
per-dimension (grouped view; never blended):
  outcome       0 pass / 0 fail / 1 inconclusive
  policy        0 pass / 0 fail / 1 inconclusive
  conversation  0 pass / 0 fail / 1 inconclusive
  speech        0 pass / 0 fail / 1 inconclusive
  reliability   0 pass / 0 fail / 0 inconclusive

Supply the call as --transcript, --trace, --state, and/or --audio and each check turns to pass or fail; a check whose evidence is absent stays INCONCLUSIVE until the evidence arrives. Full walkthrough: docs/CONVERSATION-TEST.md.

The receipt: a per-dimension scorecard with the evidence attached

The scored HTML report is the thing that ends an argument in review: a verdict per event, a time-to-yield and talk-over histogram, failure clusters by fix class, and the per-event timeline with both channels and the measured numbers. The same audio and config reproduce every figure on the page.

_{One recording, the pinned scorer, a FAIL against the labeled yield expectation. Share it in a PR with hotato card hotato-sweep.json#1 --out finding.svg.}

The loop: catch, confirm, gate, prove

Surface a moment, confirm what it should have done, pin it to a CI contract, then re-check today's agent.

1. Catch: surface the candidate moments

sweep ranks the talk-over and false-stop moments across your recent calls by how far the timing missed. Two candidate types, straight from the open scorer, each a reproducible timing measurement:

Level 1 -- candidate. Hotato reports what it measured (0.32s of overlap; 0.46s of trailing silence). Each is a candidate timing moment worth review.

2. Confirm: you label yield or hold

You decide the expected behavior for a candidate: yield (stop for the caller) or hold (keep talking through a backchannel). That judgment stays with you: a threshold that stops missing interruptions starts false-stopping on backchannels, so when both fail in one run diagnose surfaces the tradeoff instead of naming one threshold:

_{Level 2 -- human-labeled failure. A reviewer confirms the recording broke an explicit yield-or-hold expectation. When a missed interruption and a false stop collide, the fix lives in the engagement-control class.}

3. Gate: pin it to a CI contract

fixture promote saves your labeled call as a permanent regression test. On every push, CI re-scores that recording under the pinned thresholds and scorer, and exits non-zero if the stored evidence stops matching your policy.

4. Prove: re-check the current agent

The frozen recording catches the evidence, thresholds, or scorer drifting. To check that the CURRENT agent still behaves, recapture the same scenario and score a NEW contract under the same policy:

# place the same call against today's agent, capture dual-channel, then:
hotato contract create --stereo fresh-call.wav --onset 41.90 --expect yield \
    --id refund-cutoff-001-recapture --out contracts
hotato contract verify contracts/refund-cutoff-001-recapture.hotato

_{Level 4 -- fresh-recapture comparison. A newly captured call meets the same labeled policy and every submitted paired guard held. This is the claim only a fresh recapture can make. Walkthrough: docs/RECAPTURE.md.}

Five levels of evidence, each scored on its own lane

Every card, report, and CLI result names its evidence level, and the public tier is the weakest one its inputs support.

Level	Name	What it means
1	Candidate	A candidate timing moment worth human review.
2	Human-labeled failure	A reviewer confirmed this recording broke an explicit yield-or-hold expectation.
3	Stored-evidence check	The historical audio still produces the expected result under the pinned policy and scorer.
4	Fresh-recapture comparison	A newly captured call passed the same contract, and no submitted paired guard regressed.
5	External proof	An independent team confirmed a caught regression or a fresh recapture. Not yet published.

A before/after experiment (hotato fix trial, and the fleet loop) re-derives every verdict from the on-disk audio under one pinned trial manifest. A proof stands on the original recordings, intact fixtures, and that manifest, so the number is re-scored from the audio on every run.

Gate it in CI

hotato speaks CI natively.

• Every scoring command exits non-zero when a deterministic check fails and zero when they pass, so a red build is a caught regression.
• hotato contract verify contracts/ --junit contracts-junit.xml writes JUnit XML your runner already renders.
• --format json carries an exit_code field, so an agent or a workflow reads the verdict without parsing prose.
• The model-judged rubric is advisory by default and blocks a build only when you pass --gate.

Drop-in GitHub Action and a pytest plugin: docs/CI.md · docs/PYTEST.md. One bad call to a CI gate, step by step: docs/BAD-CALL-TO-CI.md · runnable examples/bad-call-to-ci/.

Scale one call into a release gate

• hotato suite run suite.yaml --agent support-bot -- run a whole suite.v1; scenario-driven tests execute offline through the deterministic scripted-caller simulator, and every run records into the local registry.
• hotato simulate --matrix scenario.yaml --out ./conv -- render a scenario.v1 with a deterministic scripted caller into origin=simulated conversation artifacts; the variation matrix expands into hundreds of runs, seeded and byte-identical on replay.
• hotato rubric run --rubrics rubrics.yaml --transcript call.json -- the model-judged lane on a pinned local model; zero egress, advisory unless --gate. docs/RUBRIC.md.
• hotato release compare BASELINE CANDIDATE -- diff two recorded releases per dimension and per scenario, digest-exact, surfacing new failures and fixed-since.
• hotato serve -- a self-hosted, read-only web app over the local registry (release readiness, scenario matrix, conversation inspector, failure clusters, production health) on 127.0.0.1, bearer-token authenticated. docs/WORKSPACE.md.

The bundled reference agent shows the shape at scale: 25 voice-agent jobs x 5 caller behaviours x 3 audio environments = 375 offline simulated runs, scored across the dimensions and byte-reproducible on replay. Run it with make reference in examples/reference-agent/ (docs/SUITE-RUN.md); the measurement-error harness is docs/BENCHMARK.md.

Connect a production stack

The demo runs on the bundled calls alone. To point Hotato at your live calls, connect once, then sweep on a schedule:

hotato connect vapi                                             # credentials stored 0600, local only
hotato sweep --stack vapi --since 7d --out hotato-sweep.html    # cron, CI, wherever

Run sweep on a timer and it becomes a scheduled batch scanner. Your audio stays on your machine unless you explicitly pull it from your stack. Full guide: docs/SET-AND-FORGET.md · runnable examples/set-and-forget/.

sweep and hotato fleet run can POST a one-line JSON summary to a webhook when they finish, off by default, opt in with --notify (repeatable for more than one URL):

hotato sweep --stack vapi --since 7d --notify https://hooks.slack.com/services/...

The payload stays metadata-only: counts, the top candidate moments (id, kind, timing numbers only), local artifact paths, and a text field a Slack incoming webhook renders directly. A down or slow webhook leaves the run intact: a delivery failure is one warning line on stderr. Egress details: docs/EGRESS.md.

Fleet: private, self-hosted

hotato fleet runs the loop across every agent from one local workspace: ingest calls, surface candidates, label them, and run a before/after experiment that recomputes both sides from audio under a pinned manifest. It recommends a change and leaves the deploy to you. Local mode is stdlib-only (SQLite plus a content-addressed store), runs entirely on your machine, and registers as many agents as you want.

hotato fleet init    -w acme
hotato fleet agent add -w acme --name support-bot --stack vapi --assistant-id asst_123
hotato fleet ingest  -w acme --agent support-bot call.wav
hotato fleet discover -w acme --agent support-bot call.wav
hotato fleet review   -w acme

Once a workspace has some history, hotato fleet trend -w acme reads the same local SQLite registry and writes one self-contained HTML page: per-agent talk-over and time-to-yield trend lines (p50/p95 per day), candidate moments discovered over time, and experiment outcomes (improved/inconclusive/refused). A day with no measurements gets no point, and a series with fewer than two days of history reports plainly as "not enough history to trend" instead of an interpolated line. Full guide: docs/GUARDIAN-FLEET.md.

Self-host in your own cloud or VPC

The whole team workspace ships as a container. One command stands up the read-only, token-authenticated workspace (hotato serve) on host loopback over a private volume; the default stack keeps all traffic on your own infrastructure at run time.

docker compose up -d                    # workspace on 127.0.0.1:8321
docker compose run --rm hotato-init     # optional: seed example data
docker compose --profile judge up -d    # optional: a local Ollama model judge

Build, connect your own calls, the local judge, air-gap, backup, and the zero-migration promise (self-host and cloud share one set of schemas): docs/SELF-HOST.md. Confirm the offline posture on your own machine with deploy/verify-zero-egress.sh.

Built for coding agents

Hotato is a first-class tool for an LLM or an agent to drive: machine JSON on every command, meaningful exit codes, a described capability manifest, llms.txt, JSON-LD, and an MCP server.

uvx --from "hotato[mcp]" hotato-mcp      # the voice_eval_run scorer + eleven fleet tools

Configs and the tool contract: docs/MCP.md · AGENTS.md · llms-full.txt.

Choose your path

You want to	Run this
Try the full loop, no credentials	hotato start --demo
Sweep the bundled demo calls	hotato sweep --demo
Sweep recent calls from your stack	hotato connect vapi then hotato sweep --stack vapi --since 7d
Add Hotato to an existing repo, CI gate included	hotato init starter --stack vapi --out . (docs/STARTER.md)
Turn a confirmed failure into a portable contract	hotato contract create --from-candidate hotato-sweep.json#1 --expect yield --id refund-cutoff-001 --out contracts (docs/CONTRACTS.md)
Verify contracts in CI	hotato contract verify contracts/ --junit contracts-junit.xml
Attach observability traces to a contract	hotato trace attach contracts/refund-cutoff-001.hotato --trace voice_trace.jsonl (docs/TRACE.md)
Test a candidate fix, before/after, fail-closed	hotato fix trial patch.json --name staging-x --before before/ --after after/ (docs/FIX-TRIAL.md)
Share a finding in a PR or slide	hotato card hotato-sweep.json#1 --out finding.svg
Drive it from a coding agent	uvx --from "hotato[mcp]" hotato-mcp (docs/MCP.md)

contract verify and a promoted fixture in CI are two different guarantees, depending on which recording goes in:

	On the frozen recording (every push)	On a fresh recapture (by hand, see docs/RECAPTURE.md)
Proves	The evidence, policy, and scorer are intact	The CURRENT agent's behavior still matches the label
Does not prove	That the deployed agent has not changed	--

A contract bundle contains call audio. Do not commit a raw customer contract to a public repository; use sanitized fixtures for anything public. See docs/CONTRACTS.md.

Install

Add hotato to a project with pip; or run any command zero-install with uvx:

pip install hotato                 # core: stdlib-only, zero dependencies
pip install 'hotato[neural]'       # optional Silero VAD cross-check
pip install 'hotato[transcribe]'   # optional ASR transcript, attached as context beside the score
pip install 'hotato[livekit]'      # LiveKit live capture
pip install 'hotato[pipecat]'      # Pipecat live capture

Contribute a labeled call

The highest-value PR is one labeled dual-channel call. A small, high-integrity corpus of recorded calls with human-labeled turn-taking ground truth is the part of hotato that compounds: the more moments the community labels, the sharper every scorer gets. Add a clip: docs/SUBMITTING.md · the corpus and its schema live in corpus/, with a recorded battery in corpus/vapi-defaults/README.md. Contributor guide: CONTRIBUTING.md.

Where Hotato fits

• Conversation QA that shows its work. See docs/COMPARE.md for how the per-dimension scorecard fits alongside runtime voice layers.
• Audio-timing scoring. It scores how the conversation is timed. The opt-in --transcribe flag attaches an ASR transcript as context to read next to the verdict, with the timing score grounded in the audio. See docs/TRANSCRIBE.md.
• Channel-based timing. Each channel is one party (caller on one, agent on the other); Hotato measures the timing between them, and the channels stay anonymous.
• Timing evidence you label. It produces candidate timing evidence; humans label intent, and CI enforces the confirmed contracts.
• Offline and out-of-band. It reads recordings after the call, so your live audio path and running agent stay in your hands.

Docs

• Set-and-forget monitoring (connect once, sweep on a schedule, promote confirmed bugs into fixtures): docs/SET-AND-FORGET.md · runnable examples/set-and-forget/
• Bad call to CI regression test, step by step: docs/BAD-CALL-TO-CI.md · runnable examples/bad-call-to-ci/
• What it measures (the three timing signals, re-derivable by hand): METHODOLOGY.md · Python API docs/API.md
• The fix ladder (each failure names a likely fix class; when the evidence maps cleanly to stack config, Hotato names the setting family and direction): docs/FIX-PLANS.md
• Rule out the non-turn-taking bugs first (STT, buffering, verbosity, refusals, wrong-language): docs/WHY.md
• Pull a call from your stack (Vapi, Twilio, Retell, LiveKit, Pipecat): adapters/README.md · status docs/ADAPTER-STATUS.md
• CI gates: GitHub Action docs/CI.md · pytest plugin docs/PYTEST.md
• Recorded-call battery: 12 scripted calls against a live voice agent on its provider's default settings, where a missed interruption and a false stop on a backchannel fail in the same run, so diagnose surfaces the tradeoff instead of naming one threshold: corpus/vapi-defaults/README.md
• Failure contracts and traces: turn a labeled candidate into a portable, CI-verified bundle and attach observability evidence: docs/CONTRACTS.md · docs/TRACE.md · docs/OTEL.md
• Deterministic assertions on the transcript/trace (phrase, PII, policy, tool-call, outcome; each scored on its own lane): docs/ASSERTIONS.md
• Checking the CURRENT agent, not just the frozen recording: the recapture walkthrough: docs/RECAPTURE.md
• Egress: a per-command network table derived from the code, what is local, what reaches your vendor, what optional extras add a hosted call: docs/EGRESS.md
• Layered failure evidence and a before/after fix trial: hotato explain breaks a failing result down by layer, and hotato fix trial tests a candidate change before/after, fail-closed: docs/EXPLAIN.md · docs/FIX-TRIAL.md · docs/APPLY.md · docs/FIX-LOOP.md
• Evidence: what Hotato validates, the input-condition trust matrix, every card and CLI block reproducible, and where Hotato fits alongside broader voice-agent testing tools: docs/VALIDATION.md · docs/TRUST-MATRIX.md · docs/GALLERY.md · docs/EVIDENCE-PACK.md · docs/COMPARE.md
• For coding agents: AGENTS.md · llms.txt · llms-full.txt · MCP server docs/MCP.md · Security SECURITY.md
• Contributing: the highest-value PR is a labeled call fixture: docs/SUBMITTING.md

Why "hotato": good turn-taking is a game of hot potato. Speak, then pass the turn the moment the caller wants it. MIT licensed (LICENSE); the open core stays open.

mcp-name: io.github.attenlabs/hotato