hotato 1.3.2

Conversation QA for voice agents. Catch the calls that pass every text check but talk over the caller, skip a required disclosure, or claim a task that never happened. Self-hosted, offline, MIT.

        ) )   ( (
       ( (     ) )
██╗  ██╗ ██████╗ ████████╗ █████╗ ████████╗ ██████╗ 
██║  ██║██╔═══██╗╚══██╔══╝██╔══██╗╚══██╔══╝██╔═══██╗
███████║██║   ██║   ██║   ███████║   ██║   ██║   ██║
██╔══██║██║   ██║   ██║   ██╔══██║   ██║   ██║   ██║
██║  ██║╚██████╔╝   ██║   ██║  ██║   ██║   ╚██████╔╝
╚═╝  ╚═╝ ╚═════╝    ╚═╝   ╚═╝  ╚═╝   ╚═╝    ╚═════╝ 

hotato

Open-source, self-hosted conversation QA for voice agents.

Score every call across five dimensions reported on their own (outcome, policy, conversation, speech, reliability), with the evidence behind every result. Deterministic checks stay separate from the model-judged rubric; never one blended number.
Offline by default · MIT · zero runtime dependencies.

Your voice agent passes every text assertion and still loses the call. It talks over the caller, skips a required disclosure, or confirms a refund that never posted. hotato scores the conversation and the outcome, with the timing evidence behind every flag, then turns a caught bug into a CI contract so it stays caught.

Quickstart: no account, no keys, no network

pip install hotato && hotato start --demo

Already have uv? Zero-install, same command:

uvx hotato start --demo

One command sweeps two bundled calls a provider's default agent failed, writes the candidate dashboard, and turns one missed-interruption moment into a demo failure contract it verifies on the spot, all offline:

[start] demo: swept 2 bundled calls, 5 candidate moments; wrote hotato-sweep.json, hotato-sweep.html, hotato-no-single-threshold.svg, contracts/demo-missed-interruption.hotato/contract.json
hotato start: swept the 2 bundled demo calls offline.
  sweep dashboard: hotato-sweep.html
  demo contract:   contracts/demo-missed-interruption.hotato
  verified contract: FAIL as expected -- the demo call missed the interruption
  [ ... then the exact next commands: promote a candidate, gate it in CI, re-check it ... ]

Open hotato-sweep.html. The candidate moments come ranked by how far the timing missed, each with a hear-the-bug playhead. This is the artifact you screenshot into a PR:

_{hotato-sweep.html · candidate moments ranked by salience, each with a playhead that sweeps the timeline in sync with the embedded audio. Candidates you review and label, never a decided verdict.}

Point it at your own recording to walk the same loop end to end:

hotato start --stereo my-call.wav          # trust -> scan -> review -> label -> contract

Every command takes a two-channel recording (caller on one channel, agent on the other). A mono file or a bad export is marked NOT SCORABLE, never turned into a confident but meaningless verdict.

What hotato catches: five dimensions, reported on their own

A voice agent can pass every text assertion and still lose the call: the refund never fires, the recording disclosure gets skipped, the caller gets talked over. hotato test run grades one call against a conversation-test file across five dimensions, kept apart and never summed into one number:

• Outcome · did the job get done, graded on tool-call and state evidence, not the transcript's say-so.
• Policy · required disclosures, PII handling, and the compliance phrases your team owns.
• Conversation · the deterministic turn-taking core: did the agent yield when the caller took the floor, and how fast.
• Speech · response latency and the timing around each turn.
• Reliability · pass@1 / pass@k / pass^k across repeated runs with a Wilson interval, so a flaky check reads as flaky.

Two lanes stay structurally separate. Deterministic checks (phrase, PII, policy, tool-call, sequence, latency, outcome, all pure regex, checksum, and span-lookup) live behind a wall from the model-judged rubric lane. A rubric verdict is deterministic: false, advisory, and never merged into a deterministic count; a judge can never silently gate CI. No output carries an overall_score, including --format json.

hotato scenario init refund-check --out conversation-test.yaml   # a starter you edit, not a claim about your call
hotato test run conversation-test.yaml --agent support-bot

success: FAIL  (required: all_deterministic_assertions_pass, no_rubric_failure)
per-dimension (grouped view; never blended):
  outcome       0 pass / 0 fail / 1 inconclusive
  policy        0 pass / 0 fail / 1 inconclusive
  conversation  0 pass / 0 fail / 1 inconclusive
  speech        0 pass / 0 fail / 1 inconclusive
  reliability   0 pass / 0 fail / 0 inconclusive

Supply the call as --transcript, --trace, --state, and/or --audio and each check turns to pass or fail; a check whose evidence is absent stays INCONCLUSIVE, never guessed. Full walkthrough: docs/CONVERSATION-TEST.md.

The receipt: a per-dimension scorecard with the evidence attached

The scored HTML report is the thing that ends an argument in review: a verdict per event, a time-to-yield and talk-over histogram, failure clusters by fix class, and the per-event timeline with both channels and the measured numbers. The same audio and config reproduce every figure on the page.

_{One recording, the pinned scorer, a FAIL against the labeled yield expectation. Share it in a PR with hotato card hotato-sweep.json#1 --out finding.svg.}

The loop: catch, confirm, gate, prove

Surface a moment, confirm what it should have done, pin it to a CI contract, then re-check today's agent.

1. Catch: surface the candidate moments

sweep ranks the talk-over and false-stop moments across your recent calls by how far the timing missed. Two candidate types, straight from the open scorer, no accuracy score:

Level 1 -- candidate. Hotato reports what it measured (0.32s of overlap; 0.46s of trailing silence), never a guess at intent. Each is a timing moment worth review, not a bug yet.

2. Confirm: you label yield or hold

You decide the expected behavior for a candidate: yield (stop for the caller) or hold (keep talking through a backchannel). No single sensitivity dial decides it for you. A threshold that stops missing interruptions starts false-stopping on backchannels, so when both fail in one run diagnose refuses to name one threshold:

_{Level 2 -- human-labeled failure. A reviewer confirms the recording broke an explicit yield-or-hold expectation. When a missed interruption and a false stop collide, the fix is the engagement-control class, not one dial.}

3. Gate: pin it to a CI contract

fixture promote saves your labeled call as a permanent regression test. On every push, CI re-scores that recording under the pinned thresholds and scorer, and exits non-zero if the stored evidence stops matching your policy.

4. Prove: re-check the current agent

The frozen recording catches the evidence, thresholds, or scorer drifting. To check that the CURRENT agent still behaves, recapture the same scenario and score a NEW contract under the same policy:

# place the same call against today's agent, capture dual-channel, then:
hotato contract create --stereo fresh-call.wav --onset 41.90 --expect yield \
    --id refund-cutoff-001-recapture --out contracts
hotato contract verify contracts/refund-cutoff-001-recapture.hotato

_{Level 4 -- fresh-recapture comparison. A newly captured call meets the same labeled policy and no submitted paired guard regressed. This is the claim the frozen-recording gate cannot make. Walkthrough: docs/RECAPTURE.md.}

Five levels of evidence, and never one blended score

Hotato never calls the weakest level a verdict. Every card, report, and CLI result names its level; the public tier is the weakest one its inputs support.

Level	Name	What it means
1	Candidate	A timing moment worth human review. Not a bug yet.
2	Human-labeled failure	A reviewer confirmed this recording broke an explicit yield-or-hold expectation.
3	Stored-evidence check	The historical audio still produces the expected result under the pinned policy and scorer.
4	Fresh-recapture comparison	A newly captured call passed the same contract, and no submitted paired guard regressed.
5	External proof	An independent team confirmed a caught regression or a fresh recapture. Not yet published.

A before/after experiment (hotato fix trial, and the fleet loop) re-derives every verdict from the on-disk audio under one pinned trial manifest. It refuses a proof built from an edited verdict, a re-encoded old call, a dropped fixture, or unrelated audio. The number comes from re-scoring the recordings every time, never from a stored field.

Gate it in CI

hotato speaks CI natively.

• Every scoring command exits non-zero when a deterministic check fails and zero when they pass, so a red build is a caught regression.
• hotato contract verify contracts/ --junit contracts-junit.xml writes JUnit XML your runner already renders.
• --format json carries an exit_code field, so an agent or a workflow reads the verdict without parsing prose.
• The model-judged rubric is advisory by default and blocks a build only when you pass --gate.

Drop-in GitHub Action and a pytest plugin: docs/CI.md · docs/PYTEST.md. One bad call to a CI gate, step by step: docs/BAD-CALL-TO-CI.md · runnable examples/bad-call-to-ci/.

Scale one call into a release gate

• hotato suite run suite.yaml --agent support-bot -- run a whole suite.v1; scenario-driven tests execute offline through the deterministic scripted-caller simulator, and every run records into the local registry.
• hotato simulate --matrix scenario.yaml --out ./conv -- render a scenario.v1 with a deterministic scripted caller into origin=simulated conversation artifacts; the variation matrix expands into hundreds of runs, seeded and byte-identical on replay.
• hotato rubric run --rubrics rubrics.yaml --transcript call.json -- the model-judged lane on a pinned local model; zero egress, advisory unless --gate. docs/RUBRIC.md.
• hotato release compare BASELINE CANDIDATE -- diff two recorded releases per dimension and per scenario, digest-exact, surfacing new failures and fixed-since.
• hotato serve -- a self-hosted, read-only web app over the local registry (release readiness, scenario matrix, conversation inspector, failure clusters, production health) on 127.0.0.1, bearer-token authenticated. docs/WORKSPACE.md.

The bundled reference agent shows the shape at scale: 25 voice-agent jobs x 5 caller behaviours x 3 audio environments = 375 offline simulated runs, scored across the dimensions and byte-reproducible on replay. Run it with make reference in examples/reference-agent/ (docs/SUITE-RUN.md); the measurement-error harness is docs/BENCHMARK.md.

Connect a production stack

The demo needs nothing. To point Hotato at your live calls, connect once, then sweep on a schedule:

hotato connect vapi                                             # credentials stored 0600, local only
hotato sweep --stack vapi --since 7d --out hotato-sweep.html    # cron, CI, wherever

Run sweep on a timer and it becomes a scheduled batch scanner. Your audio stays on your machine unless you explicitly pull it from your stack. Full guide: docs/SET-AND-FORGET.md · runnable examples/set-and-forget/.

sweep and hotato fleet run can POST a one-line JSON summary to a webhook when they finish, off by default, opt in with --notify (repeatable for more than one URL):

hotato sweep --stack vapi --since 7d --notify https://hooks.slack.com/services/...

The payload carries counts, the top candidate moments (id, kind, timing numbers only), and local artifact paths, never audio, a credential, or transcript text, plus a text field a Slack incoming webhook renders directly. A down or slow webhook never breaks the run: a delivery failure is one warning line on stderr. Egress details: docs/EGRESS.md.

Fleet: private, self-hosted

hotato fleet runs the loop across every agent from one local workspace: ingest calls, surface candidates, label them, and run a before/after experiment that recomputes both sides from audio under a pinned manifest. It recommends a change; it never deploys one. Local mode is stdlib-only (SQLite plus a content-addressed store) with no account and no hosted dependency, and no product limit on how many agents you register.

hotato fleet init    -w acme
hotato fleet agent add -w acme --name support-bot --stack vapi --assistant-id asst_123
hotato fleet ingest  -w acme --agent support-bot call.wav
hotato fleet discover -w acme --agent support-bot call.wav
hotato fleet review   -w acme

Once a workspace has some history, hotato fleet trend -w acme reads the same local SQLite registry and writes one self-contained HTML page: per-agent talk-over and time-to-yield trend lines (p50/p95 per day), candidate moments discovered over time, and experiment outcomes (improved/inconclusive/refused). A day with no measurements gets no point, and a series with fewer than two days of history reports plainly as "not enough history to trend" instead of an interpolated line. Full guide: docs/GUARDIAN-FLEET.md.

Self-host in your own cloud or VPC

The whole team workspace ships as a container. One command stands up the read-only, token-authenticated workspace (hotato serve) on host loopback over a private volume; the default stack makes no external call at run time.

docker compose up -d                    # workspace on 127.0.0.1:8321
docker compose run --rm hotato-init     # optional: seed example data
docker compose --profile judge up -d    # optional: a local Ollama model judge

Build, connect your own calls, the local judge, air-gap, backup, and the zero-migration promise (self-host and cloud share one set of schemas): docs/SELF-HOST.md. Confirm the no-external-calls posture on your own machine with deploy/verify-zero-egress.sh.

Built for coding agents

Hotato is a first-class tool for an LLM or an agent to drive: machine JSON on every command, meaningful exit codes, a described capability manifest, llms.txt, JSON-LD, and an MCP server.

uvx --from "hotato[mcp]" hotato-mcp      # the voice_eval_run scorer + eleven fleet tools

Configs and the tool contract: docs/MCP.md · AGENTS.md · llms-full.txt.

Choose your path

You want to	Run this
Try the full loop, no credentials	hotato start --demo
Sweep the bundled demo calls	hotato sweep --demo
Sweep recent calls from your stack	hotato connect vapi then hotato sweep --stack vapi --since 7d
Add Hotato to an existing repo, CI gate included	hotato init starter --stack vapi --out . (docs/STARTER.md)
Turn a confirmed failure into a portable contract	hotato contract create --from-candidate hotato-sweep.json#1 --expect yield --id refund-cutoff-001 --out contracts (docs/CONTRACTS.md)
Verify contracts in CI	hotato contract verify contracts/ --junit contracts-junit.xml
Attach observability traces to a contract	hotato trace attach contracts/refund-cutoff-001.hotato --trace voice_trace.jsonl (docs/TRACE.md)
Test a candidate fix, before/after, fail-closed	hotato fix trial patch.json --name staging-x --before before/ --after after/ (docs/FIX-TRIAL.md)
Share a finding in a PR or slide	hotato card hotato-sweep.json#1 --out finding.svg
Drive it from a coding agent	uvx --from "hotato[mcp]" hotato-mcp (docs/MCP.md)

contract verify and a promoted fixture in CI are two different guarantees, depending on which recording goes in:

	On the frozen recording (every push)	On a fresh recapture (by hand, see docs/RECAPTURE.md)
Proves	The evidence, policy, and scorer are intact	The CURRENT agent's behavior still matches the label
Does not prove	That the deployed agent has not changed	--

A contract bundle contains call audio. Do not commit a raw customer contract to a public repository; use sanitized fixtures for anything public. See docs/CONTRACTS.md.

Install

Add hotato to a project with pip; or run any command zero-install with uvx:

pip install hotato                 # core: stdlib-only, zero dependencies
pip install 'hotato[neural]'       # optional Silero VAD cross-check
pip install 'hotato[transcribe]'   # optional ASR transcript, context only -- never scored
pip install 'hotato[livekit]'      # LiveKit live capture
pip install 'hotato[pipecat]'      # Pipecat live capture

Contribute a labeled call

The highest-value PR is one labeled dual-channel call. A small, high-integrity corpus of recorded calls with human-labeled turn-taking ground truth is the part of hotato that compounds: the more moments the community labels, the sharper every scorer gets. Add a clip: docs/SUBMITTING.md · the corpus and its schema live in corpus/, with a recorded battery in corpus/vapi-defaults/README.md. Contributor guide: CONTRIBUTING.md.

What Hotato is not

• Conversation QA that shows its work. See docs/COMPARE.md for how the per-dimension scorecard fits alongside runtime voice layers.
• Not transcript scoring. It measures audio timing, not what was said. The opt-in --transcribe flag attaches an ASR transcript purely as context to read next to the verdict; it never changes the timing score. See docs/TRANSCRIBE.md.
• Not speaker ID. Channels are anonymous; nothing identifies who a person is.
• Not semantic intent detection. It produces candidate timing evidence. Humans label intent. CI enforces confirmed contracts.
• Not a hand on production config. It never sits in the live audio path and never changes a running agent.

Docs

• Set-and-forget monitoring (connect once, sweep on a schedule, promote confirmed bugs into fixtures): docs/SET-AND-FORGET.md · runnable examples/set-and-forget/
• Bad call to CI regression test, step by step: docs/BAD-CALL-TO-CI.md · runnable examples/bad-call-to-ci/
• What it measures (the three timing signals, re-derivable by hand): METHODOLOGY.md · Python API docs/API.md
• The fix ladder (each failure names a likely fix class; when the evidence maps cleanly to stack config, Hotato names the setting family and direction): docs/FIX-PLANS.md
• Rule out the non-turn-taking bugs first (STT, buffering, verbosity, refusals, wrong-language): docs/WHY.md
• Pull a call from your stack (Vapi, Twilio, Retell, LiveKit, Pipecat): adapters/README.md · status docs/ADAPTER-STATUS.md
• CI gates: GitHub Action docs/CI.md · pytest plugin docs/PYTEST.md
• Recorded-call battery: 12 scripted calls against a live voice agent on its provider's default settings, where a missed interruption and a false stop on a backchannel fail in the same run, so diagnose refuses to name one threshold: corpus/vapi-defaults/README.md
• Failure contracts and traces: turn a labeled candidate into a portable, CI-verified bundle and attach observability evidence: docs/CONTRACTS.md · docs/TRACE.md · docs/OTEL.md
• Deterministic assertions on the transcript/trace (phrase, PII, policy, tool-call, outcome; never a blended score): docs/ASSERTIONS.md
• Checking the CURRENT agent, not just the frozen recording: the recapture walkthrough: docs/RECAPTURE.md
• Egress: a per-command network table derived from the code, what is local, what reaches your vendor, what optional extras add a hosted call: docs/EGRESS.md
• Layered failure evidence and a before/after fix trial: hotato explain breaks a failing result down by layer, and hotato fix trial tests a candidate change before/after, fail-closed: docs/EXPLAIN.md · docs/FIX-TRIAL.md · docs/APPLY.md · docs/FIX-LOOP.md
• Evidence: what Hotato validates, the input-condition trust matrix, every card and CLI block reproducible, and where Hotato fits alongside broader voice-agent testing tools: docs/VALIDATION.md · docs/TRUST-MATRIX.md · docs/GALLERY.md · docs/EVIDENCE-PACK.md · docs/COMPARE.md
• For coding agents: AGENTS.md · llms.txt · llms-full.txt · MCP server docs/MCP.md · Security SECURITY.md
• Contributing: the highest-value PR is a labeled call fixture: docs/SUBMITTING.md

Why "hotato": good turn-taking is a game of hot potato. Speak, then pass the turn the moment the caller wants it. MIT licensed (LICENSE); the open core stays open.

mcp-name: io.github.attenlabs/hotato