hotato 1.5.4

Conversation QA for voice agents. Catch the calls that pass every text check but talk over the caller, skip a required disclosure, or claim a task that never happened. Self-hosted, offline, MIT.

hotato

Open-source, self-hosted conversation QA for voice agents.

Score every call across five dimensions, each reported on its own (outcome, policy, conversation, speech, reliability), with the evidence behind every result. Deterministic checks and the model-judged rubric never blend into one number.
Runs offline · MIT · zero dependencies.

Your voice agent passes every text assertion and still loses the call. It talks over the caller, skips a required disclosure, or confirms a refund that never posted. hotato scores the conversation and the outcome, with the timing evidence behind every flag, then turns a caught bug into a CI contract so it stays caught.

Quickstart: one command, on your machine

pip install hotato && hotato start --demo

Already have uv? Zero-install, same command:

uvx hotato start --demo

Offline, this sweeps two bundled calls a provider's default agent failed, writes the candidate dashboard, and turns one missed-interruption moment into a demo contract it verifies on the spot:

[start] demo: swept 2 bundled calls, 5 candidate moments;
  wrote hotato-sweep.json, hotato-sweep.html,
  hotato-no-single-threshold.svg,
  contracts/demo-missed-interruption.hotato/contract.json
hotato start: swept the 2 bundled demo calls offline.
  sweep dashboard: hotato-sweep.html
  demo contract:   contracts/demo-missed-interruption.hotato
  verified contract: FAIL as expected -- the demo call
    missed the interruption
  [ ... then the exact next commands: promote a candidate,
    gate it in CI, re-check it ... ]

Open hotato-sweep.html: candidate moments ranked by how far the timing missed, each with a hear-the-bug playhead. This is the artifact you screenshot into a PR.

_{hotato-sweep.html · candidates ranked by salience, each with a playhead synced to the audio. You review and label them into your verdict.}

Point it at your own recording to walk the same loop:

# trust -> scan -> review -> label -> contract
hotato start --stereo my-call.wav

Every command takes a two-channel recording (caller on one channel, agent on the other). A mono file or a bad export is marked NOT SCORABLE, so every verdict rests on inputs that carry the timing evidence.

What hotato catches: five dimensions, reported on their own

hotato test run grades one call against a conversation-test file across five dimensions, each kept in its own count:

• Outcome · did the job get done, graded on tool-call and state evidence, not the transcript's say-so.
• Policy · required disclosures, PII handling, and the compliance phrases your team owns.
• Conversation · the deterministic turn-taking core: did the agent yield when the caller took the floor, and how fast.
• Speech · response latency and the timing around each turn.
• Reliability · pass@1 / pass@k / pass^k across repeated runs with a Wilson interval, so a flaky check reads as flaky.

Deterministic and model-judged results never merge. The deterministic checks (phrase, PII, policy, tool-call, sequence, latency, outcome, all regex, checksum, and span-lookup) set the gate. A rubric verdict is deterministic: false, advisory, and kept in its own count. Every dimension keeps its own line, including --format json, and the scored-output schemas reject an overall_score key outright.

# a starter you edit for your own call
hotato scenario init refund-check --out conversation-test.yaml
hotato test run conversation-test.yaml --agent support-bot

success: FAIL
  (required: all_deterministic_assertions_pass, no_rubric_failure)
per-dimension (grouped view; never blended):
  outcome       0 pass / 0 fail / 1 inconclusive
  policy        0 pass / 0 fail / 1 inconclusive
  conversation  0 pass / 0 fail / 1 inconclusive
  speech        0 pass / 0 fail / 1 inconclusive
  reliability   0 pass / 0 fail / 0 inconclusive

Supply the call as --transcript, --trace, --state, and/or --audio and each check turns pass or fail; a check with no evidence stays INCONCLUSIVE until it arrives. Walkthrough: docs/CONVERSATION-TEST.md.

The receipt: a per-dimension scorecard with the evidence attached

The scored HTML report ends the argument in review: a verdict per event, time-to-yield and talk-over histograms, failure clusters by fix class, and the per-event timeline with both channels and the measured numbers. The same audio and config reproduce every figure.

_{One recording, the pinned scorer, a FAIL against the labeled yield expectation. Share it in a PR with hotato card hotato-sweep.json#1 --out finding.svg.}

The loop: catch, confirm, gate, prove

Surface a moment, confirm what it should have done, pin it to a CI contract, then re-check today's agent.

1. Catch: surface the candidate moments

sweep ranks the talk-over and false-stop moments across your recent calls by how far the timing missed, each a reproducible measurement straight from the open scorer:

_{Level 1 -- candidate. Hotato reports what it measured (0.32s of overlap; 0.46s of trailing silence). Each is a candidate timing moment worth review.}

2. Confirm: you label yield or hold

You label the expected behavior: yield (stop for the caller) or hold (keep talking through a backchannel). Intent stays yours. And because a threshold that stops missing interruptions starts false-stopping on backchannels, when both fail in one run diagnose surfaces the tradeoff instead of naming one threshold:

_{Level 2 -- human-labeled failure. A reviewer confirms the recording broke an explicit yield-or-hold expectation. When a missed interruption and a false stop collide, the fix lives in the engagement-control class.}

3. Gate: pin it to a CI contract

fixture promote saves your labeled call as a permanent regression test. On every push, CI re-scores that recording under the pinned thresholds and scorer, exiting non-zero if the stored evidence stops matching your policy.

4. Prove: re-check the current agent

The frozen recording catches evidence, threshold, or scorer drift. To check that the CURRENT agent still behaves, recapture the same scenario and score a NEW contract under the same policy:

# place the same call against today's agent, capture
# dual-channel, then:
hotato contract create --stereo fresh-call.wav --onset 41.90 --expect yield \
    --id refund-cutoff-001-recapture --out contracts
hotato contract verify contracts/refund-cutoff-001-recapture.hotato

_{Level 4 -- fresh-recapture comparison. A newly captured call meets the same labeled policy and every submitted paired guard held. This is the claim only a fresh recapture can make. Walkthrough: docs/RECAPTURE.md.}

Five levels of evidence, each scored on its own lane

Every card, report, and CLI result names its evidence level; the public tier is the weakest one its inputs support.

Level	Name	What it means
1	Candidate	A candidate timing moment worth human review.
2	Human-labeled failure	A reviewer confirmed this recording broke an explicit yield-or-hold expectation.
3	Stored-evidence check	The historical audio still produces the expected result under the pinned policy and scorer.
4	Fresh-recapture comparison	A newly captured call passed the same contract, and no submitted paired guard regressed.
5	External proof	An independent team confirmed a caught regression or a fresh recapture. Not yet published.

A before/after experiment (hotato fix trial, and the fleet loop) re-derives every verdict from the on-disk audio under one pinned trial manifest, so the number is re-scored from the audio on every run.

Gate it in CI

hotato speaks CI natively.

• Every scoring command exits non-zero when a deterministic check fails and zero when they pass, so a red build is a caught regression.
• hotato contract verify contracts/ --junit contracts-junit.xml writes JUnit XML your runner already renders.
• --format json carries an exit_code field, so an agent or a workflow reads the verdict without parsing prose.
• The model-judged rubric is advisory by default and blocks a build only when you pass --gate.

Drop-in GitHub Action and pytest plugin: docs/CI.md · docs/PYTEST.md. One bad call to a CI gate, step by step: docs/BAD-CALL-TO-CI.md · examples/bad-call-to-ci/.

Scale one call into a release gate

• hotato suite run suite.yaml --agent support-bot -- run a whole suite.v1 offline through the deterministic scripted-caller simulator; every run records into the local registry.
• hotato simulate --matrix scenario.yaml --out ./conv -- render a scenario.v1 into origin=simulated artifacts; the variation matrix expands into hundreds of seeded, byte-identical runs.
• hotato rubric run --rubrics rubrics.yaml --transcript call.json -- the model-judged lane on a pinned local model; zero egress, advisory unless --gate. docs/RUBRIC.md.
• hotato release compare BASELINE CANDIDATE -- diff two recorded releases per dimension and per scenario, digest-exact, surfacing new failures and fixed-since.
• hotato serve -- a self-hosted, read-only web app over the local registry (release readiness, scenario matrix, conversation inspector, failure clusters, production health) on 127.0.0.1, bearer-token authenticated. docs/WORKSPACE.md.

The bundled reference agent shows the shape at scale: 25 jobs x 5 caller behaviours x 3 audio environments = 375 offline simulated runs, scored and byte-reproducible on replay. Run it with make reference (examples/reference-agent/, docs/SUITE-RUN.md); the measurement-error harness is docs/BENCHMARK.md.

Connect a production stack

The demo runs on the bundled calls. To point Hotato at your live calls, connect once, then sweep on a schedule:

# credentials stored 0600, local only
hotato connect vapi
# cron, CI, wherever
hotato sweep --stack vapi --since 7d --out hotato-sweep.html

Run sweep on a timer for a scheduled batch scanner. Your audio stays on your machine unless you explicitly pull it from your stack. Full guide: docs/SET-AND-FORGET.md · examples/set-and-forget/.

sweep and hotato fleet run can POST a one-line JSON summary to a webhook when they finish, off by default, opt in with --notify (repeatable):

hotato sweep --stack vapi --since 7d \
    --notify https://hooks.slack.com/services/...

The payload is metadata-only: counts, top candidate moments (id, kind, timing numbers only), local artifact paths, and a Slack-ready text field. A down or slow webhook leaves the run intact -- a delivery failure is one stderr warning. Egress details: docs/EGRESS.md.

Fleet: private, self-hosted

hotato fleet runs the loop across every agent from one local workspace: ingest calls, surface candidates, label them, and run a before/after experiment that recomputes both sides from audio under a pinned manifest. It recommends a change and leaves the deploy to you. Local mode is stdlib-only (SQLite plus a content-addressed store) and registers as many agents as you want.

hotato fleet init -w acme
hotato fleet agent add -w acme --name support-bot \
    --stack vapi --assistant-id asst_123
hotato fleet ingest -w acme --agent support-bot call.wav
hotato fleet discover -w acme --agent support-bot call.wav
hotato fleet review -w acme

With some history, hotato fleet trend -w acme writes one self-contained HTML page from the same local registry: per-agent talk-over and time-to-yield trends (p50/p95 per day), candidate moments over time, and experiment outcomes (improved/inconclusive/refused). A series with fewer than two days of history reports "not enough history to trend" rather than an interpolated line. Full guide: docs/GUARDIAN-FLEET.md.

Self-host in your own cloud or VPC

The whole team workspace ships as a container. One command stands up the read-only, token-authenticated workspace (hotato serve) on host loopback over a private volume, keeping all traffic on your own infrastructure at run time:

# workspace on 127.0.0.1:8321
docker compose up -d
# optional: seed example data
docker compose run --rm hotato-init
# optional: a local Ollama model judge
docker compose --profile judge up -d

Build, connect your own calls, the local judge, air-gap, backup, and the zero-migration promise (self-host and cloud share one set of schemas): docs/SELF-HOST.md. Confirm the offline posture on your own machine with deploy/verify-zero-egress.sh.

Built for coding agents

Hotato is a first-class tool for an LLM or an agent to drive: machine JSON on every command, meaningful exit codes, a described capability manifest, llms.txt, JSON-LD, and an MCP server.

# the voice_eval_run scorer + eleven fleet tools
uvx --from "hotato[mcp]" hotato-mcp

Configs and the tool contract: docs/MCP.md · AGENTS.md · llms-full.txt.

Choose your path

You want to	Run this
Try the full loop, no credentials	hotato start --demo
Sweep the bundled demo calls	hotato sweep --demo
Sweep recent calls from your stack	hotato connect vapi then hotato sweep --stack vapi --since 7d
Add Hotato to an existing repo, CI gate included	hotato init starter --stack vapi --out . (docs/STARTER.md)
Turn a confirmed failure into a portable contract	hotato contract create --from-candidate hotato-sweep.json#1 --expect yield --id refund-cutoff-001 --out contracts (docs/CONTRACTS.md)
Verify contracts in CI	hotato contract verify contracts/ --junit contracts-junit.xml
Attach observability traces to a contract	hotato trace attach contracts/refund-cutoff-001.hotato --trace voice_trace.jsonl (docs/TRACE.md)
Test a candidate fix, before/after, fail-closed	hotato fix trial patch.json --name staging-x --before before/ --after after/ (docs/FIX-TRIAL.md)
Share a finding in a PR or slide	hotato card hotato-sweep.json#1 --out finding.svg
Drive it from a coding agent	uvx --from "hotato[mcp]" hotato-mcp (docs/MCP.md)

contract verify and a promoted fixture in CI are two different guarantees, depending on which recording goes in:

On the frozen recording (every push)

• Proves: The evidence, policy, and scorer are intact
• Does not prove: That the deployed agent has not changed

On a fresh recapture (by hand, see docs/RECAPTURE.md)

• Proves: The CURRENT agent's behavior still matches the label
• Does not prove: --

A contract bundle contains call audio. Do not commit a raw customer contract to a public repository; use sanitized fixtures for anything public. See docs/CONTRACTS.md.

Install

Add hotato with pip, or run any command zero-install with uvx:

# core: stdlib-only, zero dependencies
pip install hotato
# optional Silero VAD cross-check
pip install 'hotato[neural]'
# optional ASR transcript, attached as context beside the score
pip install 'hotato[transcribe]'
# LiveKit live capture
pip install 'hotato[livekit]'
# Pipecat live capture
pip install 'hotato[pipecat]'

Contribute a labeled call

The highest-value PR is one labeled dual-channel call. That corpus is the part of hotato that compounds: the more moments the community labels, the sharper every scorer gets. Add a clip: docs/SUBMITTING.md · corpus and schema in corpus/, recorded battery in corpus/vapi-defaults/README.md. Contributor guide: CONTRIBUTING.md.

Where Hotato fits

• Conversation QA that shows its work. Where the per-dimension scorecard sits next to runtime voice layers: docs/COMPARE.md.
• Audio-timing scoring. It scores how the conversation is timed. The opt-in --transcribe flag attaches an ASR transcript as context beside the verdict; the timing score stays grounded in the audio: docs/TRANSCRIBE.md.
• Channel-based timing. Each channel is one party (caller on one, agent on the other); Hotato measures the timing between them, and the channels stay anonymous.
• Timing evidence you label. It produces candidate timing evidence; humans label intent, and CI enforces the confirmed contracts.
• Offline and out-of-band. It reads recordings after the call, so your live audio path and running agent stay in your hands.

Docs

• Set-and-forget monitoring: docs/SET-AND-FORGET.md · examples/set-and-forget/
• Bad call to CI, step by step: docs/BAD-CALL-TO-CI.md · examples/bad-call-to-ci/
• What it measures (three timing signals, re-derivable by hand): METHODOLOGY.md · API docs/API.md
• The fix ladder (each failure names a likely fix class and, where evidence maps to config, the setting family + direction): docs/FIX-PLANS.md
• Rule out the non-turn-taking bugs first (STT, buffering, verbosity, refusals, wrong-language): docs/WHY.md
• Pull a call from your stack (Vapi, Twilio, Retell, LiveKit, Pipecat): adapters/README.md · docs/ADAPTER-STATUS.md
• CI gates: Action docs/CI.md · pytest docs/PYTEST.md
• Recorded-call battery (12 scripted calls on a live agent's provider defaults): corpus/vapi-defaults/README.md
• Failure contracts and traces: docs/CONTRACTS.md · docs/TRACE.md · docs/OTEL.md
• Deterministic assertions (phrase, PII, policy, tool-call, outcome; each on its own lane): docs/ASSERTIONS.md
• Checking the CURRENT agent, not just the frozen recording: docs/RECAPTURE.md
• Egress (per-command network table derived from the code): docs/EGRESS.md
• Explain a failure, trial a fix (by layer, fail-closed): docs/EXPLAIN.md · docs/FIX-TRIAL.md · docs/APPLY.md · docs/FIX-LOOP.md
• Evidence (validation, trust matrix, gallery, evidence pack, compare): docs/VALIDATION.md · docs/TRUST-MATRIX.md · docs/GALLERY.md · docs/EVIDENCE-PACK.md · docs/COMPARE.md
• For coding agents: AGENTS.md · llms.txt · llms-full.txt · MCP server docs/MCP.md · Security SECURITY.md
• Contributing: the highest-value PR is a labeled call fixture: docs/SUBMITTING.md

Why "hotato": good turn-taking is a game of hot potato. Speak, then pass the turn the moment the caller wants it. MIT licensed (LICENSE); the open core stays open.

mcp-name: io.github.attenlabs/hotato