hotato 1.12.0

Regression testing for voice agents: turn timing scored from two-channel calls, exit 0/1 in CI; plus say-do verification from your traces and content-addressed failure records that reproduce byte-for-byte. Self-hosted, MIT.

hotato: regression testing for voice agents

hotato.dev

The transcript read clean. The call still failed. Talk-over. Dead air. "You're all set" on a booking that never happened.

Hotato catches it from a two-channel recording or a timestamped transcript, then fails the build until the agent stops failing that call.

Deterministic, byte for byte. Offline. Free, MIT, no account.

First catch in seconds, one command:

$ uvx hotato start --demo
...
Conversation failed: Agent did not yield; measured talk-over was 0.25 s.
    talk-over     0.25s  seconds the agent kept talking while the caller held the floor
    response gap  2.18s  seconds of dead air from the caller's turn end to the reply

It measures turn timing and say-do, not intent.

Catch a failure on your own call

One recording in. The pinned failure becomes a CI gate:

$ hotato investigate ./call.wav
  most likely failure: [1] t=7.63s agent_stop_no_caller
  next: hotato investigate label '.hotato/investigate-state.json#1' --expect yield

$ hotato investigate label '.hotato/investigate-state.json#1' --expect yield
created hotato contract: call-8s-yield  passed: False

$ hotato contract verify contracts
  [FAIL] call-8s-yield (expect yield): did_yield=False talk_over=0.00s
  0/1 contracts pass; exit_code=1

A committed contract is a pinned bad call: the gate stays red until the agent stops failing that call, the same contract a snapshot test gives you.

Quickstart

The five commands are the whole path, first touch to a CI gate that guards every pull request:

# 1. see it catch a failure on two bundled calls (no account; this step exits 0)
uvx hotato start --demo
# 2. score your own recording (or a timestamped transcript: --transcript t.json)
hotato investigate ./call.wav
# 3. commit the caught moment as a regression contract
hotato investigate label '.hotato/investigate-state.json#1' --expect yield
# 4. open the pull request that adds the CI gate
hotato pr create --fixtures contracts/<id>.hotato --repo OWNER/REPO --title 'Add hotato contract <id>'
# 5. the gate re-runs the stored evidence (exits 1 while the pinned call stays red)
hotato contract verify contracts/

Install with pipx install hotato, drive it over MCP with uvx --from "hotato[mcp]" hotato-mcp, or walk the path step by step in docs/GETTING-STARTED.md.

How it works

two-channel recording, or timestamped transcript
  ->  measure turn timing + verify say-do from the trace
  ->  content-addressed contract
  ->  CI verdict: exit 0 pass / exit 1 fail

A catch becomes a contract addressed by its own content, so the exact failure reproduces on any machine.

What it scores

Conversation QA across five dimensions: speech, conversation, outcome, policy, reliability. Each scores on its own, then rolls up into one pass/fail verdict.

Dimension	What it scores
⏱️ Speech	Response latency and turn timing, measured from the two channels.
💬 Conversation	Did the agent yield when the caller took the floor, and how fast.
🎯 Outcome	Was the job done, judged on tool-call and state evidence.
📋 Policy	Required disclosures and PII handling.
📈 Reliability	pass@1 / pass@k / pass^k with a Wilson interval.

Timing comes straight from the two channels; say-do reads your voice_trace.v1 spans, so what the agent told the caller is checked against what the backend did.

Feed it what you already have

Every onramp feeds the same offline scoring and the same 0 / 1 / 2 exit contract.

Traces you already log (no audio needed). Wire the OTel spans you already emit into the same say-do check:

hotato trace ingest --otel traces.jsonl --out voice_trace.jsonl
hotato assert run --trace voice_trace.jsonl --transcript call.transcript.json --assertions assertions.yaml

Details: docs/ASSERTIONS.md · docs/TRACE.md · ground truth: examples/reference-agent, a 375-run offline suite.

Your stack's recorded calls. Vapi, Twilio, and Retell fetch a separated two-channel file; everything scores offline afterwards.

hotato pull --stack vapi --limit 10

Details: docs/CONNECT.md · drive a call against your live agent: docs/DRIVE-A-CALL.md

Scripted fixtures (no production audio). A deterministic scripted caller renders a scenario.v1 labelled origin=simulated; a seeded replay is byte-identical.

hotato simulate --init demo.scenario.json && hotato simulate demo.scenario.json --out ./sim

Details: docs/SIMULATE.md

Point your agent at it

Point Claude Code, Cursor, or any coding agent at this repo: it reads AGENTS.md and runs it end to end, offline, no key.

"Try hotato on the calls in ./recordings and add a CI gate that fails the build on a talk-over regression."

Wire it into CI

The step's exit code is hotato's verdict:

# .github/workflows/voice-qa.yml
name: voice qa
on: [pull_request]
permissions:
  contents: read          # read-only; runs fully offline
jobs:
  hotato:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: attenlabs/hotato@v1.12.0
        with:
          contracts: contracts/          # the catches you committed
          hotato-version: 1.12.0          # exact pin, never a range

Exit-code contract (gate on this, do not parse stdout)

Exit	Meaning
0	every scorable event passed
1	a scorable event regressed
2	usage error or unusable input (bad flags, corrupt file, mono recording, or no scorable event)

Copy-paste workflow with commit-SHA pin: docs/CI.md · docs/CONTRACTS.md.

Drive it over MCP

The MCP server from Quickstart exposes the voice_eval_run scorer plus read/verify/propose tools to any MCP client over local stdio. Setup: docs/MCP.md.

Specifications

Property	Value
Footprint	~10 MiB installed
Core dependencies	0 (stdlib-only)
Reproducibility	byte-for-byte, content-addressed contract
Exit contract	0 pass · 1 fail · 2 refuse
Release integrity	OIDC Trusted Publishing + build-provenance attested
Runtime	offline, off the production audio path

Verify the measurement yourself

Re-run the measurement benchmark

# re-run the measurement-error benchmark on the recorded AMI clips
PYTHONPATH=src python3 -m hotato.benchmark \
  --scenarios corpus/real/scenarios --audio corpus/real/audio

On 13 recorded AMI Meeting Corpus clips, the median error between measured caller-onset and the human word-alignment label is 20 ms. Provenance and caveats: corpus/real/README.md · method: METHODOLOGY.md has the details.

Two channels, one party each

Timing between two voices is measurable only when they arrive on separate channels; a mono or mixed export is marked NOT SCORABLE and refused. It measures timing, not intent: a person labels each candidate moment yield or hold.

hotato trust --stereo call.wav        # per-channel activity, swap flag, scorability

Contribute

Issues and PRs are welcome: CONTRIBUTING.md · SECURITY.md · CHANGELOG

Docs: docs/GETTING-STARTED.md · AGENTS.md · METHODOLOGY.md · docs/START.md · docs/CI.md · docs/CONTRACTS.md · docs/MCP.md

License

MIT (LICENSE)

mcp-name: io.github.attenlabs/hotato