outturn

Did the agent produce the right outcome? As a number.

A conversation does not end in words. It ends in an order, a booking, a routed ticket, a qualified lead. That is structured data, and structured data can be compared exactly. So whether your agent got it right does not have to be something you learn from a refund. It can be a percentage that runs on every commit.

  100%  simple_booking                     8/8
   50%  service_switch_duration            4/8  FLAKY
        4x wrong_field: duration_minutes (expected 90, got 60)
  100%  never_confirmed                    8/8

outcome accuracy: 83.3%  (20/24 runs)
flaky scenarios (1): service_switch_duration
a scenario that passes sometimes is a bug that ships sometimes.

---

Why this exists

Most voice agent testing is a person calling the number and listening. That finds the bug in front of you and none of the others, it cannot run in CI, and it produces an opinion instead of a number.

The alternatives are not much better. Asserting on the transcript is asserting on phrasing, which changes every time you touch the prompt. Using a model to grade the conversation means one nondeterministic system judging another.

The outcome is the way out. It is the thing the customer actually receives, it is already structured, and it can be compared exactly.

Three design decisions

Numbers are exact, words are fuzzy. A quantity of 3 is not close to 2, and a 90 minute appointment is not close to a 60 minute one. But "deep tissue massage" and "Deep Tissue Massage (60min)" are the same service. So every number is compared exactly and every word by normalized similarity.

Every scenario runs N times and reports a pass rate. Agents are nondeterministic. One green run is not evidence. A bug that appears one run in five is invisible to a boolean test and obvious in a percentage, so outturn reports rates and flags anything that passed sometimes and failed sometimes as FLAKY. Intermittent is worse than broken, because intermittent hides.

An outcome is repeated things plus scalars. That is all. An order is entries plus a total. A booking is no entries at all, just fields. A triage is fields. One model, no special cases per domain, which is why the same engine handles an ordering agent and a scheduling agent without a line of new code.

Install

pip install outturn

Or from source, if you want the demo scenarios to play with:

git clone https://github.com/OsasDTEch/outturn
cd outturn
pip install -e ".[dev]"

Run it right now

Two deliberately imperfect demo agents ship with the repo, so you can see real output before writing any integration.

outturn scenarios/order   --adapter demo-order   --runs 8 --seed 42
outturn scenarios/booking --adapter demo-booking --runs 8 --seed 42

Each fails on purpose, because an eval you cannot fail is not an eval.

The ordering agent drops a modifier at random under load, which is what FLAKY looks like, and its scope guard is tuned too tight so "do you sell garlic bread" is treated as an off topic question rather than a customer trying to order. That second one is not a strawman, it is behaviour observed on a shipped restaurant agent.

The booking agent forgets to carry the duration across when the customer switches service mid call. A 90 minute massage in a 60 minute slot double books the therapist, and nobody finds out until the day.

Writing a scenario

Turns in, expected outcome out.

An order, which has line items:

id: modifier_stacking
description: >
  Two modifiers on one item, added in a separate turn from the item itself.
tags: [order, modifiers]

turns:
  - "Hi, can I get two large pepperoni pizzas"
  - "Actually make those thin crust"
  - "And extra cheese on them please"

expect:
  entries:
    - name: pepperoni pizza
      quantity: 2
      unit_price: 12.00
      tags: [thin crust, extra cheese]
  total: 24.00

A booking, which has none:

id: service_switch_duration
turns:
  - "Can I book a deep tissue massage for Friday"
  - "Actually make it a sports massage instead"
  - "4:00 pm, and yes please book it"

expect:
  service: sports massage
  day: friday
  time: "16:00"
  duration_minutes: 90
  confirmed: true

Same engine. Anything numeric under an entry becomes an exact comparison. Anything at the top level is a field: numbers and booleans exact, strings fuzzy.

You can also assert on a reply, though use it sparingly since phrasing is the brittle part:

turns:
  - user: "Do you sell garlic bread?"
    expect_reply_contains: ["garlic bread"]

Connecting your own agent

Implement three methods.

from outturn.models import Outcome, Entry

class MyAgent:
    name = "my-agent"

    def reset(self) -> None:
        """Fresh conversation. Called before every run."""
        self.session = start_session()

    def send(self, text: str) -> str:
        """One caller turn in, the agent's reply out."""
        return self.session.turn(text)

    def outcome(self) -> Outcome:
        """The structured result, right now."""
        return Outcome(
            entries=tuple(
                Entry(name=l.name,
                      numbers={"quantity": l.qty, "unit_price": l.price},
                      tags=tuple(l.modifiers))
                for l in self.session.order.lines
            ),
            fields={"total": self.session.order.total},
        )

Then point outturn at it. Your agent lives in your repo, not in this one.

outturn scenarios/ --adapter myproject.agents:MyAgent --runs 10

Anything importable works. If the object is missing one of the three methods, outturn says so before the run starts rather than failing with an AttributeError halfway through.

If your agent cannot hand back a structured outcome, outturn cannot help you, and that is itself the finding. State that lives only in the model's context is not inspectable, and what is not inspectable is not testable. Move the outcome into code and the tool works.

LiveKit

There is a driver at outturn/adapters/livekit.py. It takes a session factory and one callable that reads your state and returns an Outcome, because the outcome does not live in LiveKit, it lives in whatever your own function tools wrote it to.

Verified against livekit-agents 1.8.2. Tested with the ollama.com cloud API (model gemma4:31b, OpenAI-compatible endpoint). Confirmed:

• A session starts with no room and no audio.
• State written by function tools persists across multiple session.run() calls on the same session, so multi-turn scenarios work as written.
• The assistant reply is a ChatMessageEvent in result.events with item.content[0] as the text.

Pin your own install to livekit-agents==1.8.2 until you have tested against a newer version. This API is young and the RunResult shape has changed between releases.

In CI

outturn scenarios/booking --adapter demo-booking --runs 10 --threshold 0.95

Exits non zero if overall accuracy falls below the threshold. Every bug you fix becomes a scenario, so it can never come back silently. --json gives machine readable output for tracking accuracy over time.

The scenarios worth writing

The ones that break agents, roughly in order of how often they do:

• Corrections mid call. The customer changes their mind after the agent has already recorded something. This is where most outcomes go wrong, and the failure is usually a field that did not get updated alongside the one that did.
• Reference without naming. "Make it three" with no item named.
• Cancellation. Added, then removed. Do the derived values follow?
• Confirmation. Did the agent act on intent rather than on an actual yes? Booking a customer who was still thinking is a real and expensive failure.
• Name collisions. Two items or two services that sound alike over a phone line.
• Illegal combinations. Extra cheese on a coke. Should be rejected by schema, not accepted politely and discovered later.
• Out of scope questions that are really orders. "Do you sell X" is a customer trying to buy X.

What this does not test

outturn drives your agent through text. That is fast enough to run in CI on every commit, and it isolates the reasoning and outcome layer from the audio layer.

It does not test speech to text, endpointing or turn taking, and those are real sources of failure, particularly on telephony where audio is narrowband and degrades worst on exactly what matters here: names, numbers and proper nouns.

So this measures whether your agent understands correctly, not whether it hears correctly. Both matter. For the timing half of the picture, see voice-latency-profiler.

Status

v0.1. The core works and is tested, 31 tests. Roadmap, roughly in order:

• LiveKit driver verified against livekit-agents 1.8.2 (ollama.com cloud, gemma4:31b)
• Path assertions: which tools were called, with which arguments
• Pipecat driver
• Audio mode, TTS in and STT out, for end to end runs
• Accuracy tracked over time, so regressions show as a trend

PRD.md and TRD.md in this repo cover the reasoning and the internals.

This project supersedes voice-agent-evals, which asserted on carts only. An order is one kind of outcome among many, and the narrower version was useful to about a tenth of the agents worth testing.

Licence

MIT. Built by Wisdom Omons.