Skip to main content

harbor-voice

Evaluate voice agents on phone-call benchmarks, on harbor.

License: Apache-2.0 Python 3.12+ Built on harbor

harbor-voice evaluates voice agents on simulated phone calls. You can use harbor-voice to:

  • Run voice benchmarks like τ-Voice and EVA against speech-to-speech agents.
  • Test agents under realistic callers: accents, background noise, interruptions.
  • Run calls in parallel, locally or in the cloud.
  • Keep every call: the recording, a trajectory and the grade.
  • Add your own voice benchmark.

Benchmarks

  • tau-voice: τ-Voice, the voice mode of Sierra's tau3-bench. Customer-service calls about airline, retail and telecom tasks.
  • EVA: ServiceNow's EVA. Enterprise calls in airline, IT service and medical HR.

Each adapter's README covers its tasks, agents, caller conditions and parity with the original harness. Other agents run on the library's built-in call loop.

Install

You need Python 3.12+ and Docker.

From PyPI:

pip install "harbor-voice[tau-voice,eva]"

The tau-voice and eva extras add each benchmark's commands; the library alone is pip install harbor-voice. Each trial installs the same versions into its environment.

From source, to convert tasks or change the code (needs uv):

git clone https://github.com/amoh-ai/harbor-voice
cd harbor-voice
uv sync   # harbor-voice, both adapters, all extras, dev tools

Each trial then installs harbor-voice and the adapter from these source folders, so a change runs as it is. Prefix the commands below with uv run.

Try it without API keys

The library has offline providers that need no keys and no network. A two-turn call on them takes a few seconds and writes the recording, the metrics and a validity report: see the example under The built-in call loop.

Run a benchmark

Each benchmark needs its API keys in your environment, and a one-time setup of its callers in your ElevenLabs account: see the tau-voice and EVA READMEs. Every call is a live session with paid APIs, so start with one call. Published datasets run without converting anything:

tau-voice job-config --hub-dataset tau-voice-control@0.1.2 \
    --task-names airline-0-d9c07603 --trials 1 -n 1 \
    --model openai.live/gpt-live-1 --agent-config '{"backend_model": "gpt-6-luna"}' \
    -o one.yaml
harbor-voice jobs start -c one.yaml

Then a full job: τ-Voice's leaderboard condition, 3 trials per task with 4 calls at a time, or all of EVA:

tau-voice job-config --hub-dataset tau-voice-regular-test@0.1.2 \
    --trials 3 -n 4 --model openai.live/gpt-live-1 \
    --agent-config '{"backend_model": "gpt-6-luna"}' -o tau.yaml
eva-adapter job-config --hub-dataset eva@0.1.2 \
    --model openai.realtime/gpt-realtime-1.5 -o eva.yaml
harbor-voice jobs start -c tau.yaml
  • The agent. openai.live/gpt-live-1 is OpenAI Live: gpt-live-1 speaks and listens, and the backend_model behind it reasons and calls tools. For OpenAI Realtime, use --model openai.realtime/gpt-realtime-1.5 with no --agent-config.
  • Concurrency. Calls run in real time, a few minutes each, so -n sets a job's wall time. Keep it within your providers' limits on concurrent sessions (ElevenLabs' especially), or calls are refused.
Dataset Tasks What
tau-voice-control 278 every task, control caller
tau-voice-regular 278 every task, regular caller
tau-voice-control-test 100 tau3-bench's test split, control caller
tau-voice-regular-test 100 the test split, regular caller: the leaderboard's condition
tau-voice-parity-control 40 the parity sample, control caller
tau-voice-parity-regular 40 the parity sample, regular caller
eva 213 every task, clean audio
eva-parity 60 the parity sample, 20 per domain

registry.json lists the datasets. Their tasks are in harbor-voice-datasets and their images on ghcr.io/amoh-ai, both pinned, so a version always runs the same thing. Published versions: 0.1.0, 0.1.1 and 0.1.2, the latest. The datasets repo's README says what changed in each and which images each pins. From 0.1.1 the images are built for x86 and ARM; each adapter's README lists the current tags. Without @VERSION, harbor picks the latest. For a condition or subset that isn't published, convert the tasks yourself from a checkout: each adapter's README shows how.

Running on cloud

The orchestrator stays on your machine. The backend creates one environment per trial and deletes it afterwards. Choose it with job-config --env <type>:

  • Daytona. Set DAYTONA_API_KEY and pass --env daytona.
  • EC2. harbor launches a VM per trial and reaches it over SSH; your account's settings go in the job config.
  • Modal, Docker Sandboxes, GKE, Prime Intellect, Vercel. --env modal, sbx, gke, prime or vercel, with that service's sign-in.

Setup for each, and prebuilt images for faster startup: docs/cloud.md.

What a run leaves behind

<trial>/
├── result.json             harbor's trial result: reward, timings, agent
├── agent/
│   ├── trajectory.json     the call, in ATIF-voice
│   └── call/
│       ├── outcome.json
│       └── <adapter>/      each attempt: the recording, the benchmark's own record and grade
└── verifier/               the reward (reward.txt for tau-voice, reward.json for EVA), result.json

CLI

Command What it does
harbor-voice jobs start / trials start harbor's runners, with the agent defaulting to harbor-voice's
harbor-voice status <job> a running job: counts, reward so far, each trial's phase
harbor-voice show <trajectory> summarise a call: turns, timeline, validity
harbor-voice listen <trajectory> --out call.wav write a call's recording to a WAV
tau-voice job-config, voices; from a checkout, converting tasks, build-image, and parity's sample and compare
eva-adapter job-config, agents, judges; from a checkout, convert, build-image, and parity's sample and compare

harbor view, harbor's viewer, shows a call as text. The library's own commands are in docs/library.md.

Development

src/harbor_voice/     the library: providers, the call loop, ATIF-voice, metrics, the CLI
adapters/tau-voice/   the tau-voice adapter (its own package)
adapters/eva/         the EVA adapter (its own package)
tests/                the library's tests (unit/) and each adapter's (adapters/)

From a source install:

uv run pytest
uv run ruff check src adapters scripts tests && uv run ruff format src adapters scripts tests
uv run ty check

Tests that need Docker and the benchmarks' images skip without them; CI runs the rest on Linux and Windows. To contribute, or to add a benchmark, see CONTRIBUTING.md.

License

Apache-2.0; see LICENSE. harbor-voice builds on harbor and depends on it as a package. The adapters run tau2-bench and EVA in containers at pinned commits and don't redistribute them, except the parts copied into the repo, which are under the MIT License: see tau2-bench's and EVA's license files.

Metadata

Release files for harbor-voice 0.1.2

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for harbor-voice 0.1.2
File Size Uploaded Provenance Actions
harbor_voice-0.1.2.tar.gz 324.1 kB Without provenance Details for harbor_voice-0.1.2.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for harbor-voice 0.1.2
File Interpreter ABI Platform Actions
harbor_voice-0.1.2-py3-none-any.whl Python 3 none any Details for harbor_voice-0.1.2-py3-none-any.whl

Total release size: 732.5 kB

Release files / harbor_voice-0.1.2.tar.gz

Download URL harbor_voice-0.1.2.tar.gz
Size 324.1 kB
Tags Source
SHA-256 checksum
How to use checksums
76173678311cd99527d2372186abff4d3a5473edc5d684958edcf4e5d8711f73
BLAKE2b-256 checksum
How to use checksums
5170f1e69dc75c052973738f885bb9fa827212e5a16195e7dc01e22e0a520d55
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via uv/0.13.8 {"installer":{"name":"uv","version":"0.13.8","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"26.04","id":"resolute","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}

Release files / harbor_voice-0.1.2-py3-none-any.whl

Download URL harbor_voice-0.1.2-py3-none-any.whl
Size 408.4 kB
Tags Python 3
SHA-256 checksum
How to use checksums
99c107b2cda101d496e14dae86a1384b8f0cb9e6c09bd7273e0e749a70a655e3
BLAKE2b-256 checksum
How to use checksums
fc4ba73a91b522b2235cf72a22fa1ccd613f9a1a4f0e6206d555d6536333199f
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via uv/0.13.8 {"installer":{"name":"uv","version":"0.13.8","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"26.04","id":"resolute","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}

Security for harbor-voice 0.1.2

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page