harbor-voice
Evaluate voice agents on phone-call benchmarks, on harbor.
harbor-voice evaluates voice agents on simulated phone calls. You can use harbor-voice to:
- Run voice benchmarks like τ-Voice and EVA against speech-to-speech agents.
- Test agents under realistic callers: accents, background noise, interruptions.
- Run calls in parallel, locally or in the cloud.
- Keep every call: the recording, a trajectory and the grade.
- Add your own voice benchmark.
Benchmarks
- tau-voice: τ-Voice, the voice mode of Sierra's tau3-bench. Customer-service calls about airline, retail and telecom tasks.
- EVA: ServiceNow's EVA. Enterprise calls in airline, IT service and medical HR.
More voice benchmarks are on the way.
Each adapter's README covers its tasks, agents, caller conditions and parity with the original harness. Support for more agents on both benchmarks is on the way; other agents run on the library's built-in call loop.
Install
You need Python 3.12+ and Docker.
From PyPI:
pip install "harbor-voice[tau-voice,eva]"
The tau-voice and eva extras add each benchmark's commands; the library
alone is pip install harbor-voice. Each trial installs the same versions
into its environment.
From source, to convert tasks or change the code (needs uv):
git clone https://github.com/amoh-ai/harbor-voice
cd harbor-voice
uv sync # harbor-voice, both adapters, all extras, dev tools
Each trial then installs harbor-voice and the adapter from these source
folders, so a change runs as it is. Prefix the commands below with uv run.
Run a benchmark
Each benchmark needs its API keys in your environment, and a one-time setup of its callers in your ElevenLabs account: see the tau-voice and EVA READMEs. Every call is a live session with paid APIs, so start with one call. Published datasets run without converting anything:
tau-voice job-config --hub-dataset tau-voice-control@0.1.0 \
--task-names airline-0-d9c07603 --trials 1 -n 1 \
--model openai.live/gpt-live-1 --agent-config '{"backend_model": "gpt-6-luna"}' \
-o one.yaml
harbor-voice jobs start -c one.yaml
Then a full job: τ-Voice's leaderboard condition, 3 trials per task with 4 calls at a time, or all of EVA:
tau-voice job-config --hub-dataset tau-voice-regular-test@0.1.0 \
--trials 3 -n 4 --model openai.live/gpt-live-1 \
--agent-config '{"backend_model": "gpt-6-luna"}' -o tau.yaml
eva-adapter job-config --hub-dataset eva@0.1.0 \
--model openai.realtime/gpt-realtime-1.5 -o eva.yaml
harbor-voice jobs start -c tau.yaml
- The agent.
openai.live/gpt-live-1is OpenAI Live:gpt-live-1speaks and listens, and thebackend_modelbehind it reasons and calls tools. For OpenAI Realtime, use--model openai.realtime/gpt-realtime-1.5with no--agent-config. - Concurrency. Calls run in real time, a few minutes each, so
-nsets a job's wall time. Keep it within your providers' limits on concurrent sessions (ElevenLabs' especially), or calls are refused.
| Dataset | Tasks | What |
|---|---|---|
tau-voice-control |
278 | every task, control caller |
tau-voice-regular |
278 | every task, regular caller |
tau-voice-control-test |
100 | tau3-bench's test split, control caller |
tau-voice-regular-test |
100 | the test split, regular caller: the leaderboard's condition |
tau-voice-parity-control |
40 | the parity sample, control caller |
tau-voice-parity-regular |
40 | the parity sample, regular caller |
eva |
213 | every task, clean audio |
eva-parity |
60 | the parity sample, 20 per domain |
registry.json lists the datasets. Their tasks are in
harbor-voice-datasets and
their images on ghcr.io/amoh-ai, both pinned, so a version always runs the
same thing. Without @VERSION, harbor picks the latest. For a condition or
subset that isn't published, convert the tasks yourself from a checkout: each
adapter's README shows how.
Running on cloud
The orchestrator stays on your machine. The backend creates one environment
per trial and deletes it afterwards. Choose it with job-config --env <type>:
- Daytona. Set
DAYTONA_API_KEYand pass--env daytona. - EC2. harbor launches a VM per trial and reaches it over SSH; your account's settings go in the job config.
Setup for both, and prebuilt images for faster startup: docs/cloud.md.
What a run leaves behind
<trial>/
├── result.json harbor's trial result: reward, timings, agent
├── agent/
│ ├── trajectory.json the call, in ATIF-voice
│ └── call/
│ ├── outcome.json
│ └── <adapter>/ each attempt: the recording, the benchmark's own record and grade
└── verifier/ the reward (reward.txt for tau-voice, reward.json for EVA), result.json
CLI
| Command | What it does |
|---|---|
harbor-voice jobs start / trials start |
harbor's runners, with the agent defaulting to harbor-voice's |
harbor-voice status <job> |
a running job: counts, reward so far, each trial's phase |
harbor-voice show <trajectory> |
summarise a call: turns, timeline, validity |
harbor-voice listen <trajectory> --out call.wav |
write a call's recording to a WAV |
tau-voice |
job-config, voices; from a checkout, converting tasks, build-image, and parity's sample and compare |
eva-adapter |
job-config, agents, judges; from a checkout, convert, build-image, and parity's sample and compare |
harbor view, harbor's viewer, shows a call as text. The library's own
commands are in docs/library.md.
Development
src/harbor_voice/ the library: providers, the call loop, ATIF-voice, metrics, the CLI
adapters/tau-voice/ the tau-voice adapter (its own package)
adapters/eva/ the EVA adapter (its own package)
tests/ the library's tests (unit/) and each adapter's (adapters/)
From a source install:
uv run pytest
uv run ruff check src adapters tests && uv run ruff format src adapters tests
uv run ty check
Tests that need Docker and the benchmarks' images skip without them; CI runs the rest on Linux and Windows. To contribute, or to add a benchmark, see CONTRIBUTING.md.
License
Apache-2.0; see LICENSE. harbor-voice builds on harbor and depends on it as a package. The adapters run tau2-bench and EVA in containers at pinned commits and don't redistribute them, except the parts copied into the repo, which are under the MIT License: see tau2-bench's and EVA's license files.
Metadata
Release files for harbor-voice 0.1.1
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| harbor_voice-0.1.1.tar.gz | 248.5 kB | Without provenance | Details for harbor_voice-0.1.1.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| harbor_voice-0.1.1-py3-none-any.whl | Python 3 | none | any | Details for harbor_voice-0.1.1-py3-none-any.whl |
Total release size: 552.2 kB
Release files / harbor_voice-0.1.1.tar.gz
| Download URL | harbor_voice-0.1.1.tar.gz |
|---|---|
| Size | 248.5 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
f8f2411bbe6a4248784378d3b5cac753e7bc9af2c77b4dad881f8a98d542b6d7
|
|
BLAKE2b-256 checksum How to use checksums |
474fe8ac784a524736f15b14d30ffe4dddb49b5c4f5c5492acef2a11640cb6bd
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
uv/0.13.7 {"installer":{"name":"uv","version":"0.13.7","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"26.04","id":"resolute","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}
|
Release files / harbor_voice-0.1.1-py3-none-any.whl
| Download URL | harbor_voice-0.1.1-py3-none-any.whl |
|---|---|
| Size | 303.7 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
84277b2c525004372cfcbfe3ab5852141a6e67ef591ba42743124fd56329b472
|
|
BLAKE2b-256 checksum How to use checksums |
099dd9c6d0dad52f1cd6ecf70200b93e0a29a4aae0b64c7782dc0932d4a13926
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
uv/0.13.7 {"installer":{"name":"uv","version":"0.13.7","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"26.04","id":"resolute","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}
|
Security for harbor-voice 0.1.1
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.