This release has been yanked by its maintainers, and will be ignored by installers, except when explicitly specified.
Consider using release 0.1.2 instead.
EVA on harbor-voice
EVA is ServiceNow's benchmark for conversational voice agents. It scores whole spoken conversations on two dimensions: EVA-A, accuracy (did the agent complete the task correctly and faithfully?), and EVA-X, experience (was the conversation natural, concise and suited to speech?). Its 213 enterprise scenarios cover airline service (50), IT service management (80) and medical HR (83), and target failures specific to voice, such as getting confirmation codes and employee IDs right over the phone. The caller is a bot as well, an ElevenLabs agent with a persona and a goal, so an evaluation runs end to end with no human listener.
This adapter runs EVA's scenarios as harbor tasks, at EVA's commit
039efa0a. EVA's own code decides the prompts, tools, validation and scores,
in a sidecar; harbor-voice runs the call. How it compares with EVA's own CLI:
PARITY.md.
The agent under test is OpenAI Realtime (openai.realtime, e.g.
gpt-realtime-1.5, EVA's reference system), the only agent the adapter takes
for now.
Run it
From a checkout of harbor-voice (uv sync). With pip install "harbor-voice[eva]", agents, judges and job-config work the same
without uv run; converting tasks needs the checkout.
# Once: EVA's two caller agents, created in your ElevenLabs account as EVA's
# setup defines them. Each prints its ID: set EVA_EN_USER_F and EVA_EN_USER_M.
uv run eva-adapter agents show # what would be created (offline)
uv run eva-adapter agents create --agent en_f --yes
uv run eva-adapter agents create --agent en_m --yes
# Tasks: one per EVA record (airline 50, itsm 80, medical_hr 83), and job.yaml
# for the reference system over EVA's k = 3 trials.
uv run eva-adapter convert --out datasets/eva
# One trial, locally (Docker):
uv run harbor-voice trials start -p datasets/eva/airline-1-1-2-17dc1179 \
-m openai.realtime/gpt-realtime-1.5
# The job:
uv run harbor-voice jobs start -c datasets/eva/job.yaml
Environment (names only; nothing is written into a task):
| Variable | Where | What for |
|---|---|---|
OPENAI_API_KEY |
main, eva | the agent under test; EVA's gpt-5.2 judges |
ELEVENLABS_API_KEY |
main | the caller (ElevenLabs Agents) |
EVA_EN_USER_F, EVA_EN_USER_M |
main | EVA's two English caller agents (eva-adapter agents) |
EVA_MODEL_LIST |
eva | EVA's LiteLLM deployments for its judges, as EVA reads it |
GEMINI_API_KEY / GOOGLE_API_KEY, VERTEXAI_*, GOOGLE_APPLICATION_CREDENTIALS |
eva, optional | Gemini, the audio judges (whichever the model list uses) |
AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_SESSION_TOKEN, AWS_REGION |
eva, optional | Bedrock, the faithfulness judge |
JUDGE_MODEL |
eva, optional | EVA's own judge override |
EVA_SIDECAR_METRIC_CONFIGS |
eva, optional | EVA's metric_configs as JSON (EVA_METRIC_CONFIGS' format), handed to EVA by the sidecar; empty is EVA's judges |
any name given to convert --sidecar-env |
eva, optional | whatever your own model list's deployments read (a proxy's URL and key, say) |
The required ones are ${VAR} in the task, so a trial refuses to start
without them; the optional ones are ${VAR:-}. EVA's own EVA_* settings are
not passed: an empty one fails EVA's config, and a set one changes its
protocol. EVA_SIDECAR_METRIC_CONFIGS is the sidecar's channel for the one
EVA setting a run may want, which judges grade (sidecar/server.py); EVA
ignores the name itself.
The judges' model list is judges.json:
EVA_MODEL_LIST="$(eva-adapter judges)". It names every judge
deployment EVA's metrics ask for, as EVA's published setup runs them, and
holds only os.environ/… references, never a key:
| Deployment | Judges | Through |
|---|---|---|
gpt-5.2 |
conciseness, conversation progression (EVA adds service_tier: flex) |
OpenAI, OPENAI_API_KEY |
gpt-5.2-medium |
user behavioural fidelity (validation) | OpenAI, OPENAI_API_KEY, reasoning_effort: medium |
gemini-3-flash-preview |
agent and user speech fidelity (audio), EVA's default | the Gemini API, GEMINI_API_KEY |
us.anthropic.claude-opus-4-6-v1 |
faithfulness | Bedrock, AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY, region from AWS_REGION |
- EVA names
gpt-5.2-mediumbut never defines it. This file reads the name as gpt-5.2 at medium reasoning effort. - EVA's own setup reaches Gemini through Vertex;
gemini-3-flash-previewhere uses a Gemini API key. The model is the same.
The audio judges. A plain run uses EVA's gemini-3-flash-preview
through the Gemini API (GEMINI_API_KEY). To judge audio with another
model, gemini-3.8-flash on Vertex AI say (as the parity run did, PARITY.md
"How it was run"):
- Add its deployment to your own copy of the list, reaching it however you
like:
"model": "gemini/gemini-3.8-flash"with"api_key": "os.environ/GEMINI_API_KEY";"vertex_ai/gemini-3.8-flash"withvertex_project,vertex_location(global) and credentials the container can read (GOOGLE_APPLICATION_CREDENTIALS); or an OpenAI-compatible LiteLLM proxy,"model": "openai/gemini-3.8-flash"with"api_base"and"api_key"asos.environ/<NAME>references. - Convert with
--sidecar-env <NAME> ...for every variable the deployment reads that the table above lacks, so it reaches the sidecar. - Set
EVA_SIDECAR_METRIC_CONFIGS(andEVA_METRIC_CONFIGSfor EVA's own CLI) to EVA'smetric_configsfor the audio metrics, e.g.{"agent_speech_fidelity":{"audio_judge_model":"gemini-3.8-flash","judge_params":{"reasoning_effort":"low"}},"user_speech_fidelity":{...the same...}}.
The sidecar refuses a trial whose judge deployment's api_base names an
empty variable: LiteLLM would otherwise send the judge to its provider's
default endpoint (OpenAI's, for an OpenAI-compatible proxy). On Vertex, keep
reasoning_effort at low or above: the model rejects thinking level
MINIMAL, EVA's default "minimal".
One trial
eva_adapter.trial:drive (the task's [call] driver), in main:
POST /open→ EVA's session for the attempt (prompt, tools, greeting) and its caller (prompt, gender).- The agent is built from the run's
[agent]with EVA's Realtime settings, and itssession.updatemust equal the one EVA's own server rendered for this record, or the trial fails before anything connects. - The call:
RealtimeCallbetweenopenai.realtimeand theelevenlabs.agentscaller (EVA's agent for the persona's gender); tools go toPOST /tool. POST /record(always, with the error if the call failed), thenPOST /validate; EVA's rerun rule decides, up to 3 attempts.POST /gradeif an attempt passed;GET /fileseither way.
Out, under the trial's /logs/agent/call/eva/: summary.json (each
attempt's end reason, validation and timings; usage for the agent, the caller
and each judge), metrics.json (EVA's, for the attempt that counts), run/
(EVA's run directory: config.json, evaluation_summary.json,
records/<id>/trial_<k>/, archived attempts, judge_token_usage/),
attempt-<n>/attempt.json and the caller agent's live config under
elevenlabs/. The ATIF-voice trajectory of the attempt that counts is at
/logs/agent/trajectory.json, audio by content hash.
Reward: [call.config].reward, EVA-A_pass (default), EVA-X_pass or
EVA-overall_pass, is harbor's reward; reward.json also holds all three
composites under their own names, and the job metric reports each one's
mean. A record no attempt validated has no reward: EVA scores
no such record, and the job metric (src/eva_adapter/metrics/eva_mean.py) counts
it as unscored. EVA's run-level numbers (pass@k, pass^k, CIs) come from
sidecar/aggregate.py over the trials' metrics.json.
Verifier: the driver grades through the sidecar itself (the summary and
trajectory need the grade, and main cannot exec into the sidecar), and the
task's tests/test.sh still runs after it and is the reward harbor records,
as in tau-voice. It calls the same /grade, which costs nothing twice:
EVA's MetricsRunner computes only metrics missing from metrics.json.
Trial index: EVA seeds nothing per trial; its trials of a record are
trial_0 … trial_2. The task's seed is that index, and the job lists the
call agent once per seed 0, 1, 2.
Images
The sidecar is EVA's image at the commit plus sidecar/server.py; main
is a slim Python with uv. Tags carry the EVA commit and a digest of our files.
-
Default: each task builds the sidecar where it runs, EVA included, from ServiceNow's public repo at the full commit (
EVA_BASE=eva-from-source: EVA's Dockerfile stage for stage). Works anywhere, Daytona included, with no registry; but every fresh sandbox fetches and installs EVA (minutes). -
For runs of any size: build once and push, then convert against the registry, so every trial pulls the same bytes:
uv run eva-adapter build-image --registry ghcr.io/<org> --push uv run eva-adapter convert --out datasets/eva --registry ghcr.io/<org>
build-imagebuildseva-base:039efa0awith EVA's own Dockerfile from the repo URL if it is missing, the sidecar on it, andmain. -
Local dev:
docker build -t eva-base:039efa0a <eva checkout>, thendocker build adapters/eva/sidecarbuilds on it (the Dockerfile's default). -
Perturbed tasks use a second
main,eva-main:perturbed-<digest>: the same, plus EVA's numpy and audioop-lts pins and EVA's seven noise WAVs (+118 MB of layers: 61 MB of pins, 58 MB of WAVs).build-imagebuilds and pushes it beside the clean one; clean tasks never use it.
Parity
Harbor's job of the sample's records against EVA's own CLI on the same
records (original.eva_command, in EVA's image), then compare writes
harbor's parity_experiment.json. The results, the
exact EVA command, what is compared and how to reproduce it:
PARITY.md.
uv run eva-adapter sample # parity_records.json (30), --all
uv run eva-adapter convert --out datasets/eva-parity --records-from adapters/eva/parity_records.json
uv run eva-adapter compare --harbor <job folder> ... --original <EVA run dir> ... \
--records adapters/eva/parity_records.json
A judge that failed during a run (a 429, say) leaves a trial graded but
without its composite (judge_errored, counted apart); EVA's own rerun of
failed metrics (EVA_RERUN_FAILED_METRICS) recomputes just those scores
from the recordings.
Perturbations
EVA's audio perturbations, under EVA's own names (EVA_PERTURBATION__*),
run by EVA's own AudioPerturbator
(vendored/perturbation.py,
EVA's file at the commit but for where it finds the WAVs; MIT, see vendored/LICENSE)
where EVA's bridge runs it: on every chunk of the caller's audio before the
agent hears it (the recording keeps the clean copy, which EVA's
user_speech_fidelity judges), and as ambient noise whenever the caller is
quiet, the agent's turns included. A new one per attempt, as EVA builds one
per conversation.
| Flag | EVA's setting | Values |
|---|---|---|
--background-noise |
EVA_PERTURBATION__BACKGROUND_NOISE |
airport_gate, baby_crying, background_music, bad_connection_static, coffee_shop, loud_construction, nyc_street, road_noise |
--snr-db |
EVA_PERTURBATION__SNR_DB |
dB, EVA's default 15 (EVA checks nothing else; its UI offers 0–40); needs a noise |
--connection-degradation |
EVA_PERTURBATION__CONNECTION_DEGRADATION=true |
μ-law codec, 3 % packet loss per 20 ms, a random gain (0.5–1.2) per chunk; stacks on a noise |
bad_connection_staticis Gaussian noise at the SNR; the other seven mix a WAV from EVA's repo (assets/noise/). The WAVs are not in this repo: a perturbed task'smainfetches them from GitHub at the full commit id when it is built, and checks each one's SHA-256.- Not supported yet: EVA's accents (they need ElevenLabs agents EVA's
repo does not define) and behaviours (they change the caller's prompt and
agent).
--accentand--behaviorare refused by name. - The randomness is EVA's. Static, packet loss and gain draw from numpy's global generator, unseeded, as in EVA. Two runs of a condition never hear the same noise: compare them statistically, never trial by trial.
- So is the arithmetic. EVA scales noise by each chunk's float32 RMS, and
numpy 2.3 changed how float32 sums over 8192 elements round, so a perturbed
maininstalls EVA's numpy (2.2.6). A trial records the numpy it ran (summary.json, the trajectory'sperturbation_plan).
One condition per dataset. Convert each condition into its own folder;
the condition is in every task's name (airline-1-1-2-17dc1179-coffee_shop-15db,
eva/airline/1.1.2@coffee_shop-15db), its [metadata] and
[call.config.perturbation], so results of two conditions never share a
task; and convert refuses a folder that holds another condition's tasks,
so they never share a job's metric either (harbor pools a dataset folder).
The trajectory's config_matrix.profile is
eva/<condition>. Without the flags, convert writes today's clean tasks
byte for byte.
# A condition, with plain harbor-voice:
uv run eva-adapter convert --out datasets/eva-coffee_shop-15db \
--background-noise coffee_shop --snr-db 15
uv run harbor-voice jobs start -c datasets/eva-coffee_shop-15db/job.yaml
# one trial: harbor-voice trials start -p datasets/eva-coffee_shop-15db/<task> \
# -m openai.realtime/gpt-realtime-1.5
# Static plus connection degradation, on Daytona with prebuilt images:
uv run eva-adapter build-image --registry ghcr.io/<org> --push # adds eva-main:perturbed-*
uv run eva-adapter convert --out datasets/eva-static-10db-degraded --registry ghcr.io/<org> \
--background-noise bad_connection_static --snr-db 10 --connection-degradation
uv run eva-adapter job-config --dataset datasets/eva-static-10db-degraded -n 3 \
--env daytona -o job-static.yaml
uv run harbor-voice jobs start -c job-static.yaml
# Parity under a condition: both sides, one condition per comparison; EVA's side
# gets the same condition as its own EVA_PERTURBATION__* settings.
uv run eva-adapter compare --harbor <job folder> ... --original <EVA run dir> ... \
--background-noise coffee_shop --snr-db 15
The conditions are every noise at any SNR, with or without connection degradation, and connection degradation alone: EVA's own grid.
Tests
In tests/adapters/eva/, with their fixtures;
scripts/goldens/ regenerates the fixtures recorded from EVA's code.
test_eva_perturbation.py: the vendored file against EVA's image; the goldens (EVA's perturbator in EVA's image on seeded inputs, 28 cases: every noise, SNRs, static, degradation, combinations, the ambient stream, a wrap of the noise file, silence, clipping) byte for byte in perturbedmainand, chunk by chunk, here; the bridge's wire, recording and ambient noise; EVA's settings reading the original side's condition in EVA's image. The Docker tests skip without Docker oreva-base:039efa0a.test_eva_sample.py,test_eva_original.py,test_eva_compare.py: the samples (fixed, stratified, the checked-in files), EVA's command (no key in it), andcompareon synthetic outputs of both sides in their real layouts (fixtures/compare): every status counted, run means ± SEM, pass^k and the range overlap, one condition and one audio judge at a time, judge-errored trials, theextendhook, the CLI.test_eva_trial.py,test_eva_convert.py: the driver's attempt loop, the parity guard against EVA's recorded session, the trajectory, the tasks.test_eva_e2e.py: one trial throughdrive_herewith the real sidecar (its judges faked), a fake OpenAI Realtime server and a fake ElevenLabs server on localhost; a validated attempt, and a rerun after EVA's gate rejects one. Skips without Docker oreva-base:039efa0a. No API is called.- The goldens (
test_eva_sidecar_goldens.py,test_eva_call_golden.py,test_eva_realtime_session.py) check each piece against EVA's own code.
Metadata
Release files for harbor-voice-eva-adapter 0.1.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| harbor_voice_eva_adapter-0.1.0.tar.gz | 111.2 kB | Without provenance | Details for harbor_voice_eva_adapter-0.1.0.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| harbor_voice_eva_adapter-0.1.0-py3-none-any.whl | Python 3 | none | any | Details for harbor_voice_eva_adapter-0.1.0-py3-none-any.whl |
Total release size: 238.1 kB
Release files / harbor_voice_eva_adapter-0.1.0.tar.gz
| Download URL | harbor_voice_eva_adapter-0.1.0.tar.gz |
|---|---|
| Size | 111.2 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
62f257d81d094adfa1c92dadf6b7e3495502fbb4edaf7a3412d2d59f58749b9a
|
|
BLAKE2b-256 checksum How to use checksums |
e2950c6dd5cad66bc0d111fabb14b26cc86b51aaffff196f44ae9541c98955ee
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
uv/0.13.7 {"installer":{"name":"uv","version":"0.13.7","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"26.04","id":"resolute","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}
|
Release files / harbor_voice_eva_adapter-0.1.0-py3-none-any.whl
| Download URL | harbor_voice_eva_adapter-0.1.0-py3-none-any.whl |
|---|---|
| Size | 126.9 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
20c3779263c51149bb3b9bedef4025067edc57321307b0416e951b2ada048feb
|
|
BLAKE2b-256 checksum How to use checksums |
85a7c8306f6cee9f7f347cc1998e0899797d20b268011ea9cf7ff2660fc9bdf4
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
uv/0.13.7 {"installer":{"name":"uv","version":"0.13.7","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"26.04","id":"resolute","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}
|
Security for harbor-voice-eva-adapter 0.1.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.