wyoming-bench

Benchmark servers that speak the Wyoming protocol:

• TTS — text-to-speech servers, via the non-streaming (synthesize) and streaming (synthesize-start / synthesize-chunk / synthesize-stop) synthesis paths. Reports latency and throughput.
• STT — speech-to-text (ASR) servers, via the non-streaming and streaming (transcript-start / transcript-chunk / transcript-stop) transcription paths. Reports speed and word-level accuracy.
• gen-corpus — synthesize a test corpus from a TTS server to feed the STT benchmark (a realistic TTS→STT round-trip).
• info — query a server's advertised services (describe/info), so you can see whether it speaks TTS, STT, or both, and whether it supports streaming.

Before benchmarking, sends a describe to confirm the server advertises the service being benchmarked. A server that answers but lacks it (e.g. an STT server for a TTS benchmark) is skipped with a notice instead of timing out on every round. Servers that do not answer describe at all are benchmarked as before. The same describe response also reports whether the server advertises streaming support: when it does not, streaming mode is skipped immediately with a notice (no behavioral probe, no per-round timeout). A server that advertises streaming — or does not report it at all — is probed once to confirm before the streaming rounds run.

Every measurement uses a fresh TCP connection, so results are isolated per round. Statistics (min / mean / median / p95 / max) are aggregated across rounds.

Vibe Warning

This project was vibe coded with Qwen 3.8 27B and the Zed Agent on local hardware.

Install

pipx install wyoming-bench

Usage

# TTS: one server, both modes, default texts (one per 1..4 sentences)
wyoming-bench tts 10.100.1.20:10210

# TTS: multiple servers, streaming only, 5 rounds
wyoming-bench tts a:10700 b:10700 --mode streaming --rounds 5

# TTS: specific voice from a JSON config
wyoming-bench tts a:10700 \
    --config '{"voice": {"name": "en_US-lessac-medium"}, "text_format": "text"}'

# TTS: verbose per-measurement output, extra chunk delay for streaming
wyoming-bench tts a:10700 --mode streaming --chunk-delay 0.2 -v

# STT: one server, both modes, over a corpus of paired recordings
wyoming-bench stt 10.100.1.20:10700 --corpus ./samples

# STT: multiple servers, accuracy + speed comparison, 3 rounds
wyoming-bench stt a:10700 b:10700 --corpus ./samples --rounds 3

# STT: streaming only, pace audio-chunk writes at 0.1s (simulated real-time)
wyoming-bench stt a:10700 --mode streaming --corpus ./samples --chunk-delay 0.1 -v

# Generate a corpus from a TTS server, then benchmark an STT server on it
wyoming-bench gen-corpus tts:10210 --out ./tts-samples \
    --texts "The quick brown fox jumps over the lazy dog."
wyoming-bench stt asr:10700 --corpus ./tts-samples

# Inspect what a server offers before benchmarking it
wyoming-bench info 10.100.1.20:10700

TTS metrics

In streaming mode the text is split into sentences (via the sentence-stream heuristics, which hold abbreviations such as Mr. and U.S. together) and sent one sentence per synthesize-chunk event, so servers that buffer until a certain boundary are exercised the way an LLM-driven pipeline would drive them. --chunk-delay paces those writes.

STT metrics

Speed:

Accuracy (word-level by default, character-level as a bonus):

Reference and hypothesis text are normalized before comparison (lowercased, punctuation stripped, whitespace collapsed).

By default the bench appends --trailing-silence (0.5s) of silence to the end of each recording before sending it. Streaming (online) ASR models commit words incrementally and need a little trailing context to finalize their last words; without it they tend to drop the final one or two, which inflates WER. RTF is still computed on the original recording length, so the padding does not speed up the timing. Set --trailing-silence 0 to send recordings exactly as stored.

Corpus format (STT)

--corpus DIR points at a directory of .wav recordings paired with .txt transcripts that share the same stem:

samples/
  data_01.wav   +   data_01.txt
  data_02.wav   +   data_02.txt
  ...

• data_01.txt holds the reference transcript for data_01.wav.
• Files without a matching counterpart are skipped with a warning.
• The WAV is sent to the server as-is (its sample rate / width / channels are read from the header). Record the corpus in the format the ASR server expects — typically 16 kHz, 16-bit, mono.

Generating a test corpus from a TTS server

Prefer not to record audio? gen-corpus synthesizes text with a TTS server and saves each sample as a .wav plus the source text as its .txt reference. A TTS→STT round-trip then yields a realistic word-error-rate measurement:

wyoming-bench gen-corpus tts:10210 --out ./tts-samples \
    --config '{"voice": "en_US-lessac-medium"}'
wyoming-bench stt asr:10700 --corpus ./tts-samples

The WAV is saved in whatever format the TTS server emits — make sure that format is acceptable to the STT server (most resample internally, but check yours).

Options

All subcommands take the server(s) as positional arguments (SERVER is HOST or HOST:PORT, repeatable). tts, stt, and gen-corpus share these flags (in addition to their own):

TTS-specific flags:

STT-specific flags:

gen-corpus-specific flags:

info-specific flags:

A bare HOST uses port 10700 (the Wyoming default).

Exit codes

0 — at least one measurement succeeded. 1 — every measurement on every server failed (or a server was skipped / no info response). 130 — interrupted.