TTSProof

Automated failure-mode QA for text-to-speech systems.

Your TTS pipeline can produce a clip that is empty, half-silent, clipped, stuck in a loop, or three times longer than it should be — and a WER score alone will miss most of it, while plain WER also fails perfectly good audio because the input said 3:30 PM and the transcript said three thirty pee em.

TTSProof runs the checks that catch what actually breaks:

• Structural audio checks (no model needed): empty/truncated audio, duration explosions, long internal silences, clipping, repeated-chunk loop detection, end-of-clip artifacts. Just numpy + soundfile.
• Equivalence-aware WER/CER: expected text and ASR transcript are both canonicalized to spoken form (numbers, decimals, dates, clock times, acronyms, single letters) before scoring — so formatting differences don't count as pronunciation errors.
• ASR-uncertainty quarantine: when audio is structurally clean but ASR disagrees on a very short utterance (a letter, an acronym, "ahh"), the sample is quarantined for human review instead of counted as a failure — because at that length, the ASR is as likely to be wrong as the TTS.

The method was evaluated on a production TTS service — 130 edge cases × 3 voices (390 samples), with a blinded human validation of the quarantine zone — and published as a citable technical report:

An Automated Failure-Mode QA Framework for Neural Text-to-Speech Systems DOI: 10.5281/zenodo.20757553 (CC-BY-4.0)

The headline results are browsable right here in the repo — RESULTS.md (no download needed); the full report with per-sample data is at the DOI above.

Install

pip install ttsproof            # structural checks + metrics + benchmark corpus
pip install "ttsproof[asr]"     # + faster-whisper for pronunciation gating

Benchmark any TTS engine in one command

TTSProof ships a built-in corpus of 817 curated edge cases across 39 categories — numbers, decimals, currencies, dates, ISO timestamps, clock times, time zones, phone numbers, URLs, emails, IP/MAC addresses, file paths, Roman numerals, ordinals, units, abbreviations, acronyms, single letters, pronunciation torture words (Worcestershire, synecdoche, colonel…), proper names (Reykjavík, Nguyễn, Tchaikovsky…), scientific/medical vocabulary, tongue twisters, homographs, Greek, Norwegian, mixed-language lines, math, punctuation abuse, hallucination traps, emoji, SQL/JSON/markup, and more.

The corpus is versioned independently of the software (this release: Benchmark Corpus 1.0) — published scores stay comparable across tool updates, and every report records both versions:

# your engine as a command template ({text} in, {out} wav path out):
ttsproof benchmark --cmd "mytts --text {text} --wav {out}"

# or score audio you already generated (files named <case_id>.wav):
ttsproof generate --out cases.jsonl        # export the corpus, synthesize it your way
ttsproof benchmark --wav-dir ./my_audio

You get a category scoreboard in the terminal…

  numbers                 98.3%   59/60 decided
  dates                   96.7%   29/30 decided
  urls                    88.9%   8/9 decided   (+0 quarantined)
  norwegian               95.0%   19/20 decided
  ----------------------------------------------------------
  OVERALL 96.1%   pass=485 fail=20 quarantine=23

…plus report.html — a self-contained page with score bars, every failure's waveform, an audio player, and what the ASR actually heard.

Each category is scored by an honest policy: strict (unambiguous spoken form — equivalence-aware WER), keywords (URLs/currencies have many valid readings — key tokens must survive the round trip), or structural (emoji and punctuation storms have no meaningful transcript — the audio just has to survive). No fake failures from formatting differences.

CI regression gate:

ttsproof regress baseline/report.json current/report.json --tolerance 1.0
# exit 1 + category-level diff when quality drops:
#   REGRESSION DETECTED:
#     OVERALL: 96.2% -> 94.7%  (-1.5 pp)
#     numbers: 99.1% -> 95.0%  (-4.1 pp)

Compare engines:

ttsproof compare xtts/report.json fish/report.json kokoro/report.json

Testing closed-source models (ElevenLabs, OpenAI, …) via SpeechSDK

You can benchmark commercial/closed-source models using SpeechSDK. See our SpeechSDK integration example for a ready-to-use wrapper script.

Integration suggested by u/pmarks98 (Jellypod / SpeechSDK).

Quickstart

Check one file (CLI):

ttsproof check output.wav --text "Hello there"

QA a folder of generated audio against a manifest:

# cases.jsonl — one case per line:
# {"id": "case_001", "text": "Meet me at 3:30 PM", "wav": "case_001.wav"}
ttsproof run --manifest cases.jsonl --wav-dir ./audio --out ./reports --asr

You get report.csv + report.json with one verdict per sample: pass / hard_fail / quarantine.

Gate any TTS system in CI (Python):

import ttsproof

def synthesize(text: str) -> bytes:
    ...  # call your TTS engine, return WAV bytes

cases = ttsproof.load_cases_jsonl("edge_cases.jsonl")
rows = ttsproof.qa_synthesize(cases, synthesize, out_dir="qa_audio")
report = ttsproof.write_reports(rows, "qa_reports")
assert report["ok"], report["summary"]

Or check existing audio with three lines:

import ttsproof

report = ttsproof.check_wav("output.wav")          # structural only
print(report.ok, report.errors)

Why "quarantine" instead of pass/fail?

Short utterances are where reference-based TTS systems break — and also where ASR is least reliable. In the published evaluation, a blinded human review of the ASR-uncertain zone found it was a genuine ~45/55 mix of real TTS failures and ASR false-negatives. Treating that zone as "needs human ears" is the honest design: hard failures stay automatic, uncertain shorts get a human, nothing gets silently mislabeled.

What it doesn't do

• It does not judge naturalness, prosody, or speaker similarity — it catches defects, not aesthetics.
• ASR-based checks inherit ASR's limits; that is exactly why the quarantine verdict exists.
• English-first normalization (with Greek letter support); contributions for other languages welcome.

Developing, and how a release is cut

git clone https://github.com/Mormolykos/ttsproof.git && cd ttsproof
pip install -e ".[dev]"       # ruff, mypy, pytest, build, twine, pyyaml
pytest -q                     # 62 tests

Run the whole CI workflow locally before pushing:

python scripts/ci.py run                       # every job, ~82s
python scripts/ci.py run --job test --python 3.13   # one cell of the matrix
python scripts/ci.py run --list                # what it would do, without doing it

It reads .github/workflows/ci.yml and executes the run: steps it finds there, one throwaway virtual environment per job — the same isolation GitHub gives each job. Steps it cannot reproduce (actions/checkout and friends) are printed as NOT-LOCAL rather than skipped quietly. uv is used when present, so --python 3.13 means 3.13 even on a machine that has 3.10.

Individual gates: ci.py attribution, version, pypi, contract, noskips, wheelcheck, artifact. They use ttsproof's own exit-code convention — 0 pass, 1 fail, 2 could not judge — so a network blip while checking PyPI is a 2, never a 1.

CI runs twelve matrix cells: Linux, Windows and macOS across Python 3.10–3.13. macOS is here and not in the sibling libraries for one reason — soundfile binds to libsndfile, a C library the three platforms ship differently.

docs/adr/001 covers the lint ruleset and what was excluded from it, why mypy runs at python_version = "3.12" rather than at the 3.10 floor, and what rollback means when PyPI will not let a version be replaced.

Cite

@techreport{gkilis2026ttsqa,
  author = {Gkilis, Panagiotis},
  title  = {An Automated Failure-Mode QA Framework for Neural Text-to-Speech
            Systems: A Production Case Study on a Reference-Based TTS Service},
  year   = {2026},
  doi    = {10.5281/zenodo.20757553}
}

License

MIT © Panagiotis Gkilis — portfolio · engineering notebook · part of the Proof family with BookProof

---

Who built this, and what he sells

Built and maintained by Panagiotis (Panos) Gkilis — solo founder, BedVibe Studios. This library is MIT and always will be. These are not:

• Available for hire. Remote ML/AI engineering — training pipelines, evaluation methodology, retrieval systems, inference infrastructure. What I have shipped and measured: ai.bedvibe.studio/work
• Licensed emotional speech datasets — multilingual, studio-recorded with cleared and paid voice actors, six emotional states, commercial licence: tts.bedvibe.studio/datasets
• BedVibe TTS — a 730M-parameter expressive text-to-speech model and platform, live and in production: tts.bedvibe.studio

If this library saved you time, the most useful thing you can do costs nothing: link to it from wherever you write about it. A followed link is worth more than a star, and it is the one thing an author of free software cannot give himself.