stt-eval-bench 0.1.1

Self-hostable STT evaluation toolkit: compare providers on WER, latency, cost, and streaming UX

STT Bench

Self-hostable STT evaluation toolkit for comparing speech-to-text providers on quality (WER/chrF), streaming latency, cost, and a dictation-oriented vibe score.

This is not a Whisper demo. It is a reproducible harness for builders choosing STT for dictation and voice-agent systems — CLI + local Docker service + PR-friendly reports.

PyPI package name: stt-eval-bench (the name stt-bench was already taken on PyPI). CLI command remains stt-bench.

Quick start (2 minutes)

pip install stt-eval-bench

stt-bench benchmark run --provider mock --force
stt-bench report --format markdown
stt-bench results leaderboard

No API keys required for mock runs.

From source / editable install:

git clone https://github.com/BrickerP/stt-bench.git
cd stt-bench
pip install -e ".[dev]"

What you get

Surface	Description
CLI	compare, listen, benchmark, report, results
Local service	stt-bench serve or docker compose up — FastAPI + SQLite
Providers	mock, whisper, deepgram, assemblyai (pluggable)
Reports	Markdown suitable for PR comments

Self-host in 60s

docker compose up --build -d
curl -s http://127.0.0.1:8766/healthz
curl -s http://127.0.0.1:8766/version

Details: docs/service.md

With real providers

cp .env.example .env   # OPENAI_API_KEY / DEEPGRAM_API_KEY / ASSEMBLYAI_API_KEY
stt-bench listen 1089-134686-0000_noisy
stt-bench compare --all

Dataset

31 samples in data/manifest.json (LibriSpeech subset + one noisy clip).

stt-bench samples
stt-bench dataset validate

Metrics

WER · chrF · punctuation F1 · TTFT · RTF · cost · vibe score — see stt-bench math.

Contributing

See CONTRIBUTING.md and docs/providers.md.
Maturity / Codex-for-OSS execution gate: docs/OSS_MATURITY.md.
Architecture: DESIGN.md.

License

MIT — see LICENSE. LibriSpeech data: CC BY 4.0.