bolo-speech 0.2.0

Ollama for speech: a small, PyTorch-free server that pulls speech models by name and serves speech-to-text, text-to-speech and voice-activity detection over the APIs your clients already use.

bolo (बोलो) is Hindi for "speak".

_{The demo runs speech recognition entirely in your browser with WebAssembly: no install, no upload, works offline after the first visit.}

_{The speech-to-text playground, live against a real server on a 4-core CPU. Lighter text is still changing; solid text is final.}

• Drop-in. OpenAI-compatible /v1/audio/transcriptions, /v1/audio/speech and /v1/models; the model-management routes of speaches; the Wyoming protocol for Home Assistant.
• Light. No PyTorch anywhere. Inference runs on ONNX Runtime (through sherpa-onnx) and CTranslate2 (through faster-whisper). The base install has no engine at all; you add the ones you want.
• Realtime on CPU. A low-latency tier, true streaming models, and stabilised partial results for models that were never designed to stream. Measured numbers below, from a 4-vCPU machine.
• A real server. Per-model queues with micro-batching, backpressure, cancellation, lazy loading, idle unloading and a memory budget, with every request traced phase by phase.
• Indian languages and phone audio. Speech recognition for all 22 scheduled languages of India, 16 of them run here on real recordings; voices for Hindi, Bengali, Telugu, Tamil, Malayalam, Marathi, Urdu, Nepali, Gujarati, Kannada, Maithili, Bhojpuri, Magahi and Chhattisgarhi, plus non-commercial MMS voices for Punjabi, Odia, Assamese and Dogri; 8 kHz G.711 μ-law/A-law in and out.
• No GPL needed for English and Hindi voices. Phonemes come from uccharan (Apache-2.0), which matches eSpeak NG on our round-trip tests; eSpeak NG stays an optional choice, per server, per voice or per request.
• Voice agents. An OpenAI Realtime API endpoint (/v1/realtime) and audio in /v1/chat/completions: speech in, your own language model in the middle (Ollama, llama.cpp, vLLM, any OpenAI-compatible endpoint), streamed speech out, with barge-in and per-turn latency traces, over WebSocket or WebRTC. Run against Ollama and llama.cpp, and from Pipecat and LiveKit Agents.
• Who said what. Speaker diarization without PyTorch, as diarized_json, in subtitles, on words, and as speaker labels on live transcripts; known speakers are named from reference recordings.
• Honest. 253 models in the catalog, each with its licence; the 108 that were actually run on CPU are marked, with what was measured. Everything else says untested.

Status: 0.2, alpha. Developed and measured on Linux x86-64, CPU only. GPUs, Apple Silicon, arm64 and Windows are untested; see What is not tested.

The console

Playground. Talk, upload a file or play a sample; see partial and final text, and where the wait after you stop speaking went.	Latency. Live percentiles against your target, the distribution, and a waterfall of every request with its phases.
Browser vs server. The same audio through the server and through a model running in the page, with both timelines.	Text to speech. Streamed playback, time to first audio, a per-chunk timeline, underrun detection, and a round trip back through recognition.
Models. Search and filter the catalog, pull with progress, load and unload, and see what was actually validated on CPU.	Benchmark. Sweep concurrency, chart latency and throughput, and find the ceiling for your latency target.

   

_{Left: the latency timeline for one utterance. Right: the same screen at phone width. A light theme is included (screenshot), as is the browser-only live demo. Indian languages are listed under their own names (Hindi on the server).}

_{All screenshots are of the real console, taken on a 4-core CPU with no GPU. See docs/assets/README.md for what each one was taken against.}

Quick start

pipx install 'bolo-speech[all] @ git+https://github.com/cb-Sigma/bolo#subdirectory=server'
bolo serve

(pip install into a virtual environment works the same way. Once the package is on PyPI this becomes pipx install 'bolo-speech[all]', or pipx run --spec 'bolo-speech[all]' bolo serve.)

Open http://127.0.0.1:8000 for the console, or use the API. The first request for a model downloads it:

# speech to text (pulls whisper-base, about 160 MB, the first time)
curl http://127.0.0.1:8000/v1/audio/transcriptions -F file=@recording.wav -F model=whisper-1

# text to speech (pulls kokoro-v1.0-int8, about 170 MB, the first time)
curl http://127.0.0.1:8000/v1/audio/speech -H 'Content-Type: application/json' \
  -d '{"model": "tts-1", "voice": "af_heart", "input": "Hello from bolo."}' -o hello.mp3

Or choose models yourself, the Ollama way:

bolo list --all -q english        # browse the catalog
bolo pull moonshine-tiny-en       # 124 MB, the fastest English recogniser here
bolo transcribe recording.wav -m moonshine-tiny-en
bolo speak "Hello there." -o hello.wav -m kokoro-v1.0-int8 --voice af_heart
bolo ps                           # what a running server has in memory
bolo doctor                       # engines, codecs and phonemiser found on this machine

With the official OpenAI SDK, only the base URL changes:

from openai import OpenAI

client = OpenAI(base_url="http://127.0.0.1:8000/v1", api_key="not-needed")
text = client.audio.transcriptions.create(model="whisper-1", file=open("recording.wav", "rb")).text
client.audio.speech.create(model="tts-1", voice="alloy", input=text).write_to_file("echo.mp3")

Installation extras: sherpa (most models), faster-whisper, av (M4A/WebM/AAC through bundled FFmpeg), wyoming, uccharan (phonemes for English and Hindi voices), all (those five), and espeak (see below; never part of all).

Phonemes without the GPL. Piper-format voices, KittenTTS, Matcha and Kokoro read phonemes. For English (US, British) and Hindi they come from uccharan (Apache-2.0, in bolo-speech[all]), so those voices work as installed. Voices in other languages need the eSpeak NG phonemiser, which is GPL-3.0-or-later and not shipped: install it when you choose to, with one click in the console (Models → Components), bolo components install espeak-ng --accept-license, pip install 'bolo-speech[espeak]', or your system's package (apt install espeak-ng). It takes effect without a restart; until then those voices answer with a clear component_missing error. You can also prefer eSpeak NG for any voice (docs/models.md). Details: Licences.

What it does

Protocols

OpenAI HTTP	transcriptions (json, text, srt, vtt, verbose_json with word timestamps, SSE streaming), translations, speech (mp3, opus, aac, flac, wav, pcm, chunked as it is synthesised), models
speaches routes	/v1/registry, POST/DELETE /v1/models/{id}, /v1/audio/voices, /api/ps, /v1/audio/speech/timestamps and its environment variables
OpenAI Realtime API	WS /v1/realtime: transcription sessions, and conversation sessions (speech → your LLM → speech) with server VAD, an optional end-of-turn model (semantic_vad), barge-in, incremental user transcripts, text turns and function calling. GA event names, tested with the official SDK's realtime client, Pipecat and LiveKit Agents. The same sessions over WebRTC (POST /v1/realtime/calls, extra webrtc) with ephemeral keys for browsers (POST /v1/realtime/client_secrets)
Chat completions with audio	POST /v1/chat/completions: input_audio parts are transcribed, the request goes to your LLM, the reply is spoken when modalities includes audio (streaming too). Everything else is proxied unchanged
Speaker diarization	diarize=true or response_format=diarized_json on transcriptions (with word timestamps and known_speaker_names[] / known_speaker_references[] if you like), POST /api/v1/diarize, diarize=true on the realtime WebSocket with revised labels at the end of the stream
Realtime WebSockets	WS /api/v1/stt/stream (partials, finals with word times, VAD events, optional speaker labels) and WS /api/v1/tts/stream (text in pieces, audio out)
Wyoming	speech-to-text and text-to-speech on one TCP port, including streaming synthesis
Native JSON API	catalog, pulls with progress, load/unload, optional components, live statistics, request traces, Server-Sent Events, benchmark jobs
Operations	Prometheus /metrics, /healthz, /readyz, optional bearer key, CORS

Server behaviour

• Models load on first use, unload after five idle minutes, and are evicted least-recently-used first under a memory budget. A newly installed model is first loaded in a child process, so a broken file cannot crash the server.
• Each model has a bounded queue. Requests that arrive while the engine is busy are decoded together in one batched call; when the queue is full the answer is 429 with Retry-After, not a growing delay. Work for clients that hung up is dropped.
• Long recordings are split at pauses (Silero VAD) and the pieces decoded as a batch; with stream=true you get text as each piece finishes.
• Downloads resume after interruptions and are verified (SHA-256 for LFS files) into a content-addressed cache.

The console (served at /): a playground for both directions with a latency timeline, the model catalog, a live latency dashboard with a per-request waterfall, and a benchmark page that sweeps concurrency.

Supported models

Task	Family	Models	Languages	Validated on CPU	Best measured RTF
DIARIZATION	pyannote + speaker embeddings	3	-	1	0.106 (pyannote-3.0-eres2net)
DIARIZATION	Nemotron diarization	1	-	1	0.039 (nemotron-3-diarization)
TTS	Parler-TTS	2	as, bn, brx, doi, en, gu, hi, hne +13	2	4.540 (parler-tts-tiny-v1)
TTS	F5-TTS	2	as, bn, en, gu, hi, kn, ml, mr +5	1	119.700 (indic-f5)
TTS	Veena	1	en, hi	0	-
TTS	Svara	1	as, bho, bn, brx, doi, en, gu, hi +11	0	-
STT	SraVaani	1	as, bn, en, gu, hi, kn, ml, mr +6	0	-
STT	IndicConformer	27	as, bn, brx, doi, gu, hi, kn, kok +14	20	0.060 (indicconformer-ta-community)
SEPARATION	SepFormer	2	en	2	1.440 (sepformer-whamr)
SEPARATION	PixIT	1	en	1	5.690 (pixit-ami)
STT	DiCoW	2	en	2	0.453 (dicow-v3.3)
TTS	Indic-Speak	1	as, bn, brx, doi, en, gu, hi, kn +15	0	-
TTS	Spark-TTS	1	en, zh	0	-
TTS	MaskGCT	1	de, en, fr, ja, ko, zh	0	-
DIARIZATION	DiariZen	1	-	0	-
SEPARATION	VoxLock	1	en	1	0.460 (voxlock-int8)
STT	Whisper	38	af, am, ar, as, az, ba, be, bg +92	20	0.081 (faster-whisper-tiny.en)
STT	Moonshine	11	ar, en, es, ja, ko, uk, vi, zh	4	0.033 (moonshine-tiny-en)
STT	Parakeet	7	bg, cs, da, de, el, en, es, et +17	4	0.030 (parakeet-tdt-ctc-110m)
STT	NeMo FastConformer	8	be, de, en, es, fr, hr, it, pl +2	3	0.334 (nemo-fastconformer-ctc-en-streaming-480ms)
STT	Canary	1	de, en, es, fr	1	0.358 (canary-180m-flash)
STT	Zipformer	16	ar, bn, de, en, es, fr, id, ja +5	7	0.028 (zipformer-en-gigaspeech)
STT	SenseVoice	1	en, ja, ko, yue, zh	1	0.039 (sense-voice-small)
STT	Paraformer	2	en, zh	1	0.091 (paraformer-zh)
STT	Dolphin	2	ar, bn, fa, gu, hi, id, ja, jv +26	1	0.021 (dolphin-base)
STT	Omnilingual ASR	1	ar, as, bn, de, en, es, fr, gu +23	1	0.112 (omnilingual-ctc-300m)
STT	Nemotron	1	ar, bg, cs, da, de, el, en, es +26	1	0.312 (nemotron-3.5-streaming-560ms)
TTS	MMS-TTS	25	as, awa, bgc, bn, de, doi, en, es +17	1	0.196 (mms-tts-eng)
TTS	Piper	85	ar, bn, ca, cs, cy, da, de, el +35	28	0.040 (piper-ne-np-google-medium)
TTS	External TTS server: llama-server	10	ar, da, de, el, en, es, fi, fr +15	1	0.716 (pocket-tts-en-llamacpp)
TTS	External TTS server: audio.cpp	40	ar, be, bg, bn, cs, da, de, el +41	0	-
TTS	Kokoro	2	en, es, fr, hi, it, ja, pt, zh	2	0.759 (kokoro-v1.0)
TTS	KittenTTS	4	en	4	0.107 (kitten-nano-0.8)
TTS	Matcha-TTS	2	en, zh	1	0.205 (matcha-en-ljspeech)
TTS	MeloTTS	1	en	1	0.278 (melo-tts-en)
TTS	VITS	26	bho, bn, en, gu, hi, hne, kn, mag +4	4	0.046 (mimic3-gu-cmu-indic)
TURN	Smart Turn	1	ar, bn, da, de, en, es, fi, fr +15	0	-
VAD	Silero VAD	2	-	2	0.006 (silero-vad-v5)

334 models in the catalog, 119 validated on CPU, 215 untested. Full list: docs/models.md.

"Validated on CPU" means downloaded and run on the machine described under Performance by server/tools/validate_models.py; the table in docs/models.md has every model with its measured real-time factor and licence. Anything else that sherpa-onnx or faster-whisper can load can be added with bolo pull hf:<owner>/<repo>.

Models that need PyTorch at run time (Indic Parler-TTS, IndicF5, Veena, Svara, SraVaani, XTTS, Chatterbox and other LLM-based voices) do not run inside bolo. bolo drives them through an out-of-process engine plugin (docs/plugins.md): the PyTorch plugin is the separate package bolo-torch (Indic Parler-TTS, IndicF5, Veena, Svara, SepFormer and PixIT separation; slower than real time on CPU), and until it is installed those catalog entries show "needs plugin torch" with its install command. bolo runs no language models itself: voice agents use the one you connect.

Licences are labelled, not filtered. The catalog includes non-commercial models (MMS-TTS, some Piper voices, non-English Moonshine) and gated ones (Indic Parler-TTS, SraVaani, the official IndicConformer): every entry states license, commercial_use and gated, the console shows them, and you decide what to download. Gated models are fetched only with your Hugging Face token.

The realtime tier

The default realtime model is moonshine-tiny-en (27M parameters, MIT); zipformer-en-20m-streaming is the true-streaming alternative. One live stream, audio sent at real-time pace through the same code path as the WebSocket, on the 4-vCPU machine described under Performance:

Model	How it streams	First partial (p50 / p95)	Final after end of speech is detected (p50 / p95)	Notes
zipformer-en-20m-streaming	native, incremental	19 / 21 ms	22 / 30 ms	lower-case, no punctuation; least accurate of the four, and it can drop the first words of audio that starts abruptly
moonshine-tiny-en	re-decode + LocalAgreement	62 / 69 ms	420 / 465 ms on a 12 s utterance; 161 / 196 ms on a 3 s one	punctuated; the default
whisper-tiny (sherpa-onnx)	re-decode + LocalAgreement	95 / 111 ms	918 / 978 ms	99 languages
faster-whisper-tiny	re-decode + LocalAgreement	570 / 640 ms	860 / 885 ms	99 languages, word timestamps

These are latency_ms values: server time from receiving the last audio that contributed to the text until the message is sent. They do not include the pause the endpointer waits for before it calls an utterance finished, which is 500 ms by default and usually the larger part of what a user perceives. Measured from the true end of speech to the final text with moonshine-tiny-en on a 3-second utterance: 355 ms with endpointing=fast (250 ms), 615 ms with the default (500 ms), 1,212 ms with patient (900 ms).

Every other speech-to-text model also works on the realtime WebSocket; it is just slower. Word error rates on a LibriSpeech test clip and the full table are in docs/models.md.

How it works, and how to tune it: docs/realtime.md.

Indian languages and telephony

Recognition: all 22 scheduled languages. One IndicConformer model per language (AI4Bharat, MIT, ungated community ONNX exports pinned to a commit): Assamese, Bengali, Bodo, Dogri, Gujarati, Hindi, Kannada, Kashmiri, Konkani, Maithili, Malayalam, Manipuri, Marathi, Nepali, Odia, Punjabi, Sanskrit, Santali, Sindhi, Tamil, Telugu and Urdu. Sixteen were run here on real recordings (FLEURS, RESPIN, a Sanskrit corpus; 10 to 60 utterances each): character error rates from 2.5 % (Bengali) to 14.9 % (Assamese), for example Hindi 7.0 %, Tamil 10.2 %, Maithili 11.6 %. Bodo, Dogri, Kashmiri, Konkani, Manipuri and Santali are in the catalog untested, for lack of public test recordings. Also: Hinglish written in Latin script (faster-whisper-base-hinglish), Whisper fine-tunes for Hindi, Tamil, Kannada, Telugu, Bengali and Punjabi, a streaming Bengali Zipformer and NVIDIA's streaming Nemotron (Hindi).

Bhojpuri, honestly. The Hindi model is the best Bhojpuri recogniser we can run: 41 % word error rate (13 % character error rate) on 120 Bhojpuri utterances, against 12 to 15 % on Hindi. The permissively licensed Whisper-small Bhojpuri fine-tune, converted here, did worse on the same audio (48 % / 22 %), so it is not in the catalog.

Synthesis. Hindi (three Piper-format voices, four Kokoro voices, two SYSPIN voices; all without eSpeak NG), Telugu (three voices), Tamil (two), Malayalam (two), Marathi, Bengali, Urdu, Nepali and Gujarati. The Telugu, Tamil, Malayalam, Marathi and newer Bengali and Urdu voices are the Piper project's own files, converted for sherpa-onnx by bolo right after the download (bolo pull piper:<voice> does that for any Piper voice); each was synthesised and transcribed back by the recogniser of its language (character error rate 0 to 7 %). Maithili, Bhojpuri, Kannada, Magahi and Chhattisgarhi voices (IISc SYSPIN, MIT) are converted by this project and published. Meta's MMS-TTS voices (non-commercial) for Punjabi, Odia, Assamese, Dogri and fourteen more are converted by this project and published too (also on Hugging Face under codebucketsolutions/). No voice that runs without PyTorch for Sanskrit, Sindhi, Kashmiri, Konkani, Manipuri, Bodo or Santali. Sentences are split at the danda, Hindi numbers are spelled out in the Indian system (250000 → दो लाख पचास हज़ार), and code-mixed text passes through untouched.

Licence classes: P permissive (MIT, Apache-2.0, CC-BY, CC0), NC non-commercial, G gated, T needs the PyTorch plugin (bolo-torch).

Language	STT	TTS
Hindi	P, run here (also G/T: SraVaani, official IndicConformer)	P (Piper, Kokoro, SYSPIN; some Piper voices NC); NC MMS; G/T Indic Parler-TTS, IndicF5
Bengali, Telugu, Tamil, Malayalam, Marathi, Urdu, Nepali, Gujarati	P, run here	P (some Piper voices NC); NC MMS for most
Kannada, Maithili	P, run here	P (SYSPIN); NC MMS
Magahi, Chhattisgarhi	Hindi model only	P (SYSPIN); NC MMS (run here)
Bhojpuri (not a scheduled language)	Hindi model (41 % WER)	P (SYSPIN)
Punjabi, Odia, Assamese	P, run here	NC MMS (run here); G/T Indic Parler-TTS
Dogri	P, untested	NC MMS (run here)
Sanskrit, Sindhi	P, run here (Sindhi without an error rate)	G/T only (Indic Parler-TTS)
Bodo, Kashmiri, Konkani, Manipuri, Santali	P, untested	none (Bodo, Kashmiri: G/T or T only)

Checked end to end: besides the runs above, Hindi text → piper-hi-in-rohan-medium → faster-whisper-tiny-hindi returns the sentence with 0% word error rate in the test-suite. All of these are small samples that show the pipelines work; they are not accuracy benchmarks, and nobody who speaks the languages reviewed the synthesised audio.

Telephony. 8 kHz G.711 is native in both directions: encoding=mulaw|alaw on the realtime WebSocket, raw audio/basic uploads, and "response_format": "mulaw" for speech, with helpers for 20 ms RTP-sized frames. The codecs match the reference implementation bit for bit. In the test-suite, text → Kokoro → 8 kHz μ-law → 160-byte frames at real-time pace → moonshine-tiny-en came back with 0% word error rate, and the same speech filtered to the 300 to 3,400 Hz telephone band with 17% (the first words were clipped).

Details, including what was not tested (no real phone network): docs/indic-telephony.md.

API compatibility

OpenAI. The request and response shapes follow the OpenAI API reference as typed in the official openai Python SDK (3.50.0), and are exercised with that SDK in the test-suite:

• Audio: plain and verbose transcription, word and segment timestamps, SSE transcription events, diarized_json (the gpt-4o-transcribe-diarize shape), translation, speech in every format, streamed speech, stream_format: "sse", the error envelope. whisper-1, gpt-4o-transcribe, tts-1 and friends map to your default models, and OpenAI voice names map onto the selected model's voices.
• Realtime (/v1/realtime, GA event names such as response.output_audio.delta): session.update, input_audio_buffer.* with server VAD, an optional end-of-turn model or manual commit, conversation.item.*, incremental input transcription events, response.create / response.cancel with audio, transcript and text deltas, function-call events, truncation, errors; 24 kHz PCM16 and G.711 μ-law/A-law. Tested with the SDK's realtime client and a raw WebSocket, and run from Pipecat 1.16 and LiveKit Agents 1.8 (their Realtime and their HTTP integrations). The WebRTC transport (/v1/realtime/calls) and ephemeral client secrets are implemented and tested with an aiortc peer, not with a browser.
• Chat completions with audio: input_audio content parts and modalities: ["text", "audio"], non-streaming and streaming.

bolo is a cascade (speech-to-text → your language model → text-to-speech), not a speech-to-speech model. Not implemented: SIP, MCP tools, image input. semantic_vad is an acoustic end-of-turn model (Smart Turn v3) on top of silence detection, optional and measured in the docs, not OpenAI's. See docs/voice-agents.md.

speaches. Point existing speaches clients at bolo and keep their model ids (Systran/faster-whisper-small, speaches-ai/Kokoro-82M-v1.0-ONNX, …) and environment variables (including CHAT_COMPLETION_BASE_URL). What matches, what differs and what is missing (speaker embeddings, its WebRTC route) is spelled out in docs/migration-from-speaches.md.

Wyoming / Home Assistant. bolo serve --wyoming exposes installed models on port 10300. Tested with the wyoming Python client; not tested against a Home Assistant instance. Setup notes: examples/home_assistant.md.

Full reference: docs/api.md and docs/openapi.json. Client examples: examples/.

Architecture

   OpenAI HTTP · speaches routes · native /api/v1 · WebSockets · Wyoming TCP · CLI
                                      │
        ┌─────────────────────────────┴──────────────────────────────┐
        │ core: transcribe · synthesize · realtime sessions · pulls   │
        ├──────────────┬───────────────┬───────────────┬──────────────┤
        │ audio I/O    │ text          │ observability │ model store  │
        │ G.711, soxr, │ sentences,    │ traces, p50/  │ resumable,   │
        │ codecs       │ normalisation │ p95, events   │ verified     │
        ├──────────────┴───────────────┴───────────────┴──────────────┤
        │ lifecycle: lazy load · probe · TTL · memory budget · LRU     │
        ├──────────────────────────────────────────────────────────────┤
        │ scheduler: one bounded queue per model · micro-batching ·    │
        │            worker threads · backpressure · cancellation      │
        ├───────────────────────┬──────────────────┬───────────────────┤
        │ sherpa-onnx           │ faster-whisper   │ optional plugins  │
        │ (ONNX Runtime)        │ (CTranslate2)    │                   │
        └───────────────────────┴──────────────────┴───────────────────┘

Engines are plugins behind four small interfaces (batch recognition, streaming recognition, synthesis, VAD), imported only when a model needs them. More in docs/architecture.md.

Performance

All numbers were measured with bolo's own load generator (bolo bench, in process) on 4 vCPUs of an Intel Xeon at 2.1 GHz, 15 GB RAM, no GPU, Linux x86-64, Python 3.12, sherpa-onnx 2.0.1, faster-whisper 1.2.1, 4 inference threads. The machine is a shared cloud VM that was running other workloads at the same time (load average 2 to 4 before bolo started), so expect better on an idle machine of the same size. Raw results: docs/measurements/.

Single request (best of three after warm-up; RTF = processing time / audio duration):

Model	RTF	Meaning
moonshine-tiny-en	0.033	30× faster than real time
parakeet-tdt-ctc-110m	0.030	33×
sense-voice-small	0.039	26×
parakeet-tdt-0.6b-v2	0.081	12×
faster-whisper-tiny / whisper-base / whisper-small	0.09 / 0.11 / 0.36	11× / 9× / 2.7×
whisper-large-v3-turbo (int8)	0.30	3.3×
piper-en-us-amy-low (TTS)	0.033 to 0.049, first audio after 140 to 190 ms	20 to 30×
kitten-nano-0.8-int8 (TTS)	0.27, first audio after about 1.0 s	3.7×
kokoro-v1.0-int8 (TTS)	0.88, first audio after 2.4 to 3.4 s	barely faster than real time

Kokoro is the default voice for quality, but on four shared cores it is only just real-time; the float32 export (kokoro-v1.0) measured faster than the int8 one, also for first audio. Voice agents, Wyoming and tts-1 therefore use piper-en-us-amy-low by default (default_realtime_tts), which starts speaking in about 150 ms and needs no optional component (phonemes from uccharan); see docs/models.md. Two streaming models were slower than real time here and are not usable live on this hardware: parakeet-unified-en-streaming-560ms (RTF 4.7) and, nearly, nemo-fastconformer-ctc-en-streaming-80ms (0.76).

Concurrent realtime streams (each client streams at 1×; p50 / p95 in ms):

Streams	zipformer-en-20m-streaming first partial	final	same, batching off: first partial	final	moonshine-tiny-en first partial	final	same, batching off: first partial	final
1	21 / 22	32 / 35	22 / 24	34 / 64	66 / 96	161 / 196	56 / 65	150 / 161
2	28 / 31	40 / 44	25 / 52	48 / 109	104 / 182	314 / 358	83 / 134	224 / 338
4	37 / 55	54 / 114	34 / 144	115 / 176	262 / 329	652 / 792	81 / 267	684 / 795
8	75 / 86	172 / 274	448 / 525	874 / 2,837	514 / 621	1,338 / 1,977	456 / 924	1,719 / 2,386

(The Zipformer rows used the 12-second sample, the Moonshine rows the 3-second one.)

Back-to-back requests (throughput in seconds of audio per second, 12-second clip):

Clients	zipformer-en-20m-streaming batched / unbatched	moonshine-tiny-en batched / unbatched	faster-whisper-tiny batched / unbatched
1	9.8 / 10.3	26.6 / 30.2	14.1 / 14.0
2	15.3 / 12.1	28.1 / 28.5	14.0 / 13.6
4	21.5 / 12.2	28.5 / 27.4	12.2 / 14.0
8	25.7 / 11.3	28.3 / 27.5	13.4 / 13.8

Thirty minutes of mixed load (moonshine and whisper-base transcription, Piper and Kitten synthesis; 1 to 8 clients; the machine was also running another project's jobs, load average 6 to 13): 1,494 requests, one refused with 429 by design, none failed. Memory reached 1.69 GB once the four models were loaded and ended at 1.80 GB, most of the rise when the client count went to 8; no unbounded growth. Under 8 clients the p95 was 1.3 s for moonshine and 3.3 s for Piper (whole responses). docs/measurements/soak-30min.json.

What batching buys on CPU, honestly:

• For the streaming transducer it is a clear win: with 8 live streams, batched decoding kept the median first partial at 75 ms against 448 ms unbatched, and whole-file throughput more than doubled (25.7× against 11.3× real time). A single stream uses little of the CPU per step, so stepping eight streams in one call shares the cores well.
• For Moonshine and Whisper it buys almost nothing in throughput (28× against 27×; 13× against 14×). A single decode already saturates four cores, so a batch of eight takes about eight times as long. It still helps fairness under load: at 8 streams Moonshine's p95 final was 1,977 ms batched against 2,386 ms unbatched, at the price of slower partials when only 2 to 4 streams are active.
• Batching is therefore on by default but never waits for a batch to fill (batch_wait_ms = 0): an idle server answers a lone request immediately.
• Four cores carry about 4 live Moonshine streams or 8 live Zipformer streams before latency climbs past half a second. Beyond that the queue bounds latency by refusing work (429), which is what you want from a server.

Speaker diarization (pyannote-3.0-eres2net: pyannote segmentation-3.0 + ERes2Net embeddings, ONNX): on a 21-second clip of two synthetic voices (Kokoro af_heart and am_michael) taking six turns, the diarization error rate was 5.2%, all of it speech missed at turn boundaries, with no speaker confusion (0% with a 250 ms collar), at RTF 0.11. Labelling live on the WebSocket (one embedding per utterance) got all six turns right and added about 90 ms per final. On four short public recordings the number of speakers was right for the three two-speaker clips and one too many (5 for 4) on the four-speaker one; with the default threshold of other embedding models it over-counts more. Pass num_speakers when you know it. That is a smoke test, not a benchmark: no overlapping speech, no noise.

A voice-agent turn (Realtime API; moonshine-tiny-en → SmolLM2-360M-Instruct on llama-cpp-python, 2 threads, on the same 4 cores → piper-en-us-amy-low), measured from the moment the end of speech is detected, median of three turns: transcript after 339 ms, first LLM token after 494 ms, first audio after 2.4 s. With Kokoro as the voice the first audio came after 3.8 s. Most of the wait is the tiny model writing its first sentence while it shares the CPU with recognition and synthesis; a language model on its own hardware changes these numbers completely, so read them as "the pipeline works", not as what to expect. The answers of a 360M model are not worth quoting. With qwen2.5:0.5b behind Ollama on the same busy machine (2 threads each): first token after 408 ms and first audio after 2.7 s with Piper (medians of three turns; docs/measurements/voice-agent-ollama.json). Over WebRTC with the same models a turn completed end to end and the reply heard on the Opus track transcribed back correctly (webrtc-real-models.json). None of these numbers describe a deployment with a real language model on its own hardware.

Text to speech under load (piper-en-us-amy-low, a 200-character paragraph): first audio after 138 ms with one client, 402 ms with two, 826 ms with four; throughput stays near 27× real time. Synthesis is not batched; requests interleave sentence by sentence.

How it compares

bolo is not the only way to serve speech, and often not the best one. Fairly:

	What it is	Choose it over bolo when	Choose bolo when
speaches	OpenAI-compatible STT/TTS server (faster-whisper, Kokoro, Piper), MIT	you need its speaker-embedding endpoint, a WebRTC transport that has been used with browsers (bolo's has only been tested with a WebRTC library), or you prefer pyannote's PyTorch diarization pipeline	you want its API (including the Realtime API and chat-completions audio) with micro-batching, backpressure, more model families, telephony audio and latency tooling, without PyTorch. Its last commit was in April 2026 and the question "Actively supported?" has been open since July 2026
LocalAI	a large, very active platform for LLMs, images, audio and more, with backends pulled as separate images, MIT	you want one gateway that also runs the language models, images and embeddings, GPU acceleration, or its much bigger ecosystem	you want a single small Python process for the speech side, with realtime streaming and per-request latency traces, in front of whatever LLM you already run
vLLM-Omni	vLLM extended to omni-modal and speech models	you have GPUs and want true speech-to-speech models, expressive LLM-based TTS or full-duplex omni models at scale	you are on CPUs or small machines and want a cascade of classic ASR/TTS models around any text LLM, with low latency
WhisperLiveKit	realtime speech-to-text with state-of-the-art streaming policies (SimulStreaming/AlignAtt), streaming diarization and translation, Apache-2.0	realtime transcription quality is the whole job: its streaming policies for Whisper are more advanced than bolo's LocalAgreement, and its live diarization handles overlapping speakers, which bolo's per-utterance labelling does not	you also need text-to-speech, batch transcription, an OpenAI-compatible API (including Realtime) and model management in the same server, without PyTorch
Kokoro-FastAPI	a polished OpenAI-compatible server for Kokoro TTS, with GPU images	you only need Kokoro, want GPU speed, word-level captions or voice mixing	you want Kokoro and other voices and speech-to-text behind one API
sherpa-onnx servers	the WebSocket servers that ship with the engine bolo builds on	you want the thinnest possible layer, or C++/mobile/embedded targets	you want HTTP APIs clients already speak, a catalog, scheduling across models and observability

bolo stands on these projects' shoulders: most of what it runs comes from the sherpa-onnx and faster-whisper teams and from the people who trained and exported the models.

What is not tested

Stated plainly, so you do not find out the hard way:

• GPU. No GPU was available. device = "cuda" is passed to faster-whisper and sherpa-onnx, and there is a Dockerfile.gpu, but none of it has been run.
• Apple Silicon and macOS, Linux arm64, Windows. Not run. The dependencies publish wheels for macOS and arm64, so it should work there; Windows is unknown.
• Python 3.10 and 3.11. The package targets 3.10 to 3.13; the test-suite was run on 3.12 and 3.13 only.
• Real telephony. The G.711 paths are tested end to end with synthesised speech, not on a phone network.
• Home Assistant and browsers. The Wyoming flows Home Assistant uses were run with the official wyoming client against a real server with real models, not with Home Assistant itself. The WebRTC transport of the Realtime API is tested with the aiortc library (also with real models), not with Chrome, Firefox or Safari, and not across NAT or a TURN relay. Pipecat and LiveKit Agents were run against bolo, but without a transport in front (a wav file played the caller).
• The GPU image, the systemd unit and the Kubernetes manifest. The CPU image was built and run here (health check, synthesis, transcription) and CI builds and smoke-tests it on every push; the compose file was run. The systemd unit passes systemd-analyze verify but was not run under systemd, the Kubernetes manifest was not applied to a cluster, and the GPU image was not built.
• Most of the catalog. 117 of 225 models are marked untested: their files were checked on the Hub and a sibling of the same architecture was run, but that model was not.
• Voice agents with a capable language model. The conversation path was run end to end with models of 360M and 500M parameters on a shared CPU, through llama-cpp-python's server, Ollama and llama.cpp's llama-server (see Performance); the answers of such models are poor and the turn latency is not representative of a real deployment. Function calling was exercised with a scripted fake model, not a real one. vLLM and hosted APIs were not tried.
• Diarization accuracy. Measured on one clip made of two synthetic voices taking turns, plus speaker counts on 18 public recordings (see docs/api.md, "How many speakers"), plus overlap handling on three synthetic two-voice mixes. No diarization error rate on real recordings; overlap and noisy rooms were not evaluated on real recordings. Known-speaker naming was tested with synthetic voices only.
• Converted voices and the release workflow. The "Voices" workflow published the 22 SYSPIN and 18 MMS-TTS voices (release assets and Hugging Face); eight MMS voices were run here from local conversions, the published MMS files were not re-run. The "Release" workflow has not been run.
• Accuracy at scale. Word error rates here come from a handful of clips. They show that the pipelines work, not how good the models are; for that, see the model publishers' evaluations.
• Memory growth over days, hostile input. Not done; a 30-minute mixed-load soak was (see Performance).
• Engines moonshine (useful-moonshine-onnx), whisper.cpp and kokoro-onnx have plugins that were written against their documentation and not run.

Licences

bolo itself is Apache-2.0. It contains no third-party source code; NOTICE lists what it works with.

• Engines: sherpa-onnx (Apache-2.0), ONNX Runtime (MIT), faster-whisper and CTranslate2 (MIT). The optional kokoro-onnx plugin is MIT but pulls in the GPL phonemizer package, so it is not part of [all].
• uccharan (Apache-2.0) turns text into phonemes for English and Hindi voices; it is a dependency of bolo-speech[all], not part of bolo's code.
• eSpeak NG (GPL-3.0-or-later) is a separate, optional component. It turns text into phonemes for Piper-format voices, KittenTTS and Kokoro in the languages uccharan does not cover. bolo's wheel does not contain it, the all extra does not install it, and the container image does not include it unless you build with --build-arg WITH_ESPEAK=1. It arrives only when you ask: the console's install button (which asks you to accept the licence), bolo components install espeak-ng --accept-license, pip install 'bolo-speech[espeak]' (the espeakng-loader package: a prebuilt library plus data), or a system package. bolo's own code stays Apache-2.0 either way: Apache-2.0 code may be combined with GPLv3 code, and bolo only calls the component's public interface. But if you redistribute a bundle that contains eSpeak NG (an image built with WITH_ESPEAK=1, an appliance, an installer) you must comply with the GPL for that component. Without it you still have every English and Hindi voice (through uccharan), Kokoro Mandarin, MeloTTS, MMS-TTS (non-commercial), the SYSPIN voices and the Coqui Bengali voices. sherpa-onnx removed espeak-ng from its own binaries in version 2.0, and the GPL piper-tts runtime is not used at all. More in docs/models.md.
• Models are downloaded from their publishers under their own licences, which the catalog records per model and the console shows. Most are MIT, Apache-2.0 or CC-BY-4.0. Some are non-commercial (MMS-TTS, non-English Moonshine, several Piper voices, flagged ⚠) or have custom terms (SenseVoice, Paraformer). A few are gated (marked gated); bolo never downloads a gated model without your own token.

Roadmap

• A Bhojpuri recogniser better than the Hindi fallback (the Whisper fine-tune we converted was not).
• The PyTorch engine plugin (Indic Parler-TTS, IndicF5, Svara, SraVaani).
• WebRTC tested with browsers and behind NAT; SIP; MCP tools.
• Overlap-aware online (live) diarization; a diarization error rate measured on real recordings.
• Word timestamps for Whisper through sherpa-onnx (exports with attention outputs) and for Moonshine.
• GPU validation, Apple Silicon validation, and published container images.
• Models trained for 8 kHz telephone speech.
• An attention-based streaming policy for Whisper as an alternative to LocalAgreement.
• Authentication beyond a single shared key.

Contributing

Issues and pull requests are welcome, especially measurements on hardware we do not have and new catalog entries: CONTRIBUTING.md explains the setup and how to add and validate a model. Please read the Code of Conduct; report security problems as described in SECURITY.md.