pipecat-local-tts-server 0.1.0

Standalone local WebSocket text-to-speech (TTS) server, client, and pluggable mlx-audio backends for the Pipecat ecosystem

pipecat-local-tts-server

Standalone, local WebSocket text-to-speech (TTS) server — text in, audio out — for the Pipecat ecosystem. It mirrors the sibling pipecat-local-stt-server: same websocket transport, an OpenAI-Realtime-inspired protocol subset, a pluggable backend abstraction, and lazy-imported per-model backends behind optional extras so a client-only consumer never pulls the heavy TTS runtime.

Distributed as pipecat-local-tts-server; the import name is tts_server (every import tts_server / python -m tts_server invocation works). Kokoro-first (mlx-audio, Apple Silicon); more backends land later.

The server owes its client exactly two things (the contracts everything else is built around):

1. An exact, stable advertised rate. server.hello.audio.rate is the true model rate (Kokoro = 24000 Hz). Every audio frame is int16-LE mono PCM at exactly that rate, with no per-utterance drift. The client resamples model-rate → device-rate off this single value.
2. A steady in-response stream. Once a response starts, audio arrives continuously (each model segment is emitted as it completes) so the client's playback buffer never starves.

Install

The base package is lean — websockets only. Backends live behind extras.

From PyPI (consumers):

# client-only lean base (websockets) — for a bot that just talks to a server
uv add pipecat-local-tts-server            # or: pip install pipecat-local-tts-server

# Kokoro backend (Apple Silicon; pulls mlx-audio==0.4.4 + misaki[en])
uv add "pipecat-local-tts-server[kokoro]"  # or: pip install "pipecat-local-tts-server[kokoro]"

From source (development):

# client-only (lean base: websockets) — for a bot that just talks to a server
uv sync --extra client

# Kokoro backend (Apple Silicon; pulls mlx-audio==0.4.4 + misaki[en], which
# drags in spacy/num2words/torch — heavy by design, kept out of lean base)
uv sync --extra kokoro

# the reference Pipecat adapter example (pulls the Pipecat framework)
uv sync --extra examples

Run every command through uv run (or activate the venv once per shell). A bare python -m tts_server … uses the system interpreter and fails with ModuleNotFoundError: No module named 'websockets'.

Running the server

# Kokoro over a Unix domain socket (recommended for local use)
uv run python -m tts_server serve --backend kokoro \
    --socket-path ~/Library/Caches/pipecat-tts/tts.sock

# pick a specific model (any compatible mlx-community Kokoro repo id)
uv run python -m tts_server serve --backend kokoro \
    --model mlx-community/Kokoro-82M-bf16 \
    --socket-path ~/Library/Caches/pipecat-tts/tts.sock

# loopback TCP instead of a socket (choose any free port)
uv run python -m tts_server serve --backend kokoro --host 127.0.0.1 --port 8765

The server logs the resolved backend + model at startup, before the (potentially slow) model load, so you can see what is being loaded. The rate is read from the loaded model, so model load completes before the first server.hello is sent.

--log-level (default INFO) sets the server's logging verbosity (any standard Python level name, e.g. DEBUG/WARNING).

On startup over a Unix socket, the server auto-clears a stale socket left by a previous crash (SIGKILL / power loss), so the documented restart works without manual rm. It refuses to start — surfacing a diagnostic instead of clobbering — if a non-socket file exists at the path, or if a live server is already listening there (the socket is genuinely in use).

Environment variables

Endpoint precedence (server and client both): URI > socket > host+port. The TTS_WS_* vars mirror STT_WS_*.

Variable	Side	Purpose
TTS_WS_URI	client	Full ws:///wss:// URI; highest endpoint precedence.
TTS_WS_SOCKET	client	Unix-socket path (used when no URI).
TTS_WS_HOST	client	TCP host (used when no URI/socket).
TTS_WS_PORT	client	TCP port (paired with host).
TTS_WS_TOKEN	client	Bearer token the client/probe sends. Never falls back to the server var.
TTS_WS_DEFAULT_SOCKET	client	Explicit fallback socket for status when nothing else is set.
PIPECAT_TTS_AUTH_TOKEN	server	Bearer token the server requires (optional auth).
PIPECAT_TTS_KOKORO_EXTRA_LANGS	server	Comma-separated ISO codes (e.g. ja,zh) to advertise after installing their extra G2P package. See Kokoro language support.

Auth notes: the server reads PIPECAT_TTS_AUTH_TOKEN; the client/probe reads TTS_WS_TOKEN (the two are deliberately separate so a probe can never mask a client 401 or leak the server secret to a remote host). A plaintext --auth-token flag is intentionally unsupported (ps exposure) — use --auth-token-file. A token-less server bound to a non-loopback TCP address logs a cleartext-remote warning; a Unix socket does not. Sending a bearer over cleartext ws:// to a remote host also warns client-side — use wss:// or a Unix socket.

Checking server health

uv run python -m tts_server status \
    --socket-path ~/Library/Caches/pipecat-tts/tts.sock

status connects, performs the handshake, requests a server.status snapshot, and prints the backend, model, audio format/rate, capabilities (streaming / binary_audio / voice_count), session id, synthesis queue depth, the voice list, buffered chars, uptime, and pid. It exits non-zero if no server is reachable.

status resolves its endpoint with the same URI > socket > host+port precedence as the client and additionally accepts --uri ws://…/wss://… (the serve path has no --uri, since it builds its listener from socket-path/host+port). Two probe-only flags: --timeout (overall probe budget in seconds, default 3.0) and --json (emit the raw hello+status JSON instead of the text summary).

For day-to-day operation on macOS the justfile carries read-only operator recipes mirroring the sibling stt server: just tts-list lists every pipecat.tts-server* launchd agent with state, pid, and live backend, and just tts-status runs the wire status probe against the canonical socket (override with just tts-status socket=…).

Protocol

The full wire contract is in docs/protocol.md. In brief: every message is a JSON text frame with a type field. The client drives the session — session.update → input_text.append* → input_text.commit — and the server streams response.audio.delta frames (base64 pcm16, seq from 0, no gaps) ending in response.audio.done. response.cancel is barge-in. Audio is base64-in-JSON for v1 (binary_audio: false); binary frames are a later optimization.

The client segments the text; the commit is the unit of work. Using capabilities.streaming and capabilities.ideal_words, the client splits long text into commits, rounding ideal_words up to the next sentence boundary — never splitting mid-sentence (a half-sentence commit makes the model apply sentence-final prosody mid-phrase). max_text_chars is the hard server cap.

Kokoro capabilities (as shipped)

Built per-backend (server.hello.capabilities). Verified against mlx-community/Kokoro-82M-bf16 (mlx-audio 0.4.4):

Field	Value	Note
rate	24000	from server.hello.audio.rate, not capabilities
streaming	false	no sub-segment streaming; segments still stream per \n+
binary_audio	false	base64-in-JSON for v1
text_formats	["plain"]	ssml/ipa not supported
languages	["en","es","fr","hi","it","pt"]	from voice prefixes, minus languages needing extra G2P — ja/zh are off by default (opt-in, see below)
voice_count	54	full list via status
extras	["speed"]	Kokoro's only effective generate() kwarg
ideal_words	40	soft target; client rounds up to a sentence boundary
max_text_chars	2000	hard server cap

Kokoro language support (advertised = synthesizable)

The advertised languages list reflects what this deployment can actually synthesize, not just what voices the model ships. The default kokoro extra pins misaki[en] only; verified live against mlx-community/Kokoro-82M-bf16:

• en uses misaki[en]; es/fr/it/pt/hi route through the espeak-ng G2P bundled with misaki[en] — all synthesize fine and are advertised. (hi's first call loads its G2P lazily and can exceed a 60 s client timeout.)
• ja and zh need an extra G2P package — misaki[ja] (pyopenjtalk) and misaki[zh] (ordered_set) respectively, which the kokoro extra does not install. Because synthesis would fail at runtime (response.failed, code=backend_error) without them, they are not advertised by default and a request for them is rejected up front with invalid_config (before a synthesis slot is consumed) rather than failing mid-response.

Enabling ja / zh is a two-step, build-time decision:

1. Install the G2P package(s): uv pip install "misaki[ja]" "misaki[zh]"
2. Opt the language(s) back into the advertised set: export PIPECAT_TTS_KOKORO_EXTRA_LANGS=ja,zh

The server logs its advertised language set at startup (including a reminder of any languages left disabled). The opt-in only re-adds a language the model already ships voices for; it cannot advertise one the model lacks. If you set the env var without installing the package, that language is advertised again and will fail at synthesis — install first.

See tests/smoke/ for the live end-to-end smoke scripts that verify this (just smoke-tone / just smoke-kokoro / just smoke-multilingual).

Kokoro cancellation caveat

Kokoro yields one segment per \n+ boundary and the cancel flag is only checked at a segment boundary. The client-visible cancel is prompt regardless: a response.cancel is acknowledged with response.cancelled in ~1 ms (measured on Apple Silicon), and no audio follows it. What runs to the segment boundary is the backend worker / Metal lock: a long single-segment commit keeps the lock until its generate() reaches the yield (≈ the full single-segment synthesis time — a few seconds for a ~1700-char segment, bounded by drain_timeout_seconds), so the next commit can't start synthesizing until then. To free the lock sooner for back-to-back commits, clients should chunk at sentence/newline boundaries for Kokoro. The server's hard guarantee is "no more audio after response.cancelled". (See the dev plan's Phase 2 measured results for the full re-measurement; the earlier "≈ tens of seconds" figure was a bridge-bug artifact.)

Examples

• examples/reference_client.py — a lightweight stdlib + websockets oracle (no tts_server install, no Pipecat). It speaks the wire protocol directly and writes the reassembled audio to a WAV. Useful for manual end-to-end smoke checks once a server is running.
• examples/pipecat_tts_service.py — a reference Pipecat-framework TTSService adapter (LocalTTSService) that wraps the async tts_server.client.TTSClient so a bot pipeline can speak through a running server. Streams TTSAudioRawFrames at the server-advertised rate and sends response.cancel on interruption. Requires the Pipecat framework (uv sync --extra examples, which pins pipecat-ai==1.4.0).

Layout

• tts_server/ — protocol, backend abstraction, server, async client, CLI.
• tts_server/backends/ — lazy-imported per-model backends (Kokoro first).
• examples/ — the stdlib oracle and the Pipecat service adapter.
• justfile — macOS operator recipes (tts-list, tts-status).
• docs/protocol.md — the wire protocol specification.
• docs/dev_plans/ — development plans.