pipecat-local-stt-server

Standalone local WebSocket transcription (STT) server, minimal Python client, and pluggable ASR backends for the Pipecat ecosystem.

Distributed as pipecat-local-stt-server (PyPI); the import name is stt_server (every import stt_server / python -m stt_server invocation keeps working). Extracted, history-preserving, from the private koda-pipecat monorepo. BSD-2-Clause.

V1 scope

• one WebSocket connection = one transcription session
• pinned wire format: PCM16LE, mono, 16000 Hz
• binary WebSocket audio frames are the default transport; JSON input_audio_buffer.append with base64 remains as a compatibility mode
• turn_detection: null required; server VAD is not implemented in V1
• commit-oriented decode (single large delta + completed)
• local-only trust model:
  • serve has no default endpoint: pass --socket-path (Unix domain socket, recommended: peer-uid check, no port) or --host/--port. The just/LaunchAgent agents default to loopback TCP (no peer-uid check), with a socket opt-in
  • optional loopback TCP (127.0.0.1) with optional bearer token
  • browser Origin headers rejected
• backend interface (TranscriptionBackend) so MLX can be swapped later
• EchoBackend reference implementation for tests and smoke-checks
• MLXWhisperBackend shipped in stt_server/backends/mlx_whisper.py (requires the mlx extra)
• ParakeetBackend shipped in stt_server/backends/parakeet.py (requires the parakeet extra; default model mlx-community/parakeet-tdt-0.6b-v3). Parakeet decodes from a temp WAV; that WAV holds raw utterance audio (PII) and is written to a per-process private 0o700 directory (created at backend start, removed on close()), never the world-listable system temp dir.
• NemotronBackend shipped in stt_server/backends/nemotron.py (requires the nemotron extra — uv sync --extra nemotron; default model mlx-community/nemotron-3.5-asr-streaming-0.6b). Nemotron decodes from a temp WAV; that WAV holds raw utterance audio (PII) and is written to a per-process private 0o700 directory (created at backend start, removed on close()), never the world-listable system temp dir.
• KyutaiBackend shipped in stt_server/backends/kyutai.py (requires the kyutai extra — uv sync --extra kyutai; default model kyutai/stt-1b-en_fr-mlx, English and French, CC-BY-4.0). Decodes in memory through moshi_mlx. The extra pins an older MLX stack and is declared conflicting with the other MLX extras, so install it on its own. It needs Python 3.12 (moshi_mlx pins sentencepiece==0.2.0, which has no 3.13 wheel; elsewhere it builds from source and needs cmake). just stt-install kyutai handles this: it syncs the extra into a separate .venv-kyutai and installs a LaunchAgent on 127.0.0.1:9665. To run it by hand, use serve --backend kyutai, e.g. under --supervised.

Running the server

Prerequisite: install deps with uv sync first, and run every command through uv run (or source .venv/bin/activate once per shell). Bare python -m stt_server … uses the system interpreter and fails with ModuleNotFoundError: No module named 'websockets'.

# UDS (recommended for local use — no port, no bearer token)
uv run --no-sync python -m stt_server --socket-path ~/Library/Caches/pipecat-stt/stt.sock --backend echo

# MLX Whisper over UDS
uv sync --extra mlx
uv run --no-sync python -m stt_server --socket-path ~/Library/Caches/pipecat-stt/stt.sock --backend mlx

# Loopback TCP — minimal form. Pick any free port; there is no default port,
# so --host and --port are both required for TCP. On loopback an auth token is
# optional (the server logs an info line and serves anyway — fine for local
# experiments). The listener comes up on 127.0.0.1:<port>.
uv run --no-sync python -m stt_server --host 127.0.0.1 --port 9900 --backend echo

# Loopback TCP with auth (recommended for anything non-experimental — use
# --auth-token-file or PIPECAT_STT_AUTH_TOKEN env; legacy KODA_STT_AUTH_TOKEN
# still honoured; there is no --auth-token flag, since argv is visible via `ps`.
# A token file must hold one token of visible ASCII; an empty or malformed one
# makes the server exit rather than run unauthenticated. The file itself must be
# an absolute path to a regular file you own, mode 600 (no group/other bits, no
# ACL), in directories only you or root can write: otherwise serve exits 1 and
# names the file and the fix, e.g. `chmod 600 <file>` or move it out of a shared
# directory; see docs/operations.md#token-file-replaces-env-tokens).
# The LaunchAgent installer and renderer refuse both env tokens: use
# PIPECAT_STT_AUTH_TOKEN_FILE there.
uv run --no-sync python -m stt_server --host 127.0.0.1 --port 9565 --auth-token-file /path/to/token

Ports: there is no baked-in default port — you choose it (--port 9900). Use --port 0 to let the OS pick a free one. V1 only permits loopback binds (127.0.0.1/::1/localhost); a non-loopback --host is rejected.

Backends need their own install. Every --backend above except echo requires a uv sync extra first — mlx/parakeet/nemotron (e.g. uv sync --extra mlx). Without it the server exits with an actionable stt_server: the '<extra>' extra is not installed … run: uv sync --extra <extra> --inexact message. See Choosing a backend and model for each backend's install command and --model defaults.

The CLI accepts both python -m stt_server <flags> (the legacy flat form, which implicitly routes to serve) and python -m stt_server serve <flags> / python -m stt_server status <flags> — see Checking server health for the status subcommand.

For persistent always-on operation on macOS, install the server as a LaunchAgent via just stt-install <backend> or scripts/install_stt_agent.sh (install|start|stop|restart|status|logs). Agents bind loopback TCP on a per-backend port by default (whisper 9565, kyutai 9665, parakeet 9765, nemotron 9865); set PIPECAT_STT_SOCKET for a Unix-socket agent and point clients at a TCP agent with STT_WS_HOST/STT_WS_PORT. Pair it with python -m stt_server status for a wire-level health probe. A consumer may wrap these behind its own CLI — see Reference consumer (Koda) for one such integration.

Managing agents with just

install_stt_agent.sh manages exactly one agent per invocation, so once you run two or three ASRs side by side there is no single command to see them or stop the idle ones. The repo-root justfile is a thin operator layer over launchctl (macOS only) that fills that gap. Run recipes from the repo root:

just                       # list available recipes
just stt-list              # every pipecat.stt-server* agent: state, pid, live backend
just stt-status            # no argument: same as stt-list
just stt-status nemotron   # wire health probe for one backend (installed plist's endpoint)
                           # stt-list and stt-status <backend> also warn on stderr if a TCP agent's port is held
                           # by another process (advisory; see docs/operations.md, "Port squat detection")
just stt-status ~/Library/Caches/pipecat-stt/stt.sock   # a path is probed as --socket-path
just stt-probe --host 127.0.0.1 --port 9565   # probe any endpoint (supervised or socket)
just stt-disable whisper   # stop until next login (keeps the plist)
just stt-enable whisper    # re-load it from the existing plist
just stt-install parakeet  # delegates to install_stt_agent.sh (+ ensures the extra)
just stt-uninstall parakeet
just smoke-peercred        # local UDS peer-cred smoke (cross-uid leg needs a 2nd uid)

Smoke tests

just smoke             # supervised (echo), peer-cred, token-file: one PASS/FAIL/SKIPPED table
just smoke --agents    # also decode a clip through each installed agent its own launchd job serves (read-only)
just smoke --launchd   # also run the launchd harness on the spare label (macOS only, opt-in)

Exit status is 1 if any step FAILED; SKIPPED never fails. --launchd needs both PIPECAT_STT_LABEL and KODA_STT_LABEL set to pipecat.stt-server.smoke, which the runner sets itself; it never runs in CI. Details and the individual scripts: docs/operations.md.

The checks were verified by hand at 0.5.0 (token-file on a spare port without launchd, a decode through each agent). New in 0.5.1: the launchd harness (scripts/smoke_launchd.py) ran live three times (2026-10-28) against the spare label and port; the last, on the closing-pass code, had all five rows PASS. Then the just smoke --agents --launchd wiring ran live twice on 2026-10-29. On 8a87046, launchd PASSed and every agents row FAILed (the installed agents' jobs were not loaded; their servers run under the Pipecat Service Toolbar app). After that became SKIPPED, on e01d7df: supervised, token-file and launchd PASS, peercred SKIPPED (no second uid), every agents row SKIPPED. Later fixes are covered by stubbed tests. Still tested only with stubs: the --agents decode through a real launchd agent (it and the harness's collision and rollback bodies run end to end in the tests, against stub launchctl/lsof and a fake agent). Treat the decode as unverified until a live just smoke --agents run with a loaded agent.

<backend> is one of whisper / parakeet / nemotron / kyutai, mapped to the labels and ports in the per-backend table (the justfile map is a checked mirror of that table — a test fails CI on drift). For a TCP agent stt-install/stt-uninstall always use that canonical host and port (the recipes strip an ambient PIPECAT_STT_HOST/PORT and the installer applies its defaults); for a custom one run the installer directly with a distinct label and backend, or it replaces the default whisper agent (PIPECAT_STT_LABEL=pipecat.stt-server.alt PIPECAT_STT_BACKEND=mlx PIPECAT_STT_PORT=9570 scripts/install_stt_agent.sh install; see Operations). stt-install refuses PIPECAT_STT_AUTH_TOKEN; use PIPECAT_STT_AUTH_TOKEN_FILE (see Operations).

stt-install / stt-enable also ensure the backend's optional Python extra is installed so a freshly installed agent doesn't crash-loop on a missing import (uv sync --extra <backend> --inexact, additive; kyutai instead syncs into its own .venv-kyutai with --frozen --python 3.12 and no --inexact); set PIPECAT_STT_SKIP_DEP_SYNC=1 to manage the extras yourself. The check runs the interpreter the agent runs: stt-install checks PIPECAT_STT_PYTHON (else the venv python) and hands that same interpreter to the installer, and stt-enable checks the one the installed plist names (ProgramArguments[0]), whatever your environment says.

stt-status <backend> reads the installed plist's endpoint (socket or TCP, plus --auth-token-file) and only falls back to the backend's canonical host and port when no plist exists, so it also works on a not-yet-migrated socket agent. A path (or an existing socket file in the repo directory) is probed as --socket-path. stt-list reads each agent's live endpoint from its plist: a TCP agent shows an endpoint: host:port line, a socket agent a socket: line in the same ~-form a consumer's config uses (e.g. onoats' config.toml [stt] ws_socket; for a TCP agent use ws_host/ws_port). Note whisper's socket is stt.sock, not whisper.sock.

stt-disable vs stt-uninstall. The LaunchAgent plist sets RunAtLoad + KeepAlive, so install_stt_agent.sh stop (a plain SIGTERM) is respawned immediately. stt-disable instead does launchctl bootout, which takes the agent down until the next login — the plist stays on disk and launchd reloads it when you log back in. It returns once launchd no longer lists the agent (up to 30 s), so a just stt-enable straight after it does not hit Bootstrap failed: 5. stt-uninstall removes the plist, so the agent stays gone. For cross-login suppression without removing the plist, use launchctl disable gui/$(id -u)/<label>.

stt-list sweeps the pipecat.stt-server* label prefix, so a custom-labelled agent still shows up, probed at the endpoint in its own plist. Legacy koda.stt-server* agents are not covered — check those manually during migration with launchctl list | grep koda.

Choosing a backend and model

The server picks the ASR, pinned at launch — the client never selects it. A client (including the Pipecat service in the integration guide) only points at an endpoint and transcribes against whatever backend that server was started with. To run Whisper vs Parakeet you start a different server (or a second one on its own endpoint — see "Multi-backend operation"); no client code changes.

# Whisper (MLX) — default model mlx-community/whisper-large-v3-turbo
uv sync --extra mlx
uv run --no-sync python -m stt_server serve --backend mlx \
    --socket-path ~/Library/Caches/pipecat-stt/stt.sock
# ...or as an always-on loopback TCP agent on 127.0.0.1:9565:
just stt-install whisper    # parakeet 9765, nemotron 9865, kyutai 9665 likewise

# Parakeet — default model mlx-community/parakeet-tdt-0.6b-v3
uv sync --extra parakeet
uv run --no-sync python -m stt_server serve --backend parakeet \
    --socket-path ~/Library/Caches/pipecat-stt/parakeet.sock

# Nemotron 3.5 — default model mlx-community/nemotron-3.5-asr-streaming-0.6b
uv sync --extra nemotron
uv run --no-sync python -m stt_server serve --backend nemotron \
    --socket-path ~/Library/Caches/pipecat-stt/nemotron.sock

# Pick a specific model with --model (any compatible mlx-community HF repo id)
uv run --no-sync python -m stt_server serve --backend mlx \
    --model mlx-community/whisper-small --socket-path .../stt.sock

--model is passed through verbatim; an unset value uses the backend-aware default (the Whisper repo for mlx/echo, parakeet-tdt-0.6b-v3 for parakeet, nemotron-3.5-asr-streaming-0.6b for nemotron, kyutai/stt-1b-en_fr-mlx for kyutai). Pointing a backend at a mismatched repo fails fast at decode.

Each of those four default models is pinned to a Hugging Face commit, so a push to the repo's main cannot change the weights under a running install. The pins live in one table, PINNED_REVISIONS in stt_server/backends/__init__.py, and both serve and prefetch use them. Any other --model value (another repo id or a local path) is not pinned and follows that repo's main. To move a pin, change its commit SHA in that table, re-run the live decode tests against it, and run prefetch so the new snapshot is cached before the agent restarts. On a cancelled stream (barge-in) only Kyutai stops its decode early, at the next 80 ms frame, and skips a decode still queued behind another; the other backends finish a decode once it has started.

Each Unix socket gets a sibling <socket>.lock file (e.g. parakeet.sock.lock, mode 0600). The server holds an flock on it for its lifetime, so a second server for the same socket fails at once with EADDRINUSE. The kernel releases it when the process exits. Neither the server nor the installer deletes the file, so every server locks the same one; leaving it is safe. On a transport change or uninstall the installer removes only the old socket, once the old job has unloaded and no process holds the lock. See Operations: single-server lock file.

Nemotron's runtime dependency is mlx-audio>=0.4.4 — the first PyPI release carrying Nemotron STT support (Blaizzy/mlx-audio#774, merged 2026-06-05). Before 0.4.4 this backend had to git-pin mlx-audio in a [dependency-groups] dev group (a direct-URL dependency PyPI rejects in a published extra); since 0.3.2 it is a clean, PyPI-installable nemotron extra like mlx and parakeet.

Common MLX Whisper models (smaller = faster + lower RAM, larger = more accurate). These are mlx-community Hugging Face repos; the first launch downloads and caches the weights.

Any mlx-community Whisper repo (e.g. …-large-v3-turbo-q4 quantised variants, or …-large-v3-turbo language-specialised forks) works as a --model value — the table lists common starting points, not an allowlist. Verify which model a running server actually loaded with python -m stt_server status (it prints backend: <name> (model: <repo>)).

Prefetching models

mlx, parakeet, nemotron and kyutai download their checkpoint on the first decode, so a cold server answers its first request only after a multi-GB download. prefetch does the download up front, and --check answers "is it cached?" without touching the network:

uv run --no-sync python -m stt_server prefetch --backend parakeet          # download
uv run --no-sync python -m stt_server prefetch --backend parakeet --check  # offline probe
uv run --no-sync python -m stt_server prefetch --backend mlx --model mlx-community/whisper-small --json

It fetches exactly what the backend loads, into the standard Hugging Face cache (HF_HOME / HF_HUB_CACHE). --model may also name a local model directory: a value starting with /, ./, ../ or ~ (or an existing relative path) is a local path, anything else a Hub repo id. A local directory is never downloaded; --check reports it cached only if it holds every file the backend requires, and a fetch fails with ModelInvalidError if it does not. kyutai loads by Hub repo id only: prefetch rejects a local path for it (ConfigError), while serve --backend kyutai --model <path> passes the value to the Hub as a repo id and fails at model load.

uv run --no-sync python -m stt_server prefetch --backend parakeet --model ~/models/parakeet --check

--check exits 0 whenever the probe ran; the answer is the cached field. With --json each stdout line is one event:

progress counts only what this run must download, as in tts_server prefetch: total_bytes / files_total exclude files already in the cache, downloaded_bytes includes partial blobs from an interrupted run, and the last progress before done has downloaded_bytes == total_bytes and files_done == files_total (all zero when everything was cached).

The event schema matches tts_server prefetch in pipecat-local-tts-server, so one supervisor can drive both. The error codes are per repo; see Error codes below.

Supervised mode

serve --supervised is for a parent process that manages the server, such as a menu bar app. Logging on stderr is unchanged. In addition:

• stdout carries only JSON lines. Once the listener is bound and the model is warm (supervised mode decodes 1 s of silence first), the server prints one {"event":"ready","backend","model","host","port","socket_path","rate","pid"} line. With --port 0, port is the real port.
• A fatal error prints {"event":"error","code","message","exc_type"} and exits 1. See Error codes for the code values.
• stdin EOF stops the server exactly like SIGTERM. The parent keeps a pipe open, so the server exits when the parent closes it or dies. During startup the process exits at once; interrupted downloads resume next time. (Embedding stt_server.server.serve() in your own process? It cancels startup and shuts down normally instead, unless you pass hard_exit_on_stop=True. A TranscriptionServer is single-use once shutdown() was called or a failed start left work running (stopping); a failed start that returned to new may be retried. Await wait_stopped() before reusing the socket: the lock can outlive shutdown(). Which of the server's other signals to read, and adopt() for your own tasks, is documented once in the TranscriptionServer class docstring.)

( sleep 5 ) | uv run --no-sync python -m stt_server serve --backend echo \
    --host 127.0.0.1 --port 0 --supervised

Skip any stdout line that does not parse as JSON: native code writing straight to file descriptor 1 can bypass the redirect.

Error codes (per repo)

The error event's code (from serve --supervised and prefetch --json) is a stable string. The lifecycle channel has the same shape as tts_server's, but each repo has its own code list:

pipecat-local-tts-server has no ModelInvalidError and adds SocketPathConflict; check its README for its list. A supervisor driving both should treat any code it does not recognise as BackendStartError.

Adding a backend

A backend is registered in one place and picked up by --backend, the --model default and prefetch. Checklist:

1. Module stt_server/backends/<name>.py: a class implementing TranscriptionBackend (backend_name, model, start(), open_stream(), close()) and a module-level prefetch_specs(model) -> list[SnapshotSpec] (stt_server.snapshot). Each SnapshotSpec.required entry is a glob (relative to the snapshot root) that must match a file for prefetch --check to report it cached; a tuple entry is a set of alternatives, any one of which suffices (see the field docstring in stt_server/snapshot.py). Import the optional extra only inside start() or the lazy model load, never at module load, and re-raise a missing extra as ModuleNotFoundError with the uv sync --extra <name> --inexact hint. Reuse DecodeGuardMixin (backends/_thread_util.py) for MLX/Metal decodes.
2. Error origin: wrap only the fetch-and-load call in snapshot.download_phase() (never a decode) and tag an unusable fetched snapshot with snapshot.mark_model_invalid().
3. Optional abort() if the backend owns cleanup that must survive a failed start or a hard exit (e.g. a private temp dir): synchronous, idempotent, safe while a decode thread still runs, and undone by the next start(). A failed start calls it after close() when the backend's start() had returned, and alone when it had not; the server may then be retried (a failed start that left work running is single-use instead, but the backend can go to a new server). Optional idle() (IdleAwareBackend) reports when no decode thread is left; the server keeps its lock until then. The full contracts are in stt_server/backend.py.
4. Registry: a BackendEntry in stt_server/backends/__init__.py BACKENDS (module, class name, default model, local_model_paths).
5. Packaging: an extra in pyproject.toml (plus a [tool.uv] conflicts entry if its pins clash with the other MLX extras) and a CI job in .github/workflows/test.yml if it needs its own environment.
6. Tests: tests/test_backend_registry.py covers every registry entry; add a tests/test_<name>_backend.py with fake modules for the stream contract.
7. Docs: the backend table above, docs/operations.md (--backend list; socket table if it gets a LaunchAgent), and CHANGELOG.md.

Documentation

Deeper references live under docs/:

• Operations and deployment — multi-backend operation, two-agent LaunchAgent install, A/B benchmarking, health checks, and Whisper hallucination-suppression knobs.
• Client and integration — Python TranscriptionClient usage, the Pipecat SegmentedSTTService example, and the Koda reference consumer.
• Server architecture — the lifecycle state machine, the single-server lock, stop-during-startup, and what a supervisor can rely on.
• Protocol subset — the V1 wire events and how they deviate from the OpenAI Realtime transcription snapshot.
• Upgrading 0.1.x → 0.2.0 — runtime-surface rename migration.

Development plans live in docs/dev_plans/.