Voxint

From sound to intelligence: end-to-end transcription, diarization, and speaker identity with human-grade quality gates.

Voxint turns any audio or video file into an enhanced, speaker-attributed transcript:

local file · upload · URL → acquire → preprocess
    → transcribe (Whisper) + diarize (pyannote) + embed (TitaNet)
    → LLM transcript enhancement → speaker matching → human adjudication

Point it at a local file (voxint submit), upload one through the browser, or hand it a URL (voxint fetch / POST /fetch) — a yt-dlp download runs as the pipeline's first stage. URL ingestion is authenticated admin egress, not a sandbox: fetch only trusted URLs unless you run the worker with restricted egress (no route to private / link-local / metadata addresses). See docs/operations.md.

What makes it different is the orchestration "glue" most pipelines skip:

• Quality gates at every stage — non-speech/digital-silence triage before you burn GPU time, hallucination soft-tagging and stripping, chunk-completeness checks, outage-vs-data-defect taxonomy with explicit retry budgets.
• Durable state, not vibes — a compare-and-swap'd run/stage state machine in Postgres; a crash at any stage is recoverable, and human pauses are database state, never a held task.
• Speaker identity done honestly — pgvector cosine matching against a grown speaker roster, a strict named ≠ grounded invariant, and machine proposals kept separate from human rulings.
• A built-in adjudication web UI — review queue, guarded slot workbench, and an immutable decision ledger, served as Jinja + htmx from the same FastAPI app (no Node toolchain).
• Operable from the browser — a keyset-paged /runs execution-history browser (with a per-stage attempt ledger), bounded file upload, and yt-dlp URL ingestion, from the same app. Submission is durable-first: a broker outage leaves the run queued for the recovery sweep, never lost. The console is append-only — no delete, no cancel.
• Measurement harnesses — speaker-attribution scoring, runnable as CLIs: name-accuracy against ground truth (McNemar / bootstrap / Wilson), acoustic agreement verdicts, and verdict-level ensemble fusion (worked example under examples/). They score who spoke, not what was transcribed — ASR accuracy / WER measurement is out of scope today.

The adjudication console

Machine proposals stay separate from human rulings: the review queue lists completed runs with voices still needing a ruling, and the slot workbench shows each voice's evidence — grounded cosine matches, unverified LLM-heard names, transcript previews — with assign / enroll / exclude / unknown actions. (Synthetic demo data pictured.)

Status

Pre-alpha. APIs, schema, and layout may change without notice through the 0.x series.

Quickstart

Requires Docker Engine with the Compose plugin ≥ 2.24 (docker compose version — the legacy v1 docker-compose binary cannot parse this stack).

No NVIDIA GPU? Start here too. Voxint does not need one — the CPU tier runs the full pipeline on plain amd64/arm64 servers and Apple Silicon with zero GPU configuration. Follow the same quickstart, then use the compose.cpu.yaml overlay where the GPU one appears — see No NVIDIA GPU? (CPU tier). AMD GPU? The ROCm tier accelerates transcription on it (4.8× the CPU baseline, amdgpu kernel driver is the only host requirement) — use compose.rocm.yaml — see AMD GPU? (ROCm tier).

git clone https://github.com/bengizmo/voxint.git && cd voxint

Guided install (recommended for a first run):

./scripts/install.sh

It asks for an admin password, a media folder, a compute tier for the model services (GPU / CPU / none for now), and — for the GPU/CPU tiers — a Hugging Face token (the pyannote diarization weights are HF-gated; the installer explains the two model gates to accept and checks the token, warnings only). It generates everything else (including a random CSRF_SECRET), pulls the pinned release images, starts the core stack plus your chosen tier's model services, waits for the API to report healthy, and prints the console URL. It is safe to re-run — an existing .env is kept unless you ask to regenerate it (which backs the old one up first), and your tier choice is remembered (VOXINT_COMPOSE_TIER). Skip the token and only the core control plane starts (console, review UI, durable pipeline state — enough to adjudicate, not to process audio); the completion notice explains how to finish.

Or configure by hand:

cp .env.example .env          # then edit at least VOXINT_PASSWORD
mkdir -p media                # media mount; pre-create so it isn't root-owned
docker compose pull           # prebuilt release images from GHCR
docker compose up -d          # Postgres+pgvector, Redis, migrate, API + review UI, worker, beat
curl http://127.0.0.1:8080/healthz   # default port; matches API_PORT if you changed it

The default compose files run the pinned release images — even from a main checkout (set VOXINT_IMAGE_TAG in .env to run a different release). A one-shot migrate service brings the schema to head before the API and worker start — it showing Exited (0) in docker compose ps -a is success, not a crash. If a default port is already in use on your host, override the published side in .env (POSTGRES_PORT, REDIS_PORT, API_PORT). Details and day-2 operations: docs/operations.md.

Open the console at http://127.0.0.1:8080/ (HTTP Basic, the VOXINT_USER / VOXINT_PASSWORD you set). On a fresh install the console holds you at a first-run setup wizard (/setup) — configure media folders, vocabulary, and optional LLM enhancement in the browser, then finish into a short guided tutorial on a bundled three-speaker sample. Full walkthrough: docs/onboarding.md.

Once onboarding is complete, browse runs at /runs and adjudicate at /review. Feed it work by uploading a file, pointing it at a URL (docker compose exec api voxint fetch <url>), or submitting a local path (docker compose exec api voxint submit path/to/file.mp3, relative to MEDIA_ROOT).

To run the GPU model services too (one NVIDIA GPU assumed), first set HF_TOKEN in .env — the pyannote service's diarization weights are HF-gated, so you need a Hugging Face token with access to the pyannote models accepted (see services/pyannote/README.md); compose refuses the GPU overlay without it.

All three services share the one GPU. Their loaded weights total roughly 3.5–4.5 GB of VRAM (whisper large-v2 int8 ~1.5 GB, pyannote ~1–2 GB, TitaNet ~1 GB); budget ~6–8 GB in practice for Whisper's batch/decode headroom and three separate CUDA contexts. An 8 GB card is comfortable. (Per-service figures live in each services/*/README.md.)

Then:

docker compose -f compose.yaml -f compose.gpu.yaml pull
docker compose -f compose.yaml -f compose.gpu.yaml up -d

Per-service details, env tunables, and image matrices: services/*/README.md; wire contracts: docs/gpu-contracts.md.

No NVIDIA GPU? (CPU tier)

The same three model services ship as multi-arch (amd64 + arm64) -cpu images — no GPU, no NVIDIA toolkit, runs on plain servers and Apple Silicon via Docker Desktop:

docker compose -f compose.yaml -f compose.cpu.yaml up -d

Be honest with your expectations: CPU inference is orders of magnitude slower — a long recording that takes minutes on a GPU takes hours on CPU. The overlay sets COMPUTE_TIER=cpu, which scales the pipeline's timeouts and stage leases so slow-but-healthy runs aren't reclaimed as hung. Same contracts, same embedding space (TitaNet runs on ONNX Runtime under a measured-equivalence parity gate). Details: docs/operations.md.

AMD GPU? (ROCm tier)

A hybrid tier for amd64 hosts with an AMD GPU: transcription (whisper) runs on the GPU via the CTranslate2 ROCm build — same engine, same code path, measured 4.8× the CPU baseline on RDNA4 — while diarization and speaker embedding run the -cpu images (MIOpen convolutions currently fail on AMD consumer GPUs; tracked in #4):

docker compose -f compose.yaml -f compose.rocm.yaml up -d

The host needs only the amdgpu kernel driver — no ROCm install, no container toolkit; the -rocm image carries its own ROCm runtime. The overlay sets COMPUTE_TIER=rocm (GPU-speed ASR, CPU-scaled leases for the rest). Details: docs/operations.md.

To run the source you checked out instead of the release images, layer the build overlays (exactly one service owns each build — see docs/operations.md):

docker compose -f compose.yaml -f compose.build.yaml build api
docker compose -f compose.yaml -f compose.build.yaml up -d

For development without Docker:

uv sync --extra dev
uv run pytest tests/unit
uv run uvicorn voxint.api.app:app --reload

The scoring harness needs none of the stack — pip install voxint gives you the voxint score CLI (pure file-in/file-out, no database or GPU services; speaker-attribution metrics only, no ASR/WER); see examples/.

Deployment model

Docker-compose-first on a single Linux machine with one NVIDIA GPU:

• compose.yaml — Postgres (+pgvector), Redis, one-shot migrate, API (+ review UI), Celery worker, Celery beat (crash-recovery sweep scheduler)
• compose.gpu.yaml — the GPU model services: faster-whisper, pyannote, TitaNet
• compose.build.yaml / compose.gpu.build.yaml — build-from-source overlays for development

Kubernetes is explicitly not required (a future optional enhancement).

Modularity

ASR, diarizer, embedder, and LLM providers sit behind typed protocols with versioned HTTP contracts (/v1/transcribe, /v1/diarize, /v1/embed). The LLM enhancement stage speaks to any OpenAI-compatible endpoint and is optional (LLM_ENABLED=false by default). Domain-specific vocabulary and prompts load from a swappable domain pack (DOMAIN_PACK_PATH); a neutral meeting/podcast pack ships as the default.

License

Apache-2.0. See LICENSE and NOTICE — model weights (e.g. pyannote's HF-gated checkpoints) are subject to their own terms and are downloaded with your credentials; Voxint never vendors them.