coro-asr 1.1.0

Self-hosted speech-to-text that knows who said what — speaks both the OpenAI and Deepgram API contracts.

Source Code: https://github.com/collectiveai-team/coro

Coro is an embedded ASR + speaker-diarization server that speaks two industry API contracts natively — OpenAI's and Deepgram's. Point the official openai SDK or the official deepgram-sdk at it and get back typed transcripts that know who said what, no custom schema package needed.

Each provider gets its own endpoint implementing that provider's own contract. Coro never bolts one vendor's data onto another vendor's format:

Responses are validated against both vendors' own published SDK types in CI, so "compatible" is asserted rather than asserted-in-prose.

The name nods to coro (Spanish for "chorus") — many voices, transcribed and attributed to who spoke them.

The key features are:

• Two native API contracts — OpenAI and Deepgram, each on its own endpoint with its own request shape, defaults and error format; neither is an approximation of the other
• OpenAI-compatible API — drop-in /v1/audio/transcriptions; clients reuse the official openai SDK types (Transcription / TranscriptionVerbose / TranscriptionDiarized) with no custom schema
• Deepgram-compatible API — drop-in POST /v1/listen and WebSocket /v1/listen; the only way to get per-word speaker labels, since no OpenAI type has a slot for one
• Audio and video input — uploads are decoded through ffmpeg, so any container it supports works: audio (.wav, .mp3, .m4a, .flac, .ogg, …) and video (.mp4, .mkv, .mov, .webm, …); the audio track is extracted to 16 kHz mono PCM automatically — same endpoint, same response shapes
• Pluggable diarization backends — pick per deployment: NVIDIA NeMo Sortformer (streaming-capable, ≤ 4 speakers) or pyannote community-1 (batch/whole-file, handles > 4 speakers); both attribute every segment to a speaker (diarized_json), so you get who spoke, when, and what
• Pluggable ASR backends — pick per deployment: onnx-asr Parakeet (the default — fastest on CPU and GPU, strongest on Spanish), Faster-Whisper (best English meeting accuracy, multilingual), or onnx-genai Nemotron (real-time streaming)
• Two transcription pipelines — full-memory (default) decodes and holds the whole recording in RAM for lowest latency on short/medium clips; streaming streams 1 s PCM chunks off disk and spills the growing transcript to a per-request on-disk store, trading a little latency for flat host RAM on arbitrarily long audio. Select with CORO_PIPELINE / --pipeline — see the pipeline comparison
• Streaming both ways — OpenAI-exact SSE (transcript.text.delta / transcript.text.done / [DONE]) with stream=true, and a Deepgram-compatible WebSocket at /v1/listen that pushes Results frames as audio arrives
• Flat-memory long audio — the streaming pipeline spills the transcript to disk so host RSS stays flat from 11 s to multi-hour recordings
• CPU & GPU — mutually-exclusive cpu / cuda extras carry the matching onnxruntime wheels; multilingual on either
• Run it your way — ephemeral uvx, a standalone uv tool install command, or a full uv sync dev checkout

Quickstart

Run the server without installing it into a project, straight from the repo, using uvx (the alias for uv tool run). Pick the hardware extra that matches your machine:

# CPU-only
uvx --from "coro-asr[cpu]" coro --port 8000

# NVIDIA GPU
uvx --from "coro-asr[cuda]" coro --port 8000

uvx builds a throwaway isolated environment and launches the coro command — no uv sync/uv run and nothing added to your current project. The server now speaks both the OpenAI and Deepgram transcription contracts at http://127.0.0.1:8000/v1.

Then write a tiny client with the official openai SDK, pointing base_url at your Coro server (api_key is required by the SDK but ignored by Coro):

pip install "openai>=2.0.0"     # or: uv pip install "openai>=2.0.0"

from openai import OpenAI

# Point the OpenAI client at your Coro server instead of api.openai.com.
client = OpenAI(base_url="http://127.0.0.1:8000/v1", api_key="not-needed")

with open("audio.wav", "rb") as f:
    result = client.audio.transcriptions.create(
        file=f,
        model="whisper-1",  # accepted but ignored; server uses its backend
        response_format="diarized_json",  # json | verbose_json | diarized_json
    )

print(result.text)
for segment in result.segments:  # who spoke, when, and what
    print(f"[{segment.start:.2f}-{segment.end:.2f}] {segment.speaker}: {segment.text}")

segments[].speaker is a summary, not a guarantee. Segment boundaries come from sentence punctuation, never from a speaker change, so a segment can span a speaker turn. Its speaker is the duration-weighted majority of its own words — the right label for display, the wrong one to treat as exact for every word inside it. For per-word truth use POST /v1/listen, which carries a speaker on every word; diarized_json has no per-word slot. The JSON shape is unchanged, so no schema diff will flag this. See ADR 0014.

Or hit the endpoint directly with curl (the same OpenAI multipart contract):

curl http://127.0.0.1:8000/v1/audio/transcriptions \
  -F file=@audio.wav \
  -F model=whisper-1 \
  -F response_format=diarized_json

That's the whole integration — because Coro returns standard OpenAI shapes, the SDK parses the response into typed objects with no custom schema. See Client integration for streaming (SSE) and the full format ↔ type mapping.

Standalone install

To run Coro as a server (not hack on it), install it as an isolated CLI tool with uv tool install. This puts coro and coro-bench on your PATH — no clone, no project environment. Pick the hardware extra that matches your machine (cpu / cuda are mutually exclusive):

uv tool install "coro-asr[cpu]"    # CPU-only
uv tool install "coro-asr[cuda]"   # NVIDIA GPU

Then run the server directly (no uv run):

coro --port 8000

Upgrade with uv tool upgrade coro; uninstall with uv tool uninstall coro. For a throwaway run without installing at all, use uvx (see Quickstart). On a GPU host the coro-asr[cuda] build still needs the libcublas.so.12 loader-path fix — see GPU on a bare host.

Run with Docker

Prebuilt images are published to GHCR with -cpu / -gpu flavour suffixes (latest, the release version, and sha-… tags). The image entrypoint is coro, so append any --flag or CORO_* env var just like the CLI; the server binds 0.0.0.0:8000 inside the container.

# CPU
docker run --rm -p 8000:8000 \
  ghcr.io/collectiveai-team/coro:latest-cpu \
  --backend-asr onnx-asr --model-asr nemo-parakeet-tdt-0.6b-v3 --asr-device cpu \
  --backend-diarization nemo

# NVIDIA GPU (needs the NVIDIA Container Toolkit)
docker run --rm --gpus all -p 8000:8000 \
  ghcr.io/collectiveai-team/coro:latest-gpu \
  --backend-asr onnx-asr --model-asr nemo-parakeet-tdt-0.6b-v3 \
  --backend-diarization nemo

The --backend-diarization nemo flag turns on Sortformer speaker labels; omit it for an ASR-only server. The diarizer device defaults to auto (GPU when one is available), so you only need --diarization-device to pin it explicitly.

Cache downloaded model weights across runs by mounting a Hugging Face cache volume (avoids re-downloading on every container start):

docker run --rm -p 8000:8000 \
  -v coro-hf-cache:/root/.cache/huggingface \
  ghcr.io/collectiveai-team/coro:latest-cpu --port 8000

To build the image yourself instead of pulling, pass the matching CORE_IMAGE / EXTRA build args (see the Dockerfile):

# CPU
docker build -t coro:cpu \
  --build-arg CORE_IMAGE=ubuntu:noble --build-arg EXTRA=cpu .

# NVIDIA GPU
docker build -t coro:gpu \
  --build-arg CORE_IMAGE=nvidia/cuda:13.0.3-cudnn-runtime-ubuntu24.04 \
  --build-arg EXTRA=cuda .

Configuration

Coro can be configured two equivalent ways — use whichever fits your deployment, or mix both:

• Environment variables — CORO_-prefixed (host, port, backends, devices, etc.).
• CLI flags — every setting is also a --kebab-case flag, auto-derived from ServerSettings via pydantic-settings. Run coro --help to list them.

Each ServerSettings field maps to both forms, e.g. backend_asr → CORO_BACKEND_ASR (env) or --backend-asr (CLI). Precedence is CLI flags > environment variables > defaults. See coro/settings.py for the full list.

# Env vars (add CORO_BACKEND_DIARIZATION to enable speaker labels; omit for ASR-only)
CORO_BACKEND_ASR=onnx-asr CORO_MODEL_ASR=nemo-parakeet-tdt-0.6b-v3 \
  CORO_ASR_DEVICE=cuda CORO_BACKEND_DIARIZATION=nemo \
  coro --port 8000

# Equivalent CLI flags
coro --backend-asr onnx-asr --model-asr nemo-parakeet-tdt-0.6b-v3 \
  --asr-device cuda --backend-diarization nemo --port 8000

The diarizer device defaults to auto (GPU when available); add --diarization-device only to pin it. Drop --backend-diarization for an ASR-only server, or swap nemo → pyannote (--pipeline full-memory, needs --extra diar-pyannote and an HF token) for > 4 speakers — see Diarization backends.

Endpoints

response_format accepts json, verbose_json, and diarized_json. With stream=true the endpoint emits OpenAI-exact SSE (transcript.text.delta / transcript.text.done / [DONE]).

For a speaker on every word, use the Deepgram-native POST /v1/listen below — no OpenAI type has a slot for one.

API docs and contracts

Both documents are generated from the code — OpenAPI by FastAPI from the routes, AsyncAPI from the same types the SSE writer and the WebSocket handler serialise — so neither is hand-maintained and neither is committed to the repo. The socket is published only as AsyncAPI, because OpenAPI cannot describe one. CI exports both, lints them with redocly, and fails a PR that breaks the REST contract. Swagger UI and ReDoc are switched off: Scalar is the only renderer that can show the event-driven contract alongside the REST one. See ADR 0013.

Two transcription pipelines (full-memory vs streaming)

Coro ships two interchangeable pipelines behind the same OpenAI endpoint and response shapes; switch between them with CORO_PIPELINE / --pipeline (default full-memory). They differ only in how the audio and transcript are held in memory — the wire format you get back is identical.

• full-memory (default) — ffmpeg decodes the upload to PCM once, in full, and the pipeline holds the entire signal plus the accumulated tokens/segments/words in RAM. Simplest and lowest-latency for short to medium clips, but host RAM grows ~linearly with recording length, so it is not suited to unbounded audio. It is the only pipeline that works with the whole-file pyannote diarizer.
• streaming — ffmpeg streams 1 s PCM chunks off disk instead of buffering the whole recording, and the growing transcript spills to a per-request on-disk SQLite (WAL) store instead of Python lists. Consumed over SSE (stream=true) it keeps flat peak host RSS, independent of recording length (11 s ≈ 58 min ≈ multi-hour): only bounded working buffers stay resident and the final transcript.text.done frame is rendered straight from the store one segment/word at a time. This is the only pipeline that can diarize live as audio arrives, and it requires a streaming-capable backend (NeMo Sortformer for diarization).

The spill directory needs to be on real disk, and that is handled for you: at startup the server picks the system temp dir when it is real disk, otherwise a directory under your cache dir, because /tmp is tmpfs (RAM-backed) on most Linux distributions and spilling there would defeat the spill. Override it with CORO_TRANSCRIPT_SPILL_DIR; pointing it at a RAM-backed path fails startup rather than silently costing you the flat-RAM property. See Benchmarks for the measured memory behaviour.

ASR backends

The ASR backend is pluggable behind a single adapter contract. Select it with CORO_BACKEND_ASR + CORO_MODEL_ASR; pick the device with CORO_ASR_DEVICE (auto | cpu | cuda).

Recommended configuration

Each setting below is shown as an env var; the equivalent CLI flag is the --kebab-case form (e.g. --backend-asr onnx-asr).

The defaults (onnx-asr + nemo-parakeet-tdt-0.6b-v3, fp32) are already the recommended configuration on both CPU and GPU — you only need the settings below if you want to move off them.

GPU (--extra cuda):

CORO_ASR_DEVICE=cuda           # fp32 (leave CORO_ASR_QUANTIZATION unset)

Or as a CLI flag:

coro --asr-device cuda --port 8000

Fastest by a wide margin with near-best accuracy. Use faster-whisper + float16 if you want the top English-meeting accuracy point; use onnx-genai only for real-time low-latency streaming.

CPU (--extra cpu): nothing to set — the default selection is the CPU pick. Measured against faster-whisper + openai/whisper-medium on the same host: ~8.8× the throughput, 23% lower Spanish WER, ~1 GB less resident memory, and English meeting WER within noise. onnx-genai is not recommended on CPU.

Do not reach for int8 for speed. For this transducer model int8 measured no throughput gain (+0.6% / −3.6% across two workload sets — inside noise) and cost 3–6% relative WER. Its real benefit is memory: it drops the resident server from ~2.7 GB to ~1.3–1.7 GB. Set CORO_ASR_QUANTIZATION=int8 to fit a memory budget, never to go faster. Full numbers: docs/benchmark.md.

Streaming on long audio: set CORO_PIPELINE=streaming so the per-request transcript spills to disk and host RSS stays flat regardless of recording length; the spill directory defaults to real disk automatically. Consume the result over SSE (stream=true).

Diarization backends

Diarization is optional (default none — an ASR-only server is valid) and pluggable behind a single DiarizationAdapter contract, dispatched by a per-capability Backend Adapter Factory (see ADR 0007). Select it with CORO_BACKEND_DIARIZATION + CORO_MODEL_DIARIZATION; pick the device with CORO_DIARIZATION_DEVICE (auto | cpu | cuda).

Off by default is a deliberate choice, not an oversight. Turning Sortformer on costs ~24% throughput and ~1 GB of resident memory, adds a ~500 MB download on first start, and caps the server at 4 speakers — so it is opt-in rather than a default that could silently mis-attribute 5-speaker audio. Enabling it is one setting: CORO_BACKEND_DIARIZATION=nemo. Rationale and numbers: docs/benchmark.md.

Which to pick:

• NeMo Sortformer — choose for ≤ 4 speakers and/or when you need the streaming pipeline (Sortformer is the only streaming-capable backend). The diar_streaming_sortformer_4spk-v2 model is designed for at most 4 speakers; on meetings with more than 4 distinct speakers it will collapse the extras and DER degrades.
• pyannote community-1 — choose when a recording may contain more than 4 speakers. It clusters speakers over the whole file, so it is batch-only and is rejected at startup if you select CORO_PIPELINE=streaming (use full-memory). The model is gated: you must accept its conditions on the Hugging Face model page and provide a token.

Model licensing

coro-asr itself is MIT (see LICENSE), but model weights carry their own licences and you are responsible for complying with them. Every diarization model this project names:

The NeMo backend accepts any Sortformer checkpoint via CORO_MODEL_DIARIZATION, so nvidia/diar_sortformer_4spk-v1 will load if you ask for it explicitly — but it is CC-BY-NC-4.0, so doing so makes your deployment non-commercial. Leave CORO_MODEL_DIARIZATION unset to get the permissively licensed streaming default. When adding a new diarization model to this project, add its licence to the table above.

NeMo Sortformer setup (default, no token)

Sortformer ships with the core install — no extra dependency, no Hugging Face token. Just turn the backend on; the default model (nvidia/diar_streaming_sortformer_4spk-v2) is selected automatically and downloaded on first run.

# Batch (full-memory pipeline, the default) — env-var form
CORO_BACKEND_DIARIZATION=nemo coro --port 8000
# equivalent CLI form:
coro --backend-diarization nemo --port 8000

Combine with an ASR backend and pin the device as usual:

coro --port 8000 \
  --backend-asr onnx-asr --model-asr nemo-parakeet-tdt-0.6b-v3 \
  --backend-diarization nemo --diarization-device cuda

Sortformer is the only streaming-capable backend. To diarize live as audio arrives, switch the pipeline to streaming (optionally tune the latency tier):

coro --port 8000 \
  --backend-diarization nemo \
  --pipeline streaming \
  --diarization-latency very-high   # very-high | high | low | ultra-low

Either way, request response_format=diarized_json to get per-segment speaker labels back, or POST /v1/listen?diarize=true for per-word labels. Sortformer handles ≤ 4 speakers; for more, use pyannote below.

Sortformer post-processing (optional, see ADR 0010)

Sortformer's raw speaker-activity predictions go through a threshold-based post-processing step (onset/offset/padding/min-duration). Left unset, coro uses NeMo's own unconfigured baseline — no smoothing, no padding. Set CORO_DIARIZATION_POSTPROCESSING to one of NeMo's own published presets, or to a path to a custom YAML in the same schema, to override it:

coro --backend-diarization nemo --diarization-postprocessing dihard3-dev
coro --backend-diarization nemo --diarization-postprocessing none  # explicit baseline

Neither preset is a coro recommendation. They are NVIDIA's own published values for two specific domains; whether either is a good fit for your deployment's audio depends on how close your traffic is to one of those domains — coro does not know that, and does not compute or tune numbers against any benchmark on your behalf. If you have a representative sample of your own traffic to validate against, supply your own YAML in the same parameters: schema instead.

The target collar is part of the parameter set, not a footnote. Zero-collar scoring rewards boundary precision and near-zero padding; collar-tolerant scoring rewards generous padding and aggressive short-segment deletion. Scoring a set against the collar it was not tuned for measures the mismatch, not the model. coro-bench-diar therefore defaults to --postprocessing auto, which picks the preset matching its --collar; pass an explicit preset name to override, or none for NeMo's baseline.

If you A/B the presets yourself, record which reference you scored against. Which preset wins depends on whether the model is missing speech or inventing it, and that split is a property of the reference as much as of the model — two defensible references for the same corpus can agree on total DER while disagreeing about the direction of the error. A ranking that holds under only one reference is not yet a reason to change a default.

Speaker-count gate

When post-processing is enabled it is applied only when the estimated speaker count is at or below CORO_DIARIZATION_POSTPROCESSING_MAX_SPEAKERS (default 4); above it, that recording falls back to NeMo's baseline. NVIDIA's own v2 results show these thresholds improve DER for four or fewer speakers and consistently degrade it at five or more, because short-segment deletion removes the brief, fragmentary evidence the model has for the extra speakers — so applying them unconditionally makes the worst case worse.

This gate cannot fire on any currently shipped Sortformer revision. They are all 4-speaker models emitting a T x 4 activity matrix, so the estimate can never exceed 4. It is implemented now so the behaviour is already correct if a >4-speaker Diarization Model Selection is configured later, and the ceiling is a setting rather than a constant for the same reason.

pyannote setup (gated model + token)

1. Install the optional dependency (kept out of the core install):

   uv sync --extra cpu --extra diar-pyannote   # or: --extra cuda --extra diar-pyannote
   

2. Accept the user conditions for pyannote/speaker-diarization-community-1 on Hugging Face, then provide a token. Any of these is read (and the value is masked in logs); .env is loaded automatically:

   # .env (auto-loaded), or any of these env vars:
   HF_TOKEN=hf_xxx                 # standard HF name
   HUGGING_FACE_HUB_TOKEN=hf_xxx   # standard HF name
   CORO_HF_TOKEN=hf_xxx            # coro-namespaced
   

3. Run with the full-memory pipeline:

   CORO_BACKEND_DIARIZATION=pyannote CORO_PIPELINE=full-memory coro --port 8000
   # equivalent CLI: coro --backend-diarization pyannote --pipeline full-memory
   

Without a valid token (or before accepting the model conditions) the pyannote pipeline fails to load at startup with an actionable error.

Settings reference

Every setting below is available as both an environment variable and a CLI flag (CLI flags take precedence). Source of truth: coro/settings.py.

Benchmarks

Picking a backend? See the full leaderboard → docs/benchmark.md (WER, DER, RTFx, VRAM and RAM across backends, with reproduction commands). TL;DR: the default (onnx-asr parakeet, fp32) is the CPU pick and the Spanish pick; faster-whisper large-v3-turbo is the best English-meeting GPU option; faster-whisper small for max GPU throughput; nemotron for real-time streaming. Don't run Whisper through the onnx-asr backend (slower and less accurate than faster-whisper).

The table below is a separate, ASR-only view (diarization off).

Long-form English meetings from AMI (Mix-Headset, far-field, overlapping speech), diarization off, on an RTX 3070 Laptop (8 GB) and a loaded laptop CPU. RTFx = audio ÷ processing time (higher is faster). Quality = normalized ORC-WER, lower is better. (Absolute WER is high because AMI Mix-Headset is a hard far-field/overlap benchmark; treat the numbers as a relative comparison.)

Why this table's RTFx differs from docs/benchmark.md. RTFx is a property of the measurement, not the model, and the two documents measure different things: this table is long-form (10 min+) audio with diarization off, the leaderboard's headline tables are 60 s clips with diarization on. Short clips do not amortise per-request cost and diarization is included in the pipeline timing, which is most of the ~18× gap that used to sit unexplained between the ~120× below and the leaderboard's 6.7×.

‡ GPU figures are historical and were not reproduced by the current benchmark program — no artifacts for them exist in this repo. The CPU column and the WER column were re-measured on 40 min of AMI clips plus 12 min of Spanish FLEURS; whisper-medium's CPU RTFx was previously listed as 1.3× and measured 0.58×. See Measured defaults (CPU).

Memory footprint — baseline (peak, model + runtime, short clip):

Memory is not just the model on long audio. The default full-memory pipeline decodes and holds the entire PCM plus the accumulated tokens/segments/words, so host RAM grows ~linearly with recording length. The streaming pipeline (CORO_PIPELINE=streaming) streams 1 s PCM chunks from disk instead of buffering the whole recording, and spills the growing transcript to a per-request on-disk SQLite (WAL) store instead of Python lists.

When consumed over SSE (stream=true), the streaming pipeline keeps flat peak host RSS, independent of recording length (11 s ≈ 58 min ≈ multi-hour): finalized segments and raw words live on disk during the stream, only bounded working buffers stay resident, and the final transcript.text.done frame is rendered straight from the store one segment/word at a time (never materialised). The wire format is unchanged.

Notes:

• The on-disk store must live on real disk for the flat-RSS property, and the default now guarantees that: startup picks the system temp dir when it is real disk and a cache directory when it is tmpfs (RAM-backed, as /tmp is on most Linux distributions). An explicit RAM-backed CORO_TRANSCRIPT_SPILL_DIR fails startup instead of silently keeping the transcript in memory.
• The non-streaming transcribe() response inherently returns the whole transcript as one object, so its peak is O(length) at assembly time — use SSE consumption for unbounded audio.
• Diarizer prediction state grows ~0.7 MB/hour (frames × speakers × 4 bytes), negligible beside the model.
• GPU VRAM is length-independent in both pipelines (inference is windowed/streamed): parakeet ~3.6 GB, nemotron ~1.4 GB, faster-whisper ~2.3–2.9 GB.

Takeaways:

• English meeting quality is close across all three on this benchmark (parakeet 51.4% vs whisper-medium 51.6% normalized ORC-WER — a wash). On Spanish the gap is real: parakeet 5.8% vs whisper-medium 7.6% WER on FLEURS es_419, a 23% relative reduction.
• Parakeet is the throughput winner on both devices — ~8.8× whisper-medium on CPU, and its offline encoder batches frames on GPU.
• Use fp32, not int8, on either device. On GPU int8 inserts many CPU↔GPU copies; on CPU it is compute-bound, so int8 measured no speed gain and a 3–6% relative WER cost. int8's only real payoff is memory.
• Nemotron is a streaming model: ~10× on GPU and impractical on CPU (~0.4×). Its value is low-latency real-time transcription, not batch speed.
• Memory: all backends fit comfortably on an 8 GB GPU; nemotron (int4) is the lightest, and parakeet int8 is the smallest CPU footprint.

Benchmark datasets

Quality runs score against trustworthy, human-or-openly-labelled references only. Each is materialized into a --clips-dir of (<stem>.wav, <stem>.ref.stm) pairs; the bench scores WER and/or DER per the reference:

Diarization-only references (e.g. VoxConverse) carry speaker turns but no words; the report shows their DER and leaves WER blank rather than emitting a meaningless score.

Spanish is WER-only. Every public Spanish corpus above is single-speaker, so the Spanish workload set validates ASR quality and yields no meaningful DER. Diarization quality is measured on AMI.

Common Voice is no longer supported. It moved off its previous free distribution channel in late 2025 and is no longer reproducibly fetchable, so the make_common_voice_clips utility was removed. Use --spanish-preset for Spanish WER.

Note — Albayzín-RTVE2020 (out of scope). It is the strongest Spanish diarization target (human-revised transcripts and speaker labels), but it is gated behind an RTVE licence and cannot be redistributed, so it is not part of the reproducible workload set. Spanish diarization decisions stay AMI-driven. (Avoid the RTVE2018 subtitle-only partitions — those captions are not verbatim.)

Spanish workload set + published-WER calibration

# Print the one-time download footprint before committing to a fetch:
uv run --group bench coro-bench quality --spanish-preset calibration --spanish-fetch-plan

# Materialize + score (audio and Reference STM files land under --spanish-root):
uv run --group bench coro-bench quality \
  --spanish-preset calibration --server-url http://127.0.0.1:8123 --out-dir run

Corpora are fetched from the Hugging Face Parquet index one row group at a time, so only the rows the preset asks for are downloaded, and the result is cached. Each materialised directory ships a LICENCES.md and corpora.json recording the licence, source and item provenance of every corpus.

The fleurs and mls presets are calibration sets: their scored normalized ORC-WER is compared against the published figure for the configured ASR Model Selection, and a deviation beyond --calibration-margin (default 0.10 WER points, two-sided) fails the run with exit code 3. Matching an external figure is the only end-to-end proof the harness is free of systematic error, so a large deviation is a harness bug until proven otherwise. Results are written to <out-dir>/quality/calibration.json.

Running benchmarks

By default coro-bench starts and stops the server it measures (a bench-managed server): it spawns coro on a free port with the CORO_* env vars implied by the --server-* flags, waits for /health to report ready and warmup-ready, runs the workload, and tears the server down afterwards. Install the bench tooling first:

uv sync --group bench                       # meeteval, nvidia-ml-py, rich

To measure a server you started yourself (a bench-attached server), pass --server-url; the --server-* flags are then rejected as mutually exclusive:

uv run --group bench coro --port 8123 &     # server under test (add --extra cuda for GPU)
uv run --group bench coro-bench all --server-url http://127.0.0.1:8123 ...

Pass --group bench (and your hardware --extra) on every uv run below: a bare uv run re-syncs to the default environment and would uninstall the bench tooling again (the same re-sync gotcha as the cuda extra — see GPU on a bare host).

Three subcommands share the same flags:

Smoke test on one small audio

A reference STM has one line per segment — <recording_id> <channel> <speaker> <start> <end> <text> — where recording_id is the audio filename stem. The package vendors an 11 s jfk.wav:

echo "jfk 1 JFK 0.000 11.000 and so my fellow americans ask not what your country can do for you ask what you can do for your country" > jfk.ref.stm

uv run --group bench coro-bench all \
  --audio coro/bench/data/jfk.wav \
  --reference-stm jfk.ref.stm \
  --out-dir ./bench-out

quality requires --reference-stm (and all needs it to score the quality half); performance does not. The run prints a report and writes REPORT.md plus responses/ hyp/ ref/ quality/ performance/ under --out-dir.

Larger workloads

• --clips-dir DIR — a directory of (<stem>.wav, <stem>.ref.stm) pairs, e.g. produced by the dataset materializers (utils.make_ami_clip, utils.make_rttm_clip).
• --ami-preset sample|eval|full (or --ami-groups / --ami-meetings) — pull AMI meetings into --ami-root (default ./amicorpus/); add --no-download to use only what is already present.
• --spanish-preset voxpopuli|fleurs|mls|calibration|all — materialize a Spanish workload set from public CC0/CC-BY corpora into --spanish-root (default ./spanish-corpora/) and score it. Mutually exclusive with --clips-dir.

Useful flags

Client integration

This server returns standard OpenAI shapes. A consuming project does not need to redefine any schemas — install the openai SDK and reuse its types.

pip install "openai>=2.0.0"

Option A — use the OpenAI client directly (returns typed objects)

from openai import OpenAI

client = OpenAI(base_url="http://<host>:<port>/v1", api_key="not-needed")

with open("audio.wav", "rb") as f:
    result = client.audio.transcriptions.create(
        file=f,
        model="whisper-1",  # accepted but ignored; server uses its configured backend
        response_format="diarized_json",  # -> openai.types.audio.TranscriptionDiarized
    )

print(result.text)
for segment in result.segments:
    print(segment.speaker, segment.start, segment.end, segment.text)

Option B — import the response types for manual validation

from openai.types.audio import (
    Transcription,  # response_format="json"
    TranscriptionVerbose,  # response_format="verbose_json"
    TranscriptionDiarized,  # response_format="diarized_json"
)

payload = httpx.post(url, files=..., data={"response_format": "verbose_json"}).json()
parsed = TranscriptionVerbose.model_validate(payload)

Option C — call the HTTP endpoint directly with curl

No SDK required — POST /v1/audio/transcriptions accepts a standard multipart form (file, model, response_format) and returns the OpenAI JSON shapes:

# Non-streaming (json | verbose_json | diarized_json)
curl http://<host>:<port>/v1/audio/transcriptions \
  -F file=@audio.wav \
  -F model=whisper-1 \
  -F response_format=diarized_json

# Streaming token deltas over SSE
curl -N http://<host>:<port>/v1/audio/transcriptions \
  -F file=@audio.wav \
  -F model=whisper-1 \
  -F response_format=json \
  -F stream=true

Format ↔ SDK type mapping

Conformance is enforced by tests/test_openai_sdk_conformance.py, which validates every server response against the SDK types.

Note: standard OpenAI types carry segment-level speaker labels only — there is no OpenAI-compatible slot for a per-word speaker. Use the Deepgram-native endpoint below to get one.

Per-word speakers: POST /v1/listen (Deepgram-compatible)

Coro assigns a speaker to every word and keeps each word's real ASR timing and confidence. No OpenAI type can carry that, so rather than bending OpenAI's format, Coro implements Deepgram's own endpoint contract: raw audio body (not multipart), Deepgram's query parameters and defaults, and Deepgram's error shape.

curl -s "http://localhost:8000/v1/listen?diarize=true&utterances=true" \
  -H "Authorization: Token any-value" \
  -H "Content-Type: audio/wav" \
  --data-binary @audio.wav

{
  "results": {
    "channels": [ { "alternatives": [ { "transcript": "hola mundo si",
      "words": [ { "word": "hola", "start": 0.0, "end": 0.5,
                   "confidence": 0.91, "speaker": 1 } ] } ] } ],
    "utterances": [ { "speaker": 1, "transcript": "hola mundo",
                      "start": 0.0, "end": 1.0, "confidence": 0.87,
                      "words": [ ... ] } ]
  }
}

Responses parse with the official SDK, enforced by tests/test_deepgram_sdk_conformance.py:

from deepgram.types.listen_v1response import ListenV1Response

ListenV1Response.model_validate(response.json())

Behaviour worth knowing:

• diarize and utterances default to false, exactly as at Deepgram, so per-word speakers need ?diarize=true. Coro does not override vendor defaults to be more helpful.
• Timestamps are float seconds; speaker numbering is Coro's (1-based), passed through rather than renumbered, so labels stay comparable with diarized_json.
• A word the diarizer does not cover has no speaker key — Deepgram never emits a null speaker, and inventing a label would be a guess.
• speaker_confidence is omitted: Coro's diarizers binarize their per-frame posteriors, so the value does not exist to report.
• Authorization is accepted and never validated. Coro has no auth.
• ?utterances=true roughly doubles the body, because the shape carries every word twice (flat and nested per utterance) — as the real API does.

What is and isn't supported

This is a documented subset, not a full clone. Of Deepgram's 37 pre-recorded parameters — 3 honoured, 16 refused, 18 ignored:

Unhonoured parameters are ignored, each documented in the OpenAPI schema with what its absence means, so a standard parameter bundle still works and a future Deepgram flag will not break the endpoint. Features Coro does not compute simply produce no response key.

Two are refused, because ignoring them fails silently instead of visibly: redact would return unredacted text under a redaction request (a compliance failure wearing a 200), and callback would leave a client waiting forever for a webhook that never fires. A missing summary key you can see; those two you cannot.

Also not implemented:

• URL ingest. {"url": "..."} bodies are refused with a clear message; submit audio as the raw request body.
• listen/v2 — WebSocket-only, and its distinguishing feature is contextual turn detection, which Coro has no equivalent of.
• interim_results, vad_events, utterance_end_ms — Coro emits only tokens it has already accepted, so every frame is final.

Live streaming: WebSocket /v1/listen

Deepgram's streaming contract is a WebSocket, so Coro implements one. This is genuine live transcription — Results frames are pushed as windows complete, not after the client stops sending.

import json, websockets

async with websockets.connect(
    "ws://localhost:8000/v1/listen?encoding=linear16&sample_rate=16000&diarize=true"
) as ws:
    await ws.send(pcm_chunk)  # binary frames: raw samples
    await ws.send(json.dumps({"type": "KeepAlive"}))
    print(json.loads(await ws.recv()))  # {"type": "Results", ...}
    await ws.send(json.dumps({"type": "CloseStream"}))

• Audio is declared, not sniffed. A socket has no container, so encoding=linear16 is required; other encodings are refused at connect time with an Error frame rather than after a minute of noise. Non-16 kHz rates are resampled.
• Control frames: KeepAlive, Finalize, CloseStream.
• Interim frames carry no speaker. The diarization timeline is incomplete while audio is arriving, so a mid-stream label would be a guess later frames contradict. With diarize=true a final attributed frame is sent once the timeline is complete — a deliberate deviation from Deepgram, which labels interim words.
• The stream always ends with a Metadata frame.

See docs/adr/0015-vendor-native-endpoints.md for the fidelity policy. /v1/audio/transcriptions is byte-unchanged, asserted in tests/test_openai_formats_unchanged.py.

Coro currently speaks two vendor dialects — OpenAI and Deepgram. Whether a further dialect is added is a question of whether that vendor's contract can be met under the fidelity policy, not of a fixed count. See ADR 0015.

Development

To hack on Coro, clone the repo and install into a project environment with uv sync. Pick the runtime that matches your hardware — the cpu / cuda extras are mutually exclusive and carry the matching onnxruntime / onnxruntime-genai wheels:

git clone https://github.com/collectiveai-team/coro && cd coro
uv sync --extra cpu     # CPU-only
uv sync --extra cuda    # NVIDIA GPU

Add --extra diar-pyannote (combinable with cpu or cuda) for the gated pyannote diarization backend — see Diarization backends:

uv sync --extra cpu --extra diar-pyannote

Run the server and the checks from the project environment with uv run:

uv run coro             # or: uv run uvicorn coro.app:app
uv run pytest           # tests
uv run ruff check .     # lint

GPU on a bare host

Running the GPU build outside the devcontainer has two gotchas:

1. uv run re-syncs to the default environment and uninstalls the cuda extra. Run the server with the extra explicitly so the GPU wheels stay installed: uv run --extra cuda coro (or re-run uv sync --extra cuda after any plain uv sync / uv run). (uv tool install "coro-asr[cuda]" is not affected — its environment is not re-synced.)
2. faster-whisper (CTranslate2) needs libcublas.so.12 + cuDNN 9, which ship in the nvidia-cublas-cu12 / nvidia-cudnn-cu12 wheels (pulled by the cuda extra) but are not on the loader path by default. If you see RuntimeError: Library libcublas.so.12 is not found, prepend the wheel lib dirs to LD_LIBRARY_PATH:

   export LD_LIBRARY_PATH="$VIRTUAL_ENV/lib/python3.12/site-packages/nvidia/cublas/lib:\
   $VIRTUAL_ENV/lib/python3.12/site-packages/nvidia/cudnn/lib:$LD_LIBRARY_PATH"
   

   The shipped Docker GPU image bakes the nvidia-cublas-cu12 wheel dir onto LD_LIBRARY_PATH for you (see the Dockerfile runtime stage). In the devcontainer or a bare uv env — both now on the nvidia/cuda:13.x base, which ships libcublas.so.13, not .so.12 — you still need the export above. The onnx-asr / onnx-genai backends use onnxruntime-gpu (CUDA 13, matching the base) and are unaffected by gotcha 2.

License

MIT © collective.ai, jedzill4