Vocal Helper

🇫🇷 · 🇬🇧

Vocal Helper belongs to a collection of libraries called AI Helpers developed for building Artificial Intelligence.

🌍 AI Helpers

The Promise

Local-first by design. vocal-helper runs entirely on your machine: transcription (turning speech into text), diarization (working out which of possibly several speakers said each part, and labelling them S0, S1, and so on) and summarisation all happen locally (whisper.cpp / pyannote / NeMo / local Ollama); your audio and transcripts are never uploaded to a third-party service, no telemetry, no account, no cloud lock-in. Your voice, and everyone else's on the recording, is among the most personal data there is, and a transcript is a verbatim record of what was said and by whom; keeping both on your own hardware is what makes this tool safe to point at a real meeting, interview, or therapy session. Part of the AI Helpers suite: sovereignty over your data through local-first Open Source.

Vocal Helper turns audio into diarized, transcribed utterances, with an optional rolling Large Language Model (LLM) summary of the conversation. Internally it is a producer/consumer pipeline: each processing step (detecting speech, identifying the speaker, transcribing, summarising) runs as its own worker, and each worker hands its output to the next one down a queue, the way a bakery's dough moves from the mixer to the oven to the cooling rack, each station working on a different batch at the same time instead of waiting idle for the whole order to finish. Two paths ship:

• Online (voh.Pipeline): live Pulse-Code Modulation (PCM) stream → live transcript + live summary. Each stage runs at its own cadence, decoupled by bounded queues. The Speech-to-Text (STT) stage warms up on start so the first caption doesn't stall on whisper's cold inference.
• Offline (voh.OfflinePipeline): full audio buffer → highest-quality diarization (pyannote 3.1 runs the whole meeting in one call; the 2026-07-14 offline map-reduce study found whole-buffer strictly best for Diarization Error Rate (DER), and chunk-and-stitch survives only as a memory backstop past ~1 h) → full-throttle batched transcript (consecutive segments concatenated into ≤ 24 s whisper calls, ~6.5× lower Real-Time Factor (RTF) at better Word Error Rate (WER) per the 2026-07-09 sweep) → summary. Opt back into per-segment Automatic Speech Recognition (ASR) with OfflinePipelineConfig(asr={"batch": False}).

Vocal Helper ships on PyPI under semantic-versioned tags (currently v2.0.4); continuous integration (CI, an automated pipeline that runs the test suite and the linter on every push) stays green on main before any tag goes out. It sits at the top of the AI Helpers dependency graph, the suite's most feature-complete and heaviest package. It builds on five siblings: os-helper for shared logging and file management; best-engine-ai-helper for the language-model routing described below; audio-helper for format-agnostic decoding; capture-helper for the live microphone source; and podcast-helper for any URL a browser could open. Each of those five has to work correctly for a single transcript to come out right, which makes this package the suite's most demanding integration test in daily use.

Documentation

💻 Documentation

🗺️ Landscape

📋 Examples

Pipelines

Every edge is a bounded asyncio.Queue ; every stage is its own coroutine. Colours follow the AI Helpers palette.

Online (streaming)

flowchart LR
    S([Source<br/><i>PCM frames</i>]):::source
      --> V[VAD<br/><i>Silero v5 ONNX</i>]:::vad
      --> D[Online Diar<br/><i>TitaNet · cosine clustering</i>]:::diar
      --> A[STT<br/><i>whisper.cpp turbo</i>]:::asr
      -.-> L[LLM analyst<br/><i>best-engine model · rolling summary</i>]:::llm

    classDef source fill:#CCE4FF,stroke:#007AFF,stroke-width:2px,color:#0b3d91
    classDef vad    fill:#00ffef,stroke:#79dbdc,stroke-width:2px,color:#003b3c
    classDef diar   fill:#EFDCF8,stroke:#AF52DE,stroke-width:2px,color:#4a1063
    classDef asr    fill:#FFEACC,stroke:#FF9500,stroke-width:2px,color:#5a3300
    classDef llm    fill:#D4F5D9,stroke:#28CD41,stroke-width:2px,color:#144d1e,stroke-dasharray: 5 5

The dashed edge marks the analyst as optional (llm=None disables it).

Offline (batch)

flowchart LR
    S([Source<br/><i>full PCM buffer</i>]):::source
      --> D[Offline Diar<br/><i>pyannote 3.1<br/>whole-buffer</i>]:::diar
      --> A[STT<br/><i>whisper.cpp turbo</i>]:::asr
      -.-> L[LLM analyst<br/><i>best-engine model · rolling summary</i>]:::llm

    classDef source fill:#CCE4FF,stroke:#007AFF,stroke-width:2px,color:#0b3d91
    classDef diar   fill:#EFDCF8,stroke:#AF52DE,stroke-width:2px,color:#4a1063
    classDef asr    fill:#FFEACC,stroke:#FF9500,stroke-width:2px,color:#5a3300
    classDef llm    fill:#D4F5D9,stroke:#28CD41,stroke-width:2px,color:#144d1e,stroke-dasharray: 5 5

No Voice Activity Detection (VAD) in the offline path: the diarizer consumes the whole buffer and does its own segmentation.

Backend router: the aiguilleur

Diarization is the one stage with a real backend fork (several competing implementations, each better in some cases), and there is no single winner: the best backend depends on the scenario. vocal_helper.router (voh.select_diarization) plays the role of an aiguilleur, French for the railway signalman who switches each train onto the right track before it arrives: it looks at a job's length, live-or-batch mode, and speaker count, then turns that measured trade-off into one explicit, tested decision, so the CLI and your own code never hard-code a backend. It reports both quality (DER) and speed (RTF) for the scenario, not just a name. Numbers were re-validated on-machine (studies/router_profile_validation.py, pyannote.metrics collar 0.25, median DER + RTF) against ground truth: bagarre (30 short mixes) + AMI dev-slice; sherpa from ADR 0002. DER = quality (lower better); RTF = speed (< 1 faster than real time):

Two findings, both measured here: offline has a real length crossover (nemo short ↔ pyannote long), so it needs a router; online has none, since vocal-helper's streaming clusterer is a latency-bound approximation where nemo wins at every length, so streaming always routes to nemo. voh.select_diarization(live=…, duration_s=…, max_speakers=…, torch_free=…, pyannote_available=…) returns a BackendPlan(mode, backend, expected_der, expected_rtf, reason); the quality/speed numbers are first-class fields and the reason carries the citation, so a choice is never a black box.

import vocal_helper as voh

plan = voh.select_diarization(live=False, duration_s=45.0, max_speakers=3)
print(plan.backend, plan.expected_der, plan.expected_rtf)  # nemo 0.142 0.051, short, ≤4 speakers
print(voh.select_diarization(live=False, duration_s=1800.0).backend)  # 'pyannote', long form

The router is enforced, not advisory: --diar-backend defaults to auto on both CLIs and POST /pipeline, so a file's real duration is probed and routed (short → nemo, long → pyannote) without you choosing. Pass an explicit pyannote / nemo / sherpa to override.

Installation

More recipes? See EXAMPLES.md for a self-contained, copy-runnable cookbook of the common workflows (live mic, URL replay, offline batch, subscribers, library + CLI usage).

Running the heavy stack on a Graphics Processing Unit (GPU)? See TECHNICAL_STACK.md for the full install recipe : CUDA + PyTorch, whisper.cpp with GGML_CUDA=on, pyannote 3.1 on MPS/CUDA, local Ollama, expected RTFs per GPU, and a reproducible install manifest covering the AI Helpers suite (os-helper, audio-helper, podcast-helper, youtube-helper, vocal-helper, music-helper).

Prerequisites: Python 3.10–3.13 and git, ffmpeg, PortAudio, cross-platform:

• 🍎 macOS (Homebrew): brew install python git ffmpeg portaudio
• 🐧 Ubuntu/Debian: sudo apt update && sudo apt install -y python3 python3-pip git ffmpeg portaudio19-dev
• 🪟 Windows (PowerShell): winget install Python.Python.3.12 Git.Git Gyan.FFmpeg (PortAudio ships inside the Python wheels)

We recommend using Python environments. Check this link if you're unfamiliar with setting one up: 🥸 Tech tips.

No compiler needed for the base install. The core (vocal-helper, no extras) pulls prebuilt wheels on every common platform: pywhispercpp ships wheels for macOS arm64, Linux x86_64/aarch64 and Windows (cp39–cp314), so nothing compiles. The heavy pieces are opt-in: the [nemo] extra brings ~5 GB of PyTorch, and offline diarization fetches a model bundle on first use (see Model weights below). Base install = library + CLIs + ASR/VAD + LLM analyst.

From PyPI (recommended)

pip install 'vocal-helper[all]'

From source (no PyPI)

git clone https://github.com/warith-harchaoui/vocal-helper.git
cd vocal-helper
pip install -e ".[all]"

The [all] extra brings the mic and stream sources, every diarization backend (NeMo, the default, plus pyannote and sherpa), and the language-ID cross-check. Pick à la carte if you don't need everything:

The LLM analyst ships in the base install, but it still needs an engine resolved once per machine before its serving backend is up. best-engine-ai-helper reads the committed vocal_helper/llm.brief.yaml, chooses the backend + model for your hardware (Ollama on macOS / CPU, vLLM on a discrete GPU), and writes a gitignored vocal_helper/llm.engine.yaml. voh.resolve_engine() does this on first use, or run it explicitly:

# From the vocal_helper package directory (holds llm.brief.yaml):
best-engine-ai-helper resolve --brief llm.brief.yaml --out llm.engine.yaml
# then start the resolved backend, e.g. on macOS:
ollama serve   # the engine file's `serve:` line names the exact `ollama pull …`

Model weights: no HuggingFace needed

All model weights ship in a single self-hosted diarization-engines bundle (offline pyannote 3.1, NeMo Sortformer, the online pyannote/embedding embedder, SpeechBrain VoxLingua107, and the torch-free sherpa ONNX, pyannote-3.0 segmentation + TitaNet). Point vocal-helper at it once and the whole stack runs HuggingFace-free: no token, no gated downloads, HF_HUB_OFFLINE=1 safe.

Configure it in settings.yaml (the only config the project needs):

cp settings.yaml.example settings.yaml
# settings.yaml already contains:
#   engines:
#     diarization_url: https://deraison.ai/diarization-engines-slim.zip
# settings.yaml is git-ignored.

What the URL is

https://deraison.ai/diarization-engines-slim.zip is a self-hosted ZIP (~800 MB) that mirrors every gated/hub-hosted model the pipeline needs, so the project never has to authenticate against HuggingFace. It contains:

On first use it is downloaded once, verified against manifest.json, and cached under ~/.cache/vocal-helper; later runs load straight from the cache. Set $VH_DIARIZATION_ENGINES to a local directory (or your own mirror URL) for air-gapped / self-hosted deploys. TitaNet (the default online-diar embedder) loads from NVIDIA NGC, also without HuggingFace.

Live microphone → terminal

# No token, no HuggingFace: weights come from the diarization-engines bundle.
vocal-helper mic --llm

Python API

import asyncio
import vocal_helper as voh


async def main():
    pipeline = voh.Pipeline(
        source=lambda: voh.sources.from_microphone(),
        config=voh.PipelineConfig(
            diar={"backend": "pyannote"},
            asr={"model": "large-v3-turbo-q5_0", "language": "auto"},  # discovered from the audio
            llm={"engine": voh.resolve_engine()},  # remove to disable
        ),
    )
    async for ev in pipeline.run():
        if "text" in ev:
            print(f"[{ev['t0']:.1f} {ev['speaker']}] {ev['text']}")
        elif "summary" in ev:
            print(f"--- rolling summary ---\n{ev['summary']}")


asyncio.run(main())

Replay a WAV through the online pipeline

vocal-helper file path/to/conversation.wav --llm

The file source preserves real-time pacing by default ; pass --no-real-time for as-fast-as-possible batch processing.

Offline batch on a WAV (full-buffer pyannote 3.1)

import asyncio, vocal_helper as voh


async def main():
    pipeline = voh.OfflinePipeline(
        source=lambda: voh.sources.from_wav_file("meeting.wav", real_time=False),
        config=voh.OfflinePipelineConfig(
            diar={"backend": "pyannote"},  # or "nemo" for ≤ 60 s clips
            asr={"language": "auto"},  # discovered from the audio, no default
            llm={"engine": voh.resolve_engine()},  # remove to disable
        ),
    )
    async for ev in pipeline.run():
        if "text" in ev:
            print(f"[{ev['t0']:.1f} {ev['speaker']}] {ev['text']}")
        elif "summary" in ev:
            print(f"--- digest ---\n{ev['summary']}")


asyncio.run(main())

When to use which, and the router picks the backend for you:

A toolbox: library, CLI, HTTP, MCP & GUI

vocal-helper is a toolbox, not an app. It exposes the same local pipeline through coherent surfaces so it composes into your own project without re-implementing the wiring. Everything runs locally: no surface sends audio to a remote service.

# argparse; language is discovered by default ('auto'), pass --language xx only to force one
vocal-helper transcribe clip.wav
vocal-helper file meeting.wav --offline --llm

# click twin: same operations, composable sub-commands
vocal-helper-click transcribe clip.wav

# local HTTP surface + transcript-viewer GUI (open http://127.0.0.1:8000/gui)
uvicorn vocal_helper.api:app --host 127.0.0.1 --port 8000 &
curl -F 'file=@clip.wav' http://localhost:8000/transcribe        # language auto-discovered
curl -F 'url=https://youtu.be/…' http://localhost:8000/pipeline  # URL fetched locally ([stream])

# MCP surface: the same local app, exposed as agent tools
vocal-helper-mcp

The transcript-viewer GUI (GET /gui)

A self-contained single page (HTML + Tailwind CDN + vanilla JS, no build step) served same-origin by the API. Drop an audio file or paste a URL, run diarized transcription locally, and read a speaker-labelled, colour-coded transcript (one stable colour per speaker) alongside the rolling summary. It POSTs to the same /pipeline endpoint (zero extra server logic) and contacts only the local server, so your audio never leaves the machine. Utterances reveal progressively (motion-guarded) so a long transcript reads as if it streams in.

Use it as an agent skill

skills/vocal-helper/ packages vocal-helper as a Claude Skill and an OpenCode skill so an agent can transcribe / diarize / summarise on your behalf. See skills/README.md to install (symlink into ~/.claude/skills/ and ~/.opencode/skills/), and TRIGGERS.md for the exhaustive catalogue of what invokes it.

Public surface added 2026-10-19

make_embedder(backend) and resolve_sherpa_models() replace the private names three repositories were importing across a boundary (_TitaNetEmbedder, _SherpaEmbedder, _resolve_sherpa_models). Both embedder backends emit the same 192-d TitaNet-large vector, so the choice is installation and platform, never quality: nemo through torch, sherpa through onnxruntime for iOS and Android.

Subscribers: fan-out without owning the loop

Every stage can be observed without consuming the merged output stream :

async def on_voiced(seg):
    print("VAD:", seg["t0"], seg["t1"])


async def on_diar(seg):
    print(" → ", seg["speaker"], seg["t0"], seg["t1"])


pipeline.subscribe_voiced(on_voiced)
pipeline.subscribe_diarized(on_diar)

async for ev in pipeline.run():
    ...

Useful for WebSocket / SSE relays, live User Interface (UI) updates, or JSONL persistence.

Diarization choice: offline and online have different winners

Measured, not assumed, on 2026-10-19.

Offline — pyannote community-1. On AMI (8 meetings, collar 0, overlap scored) it ties speaker-diarization-3.1 at a 0.116 median DER, so a median comparison says there is nothing to choose between them. The difference is the tail: 3.1's worst meeting is 0.474, community-1's is 0.149, and the spread collapses from 0.412 to 0.086. One meeting carries all of it, where community-1 is 3.3× better. On three real 64-minute six-speaker panels it also over-splits least of everything measured — 2.7×, against 3.0× for 3.1, 3.7× for Sortformer 4spk-v1 and 19.7× for the sherpa-onnx path. It is the default for _PyannoteOfflineDiar; a bundle that predates it falls back to 3.1 with a warning.

Online — diart. The opposite verdict, from the measurement operator-assistance-streaming had already made: 6.04 % speaker error against 11.7 % for the community-1 path on its long reference, and a first label in 500 ms against 20 s. DiartDiarStage is the port. It costs an environment of its own, and it revises what it already emitted — diart re-sends its whole annotation each step, so snapshot_to_delta turns that into a stream and counts the revisions, which is the honest price of the 500 ms. Read it next to the latency with LiveMetrics, never alone.

Still the default for the built-in online stage — cosine clustering. The pdbms study (2026-06-29, N=2089 per system) ranks streaming diarizers as hungarian_nemo (w=20 s, DER 0.13–0.20) up to 300 s and hungarian_pyannote (w=30 s, 0.30–0.45) beyond. OnlineDiarStage specialises that: the VAD already isolates each voiced segment, so the sliding window collapses to per-segment embedding plus cosine running-mean clustering. join_threshold=0.30 comes from the 2026-06-30 AMI dev-slice sweep.

Scoring: vocal_helper.bench

Plain WER cannot see how multi-speaker audio actually fails, so the module carries the metrics that can.

Live sessions: vocal_helper.live_metrics

Offline quality has WER and DER; live has latency, flicker and queue wait, and no one of them is the answer alone.

m = voh.LiveMetrics()
pipeline = voh.Pipeline(source=..., metrics=m)
...
print(m.summary())

Latency is counted from the end of the speech, because that is when a listener starts waiting. The revision rate matters because text that jumps is unusable at any word error rate — and zero revisions is not automatically good, it is paid for in latency. Per-queue waits say where the time went: a slow total with no queue wait is a slow model, the same total with a large one is a starved consumer. Opposite fixes.

violations enforces the one rule the whole live design rests on: no update appears before the audio that produced it. A non-zero count means a clock, a buffer or a mock is lying, and every number beside it is fiction.

Failures are loud

A pipeline that lost a stage used to return a normal, empty transcript with the reason only in a log. run() now raises StageCrashed, naming every stage that died, and pipeline.stage_failures carries the exceptions. Pass strict=False to opt back into leniency — a choice somebody makes rather than inherits.

Spoken-language identification

Before a word is transcribed, vocal_helper.lid decides which language is being spoken for the whole file, or per region of a code-switched recording. This matters because a plain whisper "auto" pass locks onto the first language it hears and translates the rest into it; identifying the language acoustically first lets each region be transcribed in its own language. It also catches mislabeled data: on a 423-call corpus the acoustic census overrode the folder labels on 21 files (English and Dutch calls filed under "FR", etc.).

Discovery-first: no default language, no pairing. Detection returns the language the input actually is (whisper's true argmax over its full language head). There is no default language and no language pair; the language is discovered from the audio itself.

import vocal_helper as voh

# Fast path, the right default for batch corpora that are mostly monolingual:
regions = voh.detect_language_regions_fast(pcm, 16_000)
for r in regions:
    print(f"{r.lang}  [{r.t0:.1f}–{r.t1:.1f}s]")

Opt-in routing hint. If you can only route a fixed set of languages, pass supported=("en", "fr", "es", "it", "pl", "nl") to re-rank detection within that set (so a close but un-routable relative, Galician over Spanish on a short window, never wins). This is entirely optional: leave it unset (None, the default) and the input speaks for itself.

Roadmap

Shipped since this list was first written: the --jsonl output writer, and the auto LLM-engine selector (voh.resolve_engine(), landed in 2.0.0). What's still open:

• SemanticEOTStage enabled by default, once the 2026-06-30 EOT study's false-cut reduction on AMI is confirmed at scale (LiveKit-style; see studies/eot_semantic_vs_silero.py). It ships today behind the opt-in --eot flag.
• A standard WebSocket relay for the online pipeline. The FastAPI surface (vocal_helper.api) intentionally stays offline-only; a WebSocket surface for the streaming path is planned but lives outside this package for now.
• Language-locked Whisper rejection for ASR hallucinations on silence.
• Out of scope by design: speaker ID anchoring via pre-enrolled voiceprints, excluded by the user's industrial deployment compliance constraints. IDs stay anonymous (S0, S1, …) within a session.
• Replace the in-stage _PyannoteEmbedder with the overlap-aware variant from pdbms.diar.backends.pyannote.embed_overlap_aware for noisy mixes.
• Pipecat-style typed Frame events with a SystemFrame priority queue in the production pipeline itself (clean shutdown, out-of-band control signals that bypass the data queues). vocal_helper.parallel_pipelines already ports Pipecat's fan-out primitive, but only for study scripts under studies/, not for Pipeline/OfflinePipeline.

Versioning & stability

vocal-helper reached its first stable release at 1.0.0 and is currently at 2.x, following Semantic Versioning:

• The public API is the names exported from vocal_helper.__all__ plus the documented CLI flags. That's what stability promises apply to.
• Breaking behaviour and default changes land only in MAJOR releases (1.x → 2.0.0). A MINOR release adds features compatibly; a PATCH is bug-fixes and docs only, it will never change a default under you.
• 2.0.0 is the one MAJOR shipped so far since 1.0.0: it replaced the hard-coded LLM model tag with a per-machine engine resolved from a committed brief (see the CHANGELOG), which changed the analyst's CLI/API surface, hence the major bump rather than a minor one.
• Deprecations get a release with a warning before removal.

Author

Warith HARCHAOUI, warithmetics@deraison.ai

Acknowledgements

Special thanks to Mohamed Chelali, Bachir Zerroug and Edmond Jacoupeau.

License

This project is licensed under the BSD-3-Clause License: see the LICENSE file for details.