livekit-plugins-voicefilter-lite

LiveKit Agents plugin that streams target-speaker enhancement in front of an arbitrary streaming STT by wrapping open-voicefilter-lite. It slots in as a livekit.agents.stt.STT so it can sit right next to the speech recogniser, getting read access to the underlying STT's SpeechEvent.alternatives[0].start_time (sentence-start times) for snapping the d-vector enrolment window to a word boundary.

The aim is to reduce false barge-in / phantom recognitions in multi-talker environments by isolating the target speaker's voice before the STT sees the audio. A two-phase startup handles the cold-start problem:

1. Buffering — no target-speaker reference yet, so audio is forwarded to the underlying STT unchanged, and accumulated in a rolling PCM ring the plugin maintains.
2. Enhancing — once enough enrolment audio is collected, it's passed to open_voicefilter_lite.StreamingVoiceFilter.set_enrollment; from then on every frame is filtered through the mask predictor before reaching the underlying STT.

The "enough audio" check has two paths because phone-call scenarios rarely give you 5 s of continuous target speech up front:

• Single continuous slice — the user speaks without pause for ≥ enrollment_seconds. The last N seconds of the rolling ring are fed to the d-vector encoder, optionally snapped to the most recent sentence start from STT transcripts so the window starts on a word boundary.
• Multi-turn stitcher (phone-call / short-utterance case) — each turn's transcribed audio range (read out of SpeechData.start_time / end_time) is sliced out of the rolling ring and appended to a separate segments pool. When the cumulative pool size reaches enrollment_seconds, the chunks are concatenated and passed to set_enrollment. This is what kicks in when the user only says a few words per turn and there is no pre-recorded enrolment sample.

Install

# uv
uv add livekit-plugins-voicefilter-lite

# pip
pip install livekit-plugins-voicefilter-lite

Quick start

from livekit.agents import AgentSession, Agent
from livekit.plugins import deepgram, voicefilter_lite as vfl

stt = vfl.STT(
    underlying_stt=deepgram.STT(model="nova-2"),
    enrollment_seconds=5.0,
)

session = AgentSession(
    vad=...,                     # any VAD works
    stt=stt,                     # ← the plugin sits here
    llm=...,
    tts=...,
)
await session.start(agent=MyAgent(), room=ctx.room)

The plugin automatically harvests sentence-start times from the wrapped STT's SpeechEvents (FINAL_TRANSCRIPT, INTERIM_TRANSCRIPT, PREFLIGHT_TRANSCRIPT). When the buffer hits 5 s and a usable sentence boundary exists, enrolment happens on the next frame and subsequent frames are enhanced.

If your provider's transcripts don't carry aligned timestamps — or you want to anchor enrolment differently — feed sentence starts manually:

stream = stt.stream()
async for ev in stream:
    if ev.type in (stt.SpeechEventType.INTERIM_TRANSCRIPT, stt.SpeechEventType.FINAL_TRANSCRIPT):
        for alt in ev.alternatives:
            if alt.start_time:
                stream.mark_sentence_start_at_seconds(alt.start_time)

Or supply a pre-recorded enrolment:

import soundfile as sf
pcm, sr = sf.read("target_speaker.wav", dtype="float32")
# Triggered from the same event loop driving the stream:
stream = stt.stream()
stream.enroll_with_audio(pcm, sample_rate=sr)

Phone-call / no-preroll scenario

The default enrollment_seconds=5.0 is calibrated for the d-vector quality floor, but it does not require 5 s of continuous target speech. In a phone call there is no pre-recorded sample, and the target speaker typically only says a few words per turn ("yes", "book it", "Tuesday at three"). The wrapper handles this by stitching the d-vector across multiple turns:

• Every FINAL_TRANSCRIPT (and PREFLIGHT_TRANSCRIPT) event whose alternatives carry both start_time and end_time has its transcribed range sliced out of the rolling PCM ring and appended to a separate segments pool.
• When the cumulative pool size reaches enrollment_seconds (e.g. after 4–6 short turns), the chunks are concatenated and passed to StreamingVoiceFilter.set_enrollment.
• Once enrolled, the wrapper starts enhancing every frame and the continuous ring stops mattering — subsequent barge-in attempts from off-target speakers are filtered out before the STT sees them.

If the user instead happens to talk nonstop for ≥ enrollment_seconds, the continuous-slice path kicks in instead and the d-vector is built from a single aligned slice. The two paths coexist; whichever fires first wins.

API

vfl.STT(...) (alias for VoicefilterLiteSTT)

The wrapper inherits the underlying STT's STTCapabilities (interim results, diarization, keyterms, chat_context) so the agent's downstream behaviour is unchanged.

VoicefilterLiteStream

Each vfl.STT.stream() call returns one of these. Useful accessors:

EnrollmentAccumulator

A small standalone helper — handy in tests, or when only the alignment math is needed without an STT. Keeps a rolling PCM ring and a list of sentence-start offsets.

acc = vfl.EnrollmentAccumulator(sample_rate=16000, max_seconds=30.0)
acc.add_frame_samples(samples_1d_float32)
acc.mark_sentence_start_at_seconds(0.4)
# … many frames later …
if acc.has_enrollment_audio:
    audio_1d = acc.take_enrollment(seconds=5.0)  # 16 kHz mono float32

How it works

              ┌─────────────────── buffering ─────────────────────┐
              │                                                   │
   mic ──────▶│ RoomInput ──▶ VoicefilterLiteStream               │
              │ (frames → _input_ch)                              │
              │                │                                  │
              │   ┌─── passthrough ──────────────────────────┐    │
              │   │                                          ▼    │
              │   │                            underlying_stream ──▶ STT (provider)
              │   │                                ▲                │
              │   │                                │                ▼
              │   │       ┌── EnrollmentAccumulator (continuous    final / interim transcripts
              │   │       │   ring)                                back unchanged to the agent
              │   │       │   ← frames from _input_ch              │
              │   │       │   ← sentence-start markers from        │
              │   │       │     FINAL/INTERIM/PREFLIGHT events     │
              │   │       │                                        │
              │   │       └── segments pool (FIFO)                 │
              │   │           ← transcribed ranges only            │
              │   │             ([start_time, end_time]            │
              │   │              sliced per FINAL/PREFLIGHT)       │
              │   │                                                │
              │   ├─ path 1 ─ stitched segments ≥ 5 s ──┐         │
              │   │   (phone-call / short utterances)   │         │
              │   │                                       ▼         │
              │   │                            take_segments_enrollment
              │   │                                       │         │
              │   ├─ path 2 ─ continuous ring ≥ 5 s ─┐    │         │
              │   │   (user talking nonstop)         │    │         │
              │   │                                  ▼    ▼         │
              │   │                            take_enrollment     │
              │   │                              (align to marker)  │
              │   │                                       │         │
              │   └───────────────────────────────────────┘         │
              │                                                   │
              │                            ▼                      │
              │                 StreamingRecognitionEnhancer      │
              │                 (StreamingVoiceFilter.set_enrollment)
              │                            │                      │
              └────────────────────────────┴──────────────────────┘
                                              ▼
                                       enhanced frames → underlying stream
                                       (only target-speaker audio reaches STT)
                                       ↓
                                       fewer false barge-ins

Why wrap the STT rather than plug into noise_cancellation?

LiveKit's room_io.AudioInputOptions(noise_cancellation=...) slot is a purely-audio frame hook — it sees audio in, audio out, nothing else. By wrapping the STT, we sit at the closest possible position to the recogniser and gain read access to the upstream provider's SpeechEvent.alternatives[0].start_time. That timestamp is the linchpin of the enrolment-window alignment: it lets us trim the 5 s d-vector clip to start on a sentence boundary rather than mid-word.

If your use case only needs to enhance audio (no STT-driven enrolment alignment) you can also wire the bundled StreamingVoiceFilter directly at the audio front-end — the plugin's core modules (EnrollmentAccumulator, StreamingRecognitionEnhancer) are split out so the PCM path is reusable independently.

Notes / caveats

• Sample-rate: the shipped open-voicefilter-lite ONNX locks the model to 16 kHz. The wrapper installs an automatic AudioResampler on the stream — pass sample_rate only if your upstream is also 16 kHz (skipping the resampler shaves a few ms of latency per frame).
• Latency: per-step latency is one hop (10 ms at 16 kHz). The OLA inside StreamingVoiceFilter keeps (L-1)·hop + n_fft ≈ 85 ms of context. The d-vector extraction at enrolment runs once.
• Effect when target is silent: voicefilter-lite's adaptive suppression smoother w collapses towards 0 in non-speech regions, so the model harmlessly returns to passthrough behaviour when nobody is talking.
• Streaming-only: the wrapper requires underlying.capabilities.streaming. For non-streaming STTs, wrap them first with livekit.agents.stt.StreamAdapter(stt=…, vad=…).
• One-shot recognition: recognize() (single-shot) is delegated verbatim — the d-vector lives on the streaming path.