livekit-plugins-speakerbeamss

A LiveKit Agents STT plugin that runs OpenSpeakerBeam-SS target-speaker extraction in front of any inner STT instance. The agent hears only the user — even with competing speakers, music, or noise in the room.

The plugin is ONNX-only. The PyTorch separator never ships here. The trained INT8 model is bundled inside the wheel at livekit/plugins/speakerbeamss/models/speakerbeam_ss_trained_int8.onnx (≈13 MB), so a fresh pip install livekit-plugins-speakerbeamss is enough — no separate download. Follow step 0 below only if you want to swap in a different export (FP32, the base release, or your own fine-tune).

The reference audio for the target speaker is taken from the agent's first user turn: the first sentence the user speaks is forwarded to the inner STT as-is so the agent can still understand it, and is also kept in a buffer. When the inner STT emits END_OF_SPEECH, the buffer is collapsed to a Resemblyzer d-vector and the plugin flips to extraction mode for every subsequent utterance.

You can also pass a pre-recorded enrollment audio via reference_audio="path.wav" to skip the capture phase entirely.

Installation

# 1. Install this plugin (from the plugin source root):
cd livekit/plugins/speakerbeamss
pip install -e .

That's it — pip install wires up livekit.plugins.speakerbeamss and ships the trained ONNX bundle, which the loader resolves via importlib.resources. No env vars, no separate download.

Using a different exported variant

If you have your own export (FP32, base release, a re-trained checkpoint, …) you'd rather use, point the loader at it explicitly:

from livekit.plugins.speakerbeamss import STT

sb_stt = STT(
    inner=inner,
    onnx_dir="/Users/linsan/Public/projects/OpenSpeakerBeam-SS/exports",
    onnx_filename="speakerbeam_ss_int8.onnx",   # base INT8 release
)

The loader's lookup order is:

1. Bundled — livekit.plugins.speakerbeamss.models/<​filename> (the trained INT8 lives here by default).
2. Disk — {onnx_dir}/{filename} if set, otherwise the ~/Public/projects/OpenSpeakerBeam-SS/exports/ fallback.

To export a different variant yourself:

cd ~/Public/projects/OpenSpeakerBeam-SS
pip install -r requirements.txt
python export_onnx.py --ckpt checkpoints/best_model.pth
ls exports/   # -> speakerbeam_ss_{fp32,int8,trained_fp32,trained_int8}.onnx

Usage — wrap any inner STT

from livekit.plugins.speakerbeamss import STT
from livekit.plugins.volcengine import STT as VolcSTT  # any streaming inner STT

inner = VolcSTT()                          # read keys from env, etc.
stt = STT(inner=inner)
# ... pass `stt` to your VoicePipelineAgent as usual.

That's it. The agent will see exactly the transcripts it would have seen without the wrapper, but with the first turn's enrollment frozen into a d-vector and applied to every chunk that follows.

Usage — offline

import numpy as np
from livekit.plugins.speakerbeamss import OnnxSpeakerBeamExtractor

extractor = OnnxSpeakerBeamExtractor()                     # INT8 by default
extractor.enroll(np.random.randn(16000).astype(np.float32))  # reference audio

chunk = np.random.randn(8000).astype(np.float32)             # < 1 s
print(extractor.append(chunk))    # []  (no full chunk yet)
print(extractor.append(np.zeros(8000, dtype=np.float32)))    # [ndarray of len 16000]

How the state machine works

Configuration knobs

STT(...) arguments:

Environment variables:

Verified against

• LiveKit Agents >= 1.5.13
• OpenSpeakerBeam-SS commit exporting speakerbeam_ss_trained_int8.onnx (~13 MB) at opset 20, chunk length 16 000 samples / 16 kHz — the bundle that ships inside the wheel.

License

Apache-2.0 (same as LiveKit Agents).