smart-turn-livekit

LiveKit Agents plugin for Smart Turn end-of-turn detection.

A LiveKit Agents plugin that runs Smart Turn models to detect when a user has finished speaking based on audio prosody. It handles the mel front-end, the ONNX session, resampling, per-model language gating, and the threading, so any checkpoint plugs into an AgentSession in one line.

Works with upstream's smart-turn-v3, a fine-tune, or your own ONNX file. Fully local — CPU-only ONNX inference, no PyTorch, no inference service, and no network call on the hot path. See Offline and air-gapped.

pip install 'smart-turn-livekit[livekit]'

Quickstart

from livekit.agents import AgentSession
from livekit.plugins import silero
from smart_turn_livekit import SmartTurnDetector

session = AgentSession(
    vad=silero.VAD.load(min_silence_duration=0.25),   # must be >= 0.25
    turn_handling={
        "turn_detection": SmartTurnDetector(),         # smart-turn-v3
        "endpointing": {"min_delay": 0.3, "max_delay": 2.5},
    },
)

Weights download from Hugging Face on first use and are cached. Nothing to configure, nothing bundled in the wheel.

Inspect each decision with on_prediction:

SmartTurnDetector(
    on_prediction=lambda p: print(
        f"end of turn {p.probability:.1%} -> "
        f"{'complete' if p.complete else 'incomplete'} ({p.inference_s * 1000:.0f} ms)"
    )
)

end of turn 1.2%  -> incomplete (55 ms)
end of turn 94.5% -> complete   (58 ms)

Models

SmartTurnDetector()                                  # smart-turn-v3, default
SmartTurnDetector(model="smart-turn-tamil-tiny")     # a fine-tune
SmartTurnDetector(model_path="/opt/my-model.onnx")   # your own

key	languages	size	accuracy
smart-turn-v3 (default)	multi	8.7 MB	92.63% on upstream's benchmark
smart-turn-v3-fp32	multi	32 MB	unquantised reference graph

Language-specific fine-tunes

key	language	size	accuracy
smart-turn-tamil-tiny	ta	8.7 MB	83.71% on real Tamil telephone speech
smart-turn-tamil-base	ta	21 MB	86.13% on the same

MODEL_REGISTRY is the only place that knows one model from another, so adding a checkpoint is a dict entry. Anything not listed still runs via model_path=.

Latency

threads	p50	p95
1	84.5 ms	91.9 ms
2	53.1 ms	61.7 ms

smart-turn-v3, batch 1, inference only, idle Intel i5-12450H. Measure on your own box — python -m smart_turn_livekit.bench. This number lands directly on the gap between the user finishing and the agent replying, so budget it against min_delay. The bench refuses to print a table without warning you when machine load is high, because a benchmark taken under load once produced a wrong shipping recommendation here.

Gotchas

• min_silence_duration >= 0.25. LiveKit will not request a prediction below MIN_SILENCE_DURATION_MS + 50 ms and raises at session start if the VAD is faster. A floor, not a ceiling — it fails loudly.
• The language must be one the model claims, or unknown. supports_language() returns False otherwise, using the languages the model was benchmarked on. This one fails silently — the detector is skipped and the agent falls back to fixed timing. Override with languages=(...).

Threshold

Under LiveKit the threshold picks how long to wait, not whether to end the turn: below it the session takes max_delay instead of min_delay. So a false complete costs a shorter pause rather than an interruption — a softer knob than the same number under Pipecat.

SmartTurnDetector(threshold=0.72)                       # one value
SmartTurnDetector(threshold={"en": 0.6, "ta": 0.85})    # per language
SmartTurnDetector(operating_point="polite")             # if the model ships one

0.5 is the default threshold. Some language-specific fine-tunes offer predefined operating points (like "polite" or "balanced") that are calibrated on held-out evaluation sets to provide a different tradeoff between interruption rate and latency.

A dict is an override table, resolved on exact code, then base language, then the operating point — so {"en-US": 0.6} answers for en-US only, while {"en": 0.6} answers for en, en-US and en-GB alike.

Outside LiveKit

The detector is a thin adapter over a plain class you can call directly:

from smart_turn_livekit import SmartTurn

st = SmartTurn()                  # smart-turn-v3
st.probability(wave_16k)          # P(speaker has finished), 0.0 .. 1.0
st.is_complete(wave_16k)          # bool, against st.threshold

16 kHz mono float, last 8 seconds only. Other sample rates need soxr.

Pipecat needs no adapter — its built-in analyzer takes any Smart Turn ONNX, and this package can still resolve the file for it:

from smart_turn_livekit import resolve_model
from pipecat.audio.turn.smart_turn.local_smart_turn_v3 import LocalSmartTurnAnalyzerV3

analyzer = LocalSmartTurnAnalyzerV3(smart_turn_model_path=str(resolve_model()))

Offline and air-gapped

Inference is entirely local: ONNX Runtime on CPU, in your own process. There is no inference service, no API key, and no network call on the prediction path.

The one network access is fetching the weights, which happens once on first use and is then cached. Any of these removes it:

SmartTurnDetector(model_path="/opt/models/smart-turn.onnx")   # ship the file yourself
SmartTurnDetector(local_only=True)                            # cached weights only, fail otherwise

or SMART_TURN_MODEL=/path/to.onnx, or HF_HUB_OFFLINE=1. Pre-cache the weights in your image build and the running agent never reaches the network — which is what a strict data-residency or air-gapped deployment needs.

An explicit model_path claims no language until you pass languages=(...).

How it works

LiveKit exposes two turn-detector protocols — one reads text, one reads audio — as plain typing.Protocol classes with no base class to import. This package implements the audio one structurally, so it's accepted by duck typing alone; there is nothing to register.

One ONNX code path runs every registered model, because every Smart Turn checkpoint shares the same signature: input_features (batch, 80, 800) in, a logits output that is already a sigmoid out. Applying a second sigmoid to that output gives confident, plausible, wrong answers, so SmartTurn.probability checks the output is in [0, 1] and raises rather than return a silently-broken score.

Inference runs in a single-thread executor so a ~10–30 ms ONNX call never stalls the event loop. Construction resolves LiveKit's event class and runs one dummy inference up front (warmup=True, default) — skipping that would put a ~1.7 s import penalty on the first prediction of the first call, a cost LiveKit's own CLI normally hides by paying it earlier.

The mel front-end matches WhisperFeatureExtractor(chunk_length=8) and is checked against Pipecat's own extractor to <1e-3.

Tests

pip install -e '.[dev,livekit]'
pytest                                  # 26 tests; 3 skip without the research corpus
python -m smart_turn_livekit.bench

Checks features against Pipecat's extractor, the LiveKit Protocols by isinstance, and that every registered model loads and returns a probability — including upstream's, which we do not control.

Limitations

• Not a VAD. It answers "did they finish?", not "is anyone talking?" You still need Silero upstream.
• Audio-only, no language token. Nothing stops any of these models running on any language — supports_language() reflects what was measured, not what is possible.
• Written against livekit-agents 1.7.0. TurnDetectionEvent and both turn detector Protocols are private (_-prefixed) there, so treat a minor-version bump as something to re-check. The test suite fails loudly if the shape moves.

Author

Built and maintained by Santhosh (@santhosh-005).

Licence

This package is BSD-2-Clause. It bundles Whisper's mel filterbank (MIT, OpenAI).

The weights are downloaded, not distributed here, and licensed separately. smart-turn-v3 is BSD-2-Clause by Daily.