smart-turn-livekit

LiveKit Agents plugin for Smart Turn end-of-turn detection.

A LiveKit Agents plugin that runs Smart Turn models to detect when a user has finished speaking based on audio prosody. It handles the mel front-end, the ONNX session, resampling, per-model language gating, and the threading, so any checkpoint plugs into an AgentSession in one line.

Works with upstream's smart-turn-v3, a fine-tune, or your own ONNX file. CPU-only, no PyTorch.

pip install 'smart-turn-livekit[livekit]'

Quickstart

from livekit.agents import AgentSession
from livekit.plugins import silero
from smart_turn_livekit import SmartTurnDetector

session = AgentSession(
    vad=silero.VAD.load(min_silence_duration=0.25),   # must be >= 0.25
    turn_handling={
        "turn_detection": SmartTurnDetector(),         # smart-turn-v3
        "endpointing": {"min_delay": 0.3, "max_delay": 2.5},
    },
)

Weights download from Hugging Face on first use and are cached. Nothing to configure, nothing bundled in the wheel.

Inspect each decision with on_prediction:

SmartTurnDetector(
    on_prediction=lambda p: print(
        f"end of turn {p.probability:.1%} -> "
        f"{'complete' if p.complete else 'incomplete'} ({p.inference_s * 1000:.0f} ms)"
    )
)

end of turn 1.2%  -> incomplete (55 ms)
end of turn 94.5% -> complete   (58 ms)

Models

SmartTurnDetector()                                  # smart-turn-v3, default
SmartTurnDetector(model="smart-turn-tamil-tiny")     # a fine-tune
SmartTurnDetector(model_path="/opt/my-model.onnx")   # your own

key	languages	size	accuracy
smart-turn-v3 (default)	multi	8.7 MB	92.63% on upstream's benchmark
smart-turn-v3-fp32	multi	32 MB	unquantised reference graph

Language-specific fine-tunes

key	language	size	accuracy
smart-turn-tamil-tiny	ta	8.7 MB	83.71% on real Tamil telephone speech
smart-turn-tamil-base	ta	21 MB	86.13% on the same

MODEL_REGISTRY is the only place that knows one model from another, so adding a checkpoint is a dict entry. Anything not listed still runs via model_path=.

Latency

threads	p50	p95
1	84.5 ms	91.9 ms
2	53.1 ms	61.7 ms

smart-turn-v3, batch 1, inference only, idle Intel i5-12450H. Measure on your own box — python -m smart_turn_livekit.bench. This number lands directly on the gap between the user finishing and the agent replying, so budget it against min_delay. The bench refuses to print a table without warning you when machine load is high, because a benchmark taken under load once produced a wrong shipping recommendation here.

Gotchas

• min_silence_duration >= 0.25. LiveKit will not request a prediction below MIN_SILENCE_DURATION_MS + 50 ms and raises at session start if the VAD is faster. A floor, not a ceiling — it fails loudly.
• The language must be one the model claims, or unknown. supports_language() returns False otherwise, using the languages the model was benchmarked on. This one fails silently — the detector is skipped and the agent falls back to fixed timing. Override with languages=(...).

Threshold

Under LiveKit the threshold picks how long to wait, not whether to end the turn: below it the session takes max_delay instead of min_delay. So a false complete costs a shorter pause rather than an interruption — a softer knob than the same number under Pipecat.

SmartTurnDetector(threshold=0.72)
SmartTurnDetector(operating_point="polite")   # if the model ships one

0.5 is the default threshold. Some language-specific fine-tunes offer predefined operating points (like "polite" or "balanced") that are calibrated on held-out evaluation sets to provide a different tradeoff between interruption rate and latency.

Outside LiveKit

The detector is a thin adapter over a plain class you can call directly:

from smart_turn_livekit import SmartTurn

st = SmartTurn()                  # smart-turn-v3
st.probability(wave_16k)          # P(speaker has finished), 0.0 .. 1.0
st.is_complete(wave_16k)          # bool, against st.threshold

16 kHz mono float, last 8 seconds only. Other sample rates need soxr.

Pipecat needs no adapter — its built-in analyzer takes any Smart Turn ONNX, and this package can still resolve the file for it:

from smart_turn_livekit import resolve_model
from pipecat.audio.turn.smart_turn.local_smart_turn_v3 import LocalSmartTurnAnalyzerV3

analyzer = LocalSmartTurnAnalyzerV3(smart_turn_model_path=str(resolve_model()))

Pinned or air-gapped:

SmartTurnDetector(model_path="/opt/models/smart-turn.onnx")
SmartTurnDetector(local_only=True)                  # cached weights only

or SMART_TURN_MODEL=/path/to.onnx, or HF_HUB_OFFLINE=1. An explicit model_path claims no language until you pass languages=(...).

How it works

LiveKit exposes two turn-detector protocols — one reads text, one reads audio — as plain typing.Protocol classes with no base class to import. This package implements the audio one structurally, so it's accepted by duck typing alone; there is nothing to register.

One ONNX code path runs every registered model, because every Smart Turn checkpoint shares the same signature: input_features (batch, 80, 800) in, a logits output that is already a sigmoid out. Applying a second sigmoid to that output gives confident, plausible, wrong answers, so SmartTurn.probability checks the output is in [0, 1] and raises rather than return a silently-broken score.

Inference runs in a single-thread executor so a ~10–30 ms ONNX call never stalls the event loop. Construction resolves LiveKit's event class and runs one dummy inference up front (warmup=True, default) — skipping that would put a ~1.7 s import penalty on the first prediction of the first call, a cost LiveKit's own CLI normally hides by paying it earlier.

The mel front-end matches WhisperFeatureExtractor(chunk_length=8) and is checked against Pipecat's own extractor to <1e-3.

Tests

pip install -e '.[dev,livekit]'
pytest                                  # 22 tests; 3 skip without the research corpus
python -m smart_turn_livekit.bench

Checks features against Pipecat's extractor, the LiveKit Protocols by isinstance, and that every registered model loads and returns a probability — including upstream's, which we do not control.

Limitations

• Not a VAD. It answers "did they finish?", not "is anyone talking?" You still need Silero upstream.
• Audio-only, no language token. Nothing stops any of these models running on any language — supports_language() reflects what was measured, not what is possible.
• Written against livekit-agents 1.7.0. TurnDetectionEvent and both turn detector Protocols are private (_-prefixed) there, so treat a minor-version bump as something to re-check. The test suite fails loudly if the shape moves.

Author

Built and maintained by Santhosh (@santhosh-005).

Licence

This package is BSD-2-Clause. It bundles Whisper's mel filterbank (MIT, OpenAI).

The weights are downloaded, not distributed here, and licensed separately. smart-turn-v3 is BSD-2-Clause by Daily.