livekit-plugins-duplexjev

Audio end-of-turn detection for LiveKit Agents, powered by DuplexJev. It listens to how the user speaks, not only what they say, and tells your agent whether the user is finished, still talking, just backchanneling ("mm-hmm") or asking you to wait. The same request can also return emotion, laughter / breathing / coughs / sighs and barge-in.

• Reads the audio, not the transcript. Every answer is one constrained token of a speech LLM: nothing is decoded, so the decision does not wait for STT.
• Fast. Ten decisions about one utterance take 92 ms on one GPU, against 1,978 ms for ASR → LLM with the same LLM.
• Accurate. On the CoDeTT turn-taking benchmark (English, zero-shot), DuplexJev-32B-Turn scores 70.0, against 51.4 for Smart Turn v3. See the paper.
• More than turns, same request. Ask for emotion, non-verbal sounds or your own multiple-choice questions.
• Chinese and English. CPU only on the agent side: the plugin calls an HTTP API.

Tested with livekit-agents 1.8.6. Maintained by Adventists.ai (AI降临派).

Getting the published turn-taking numbers. LiveKit eot-bench (English, false cut-offs at 300 ms): 20.0% for DuplexJev-32B-Para v1.2 with the agent's previous message, 23.5% audio only, vs 35.2% for Smart Turn v3.2. That needs DuplexJev-32B-Para v1.2 on your own GPU behind duplexjev gateway >= 0.6 (see Run the model yourself), the default readout="turn", and the agent's previous message (below). The free trial API runs the smaller DuplexJev-4B-Para (no v1.2 turn-taking update): use it to try the plugin, not to judge turn-taking quality.

Install

pip install livekit-plugins-duplexjev

Use

from livekit.agents import AgentSession
from livekit.plugins import duplexjev, silero

session = AgentSession(
    stt=..., llm=..., tts=...,
    vad=silero.VAD.load(),
    turn_handling={"turn_detection": duplexjev.TurnDetector(
        api_url="http://localhost:8420",       # your own gateway; omit for the free hosted trial API
        extra_questions=["emotion", "sound"],
        on_answers=lambda a: print({k: v["answer"] for k, v in a.items()}),
    )},
)

Give it the agent's last message (new in 0.2.0). Turn-taking is clearly better when the model knows what the agent had just said. Keep a reference to the detector and attach it to the session once:

detector = duplexjev.TurnDetector(api_url="http://localhost:8420")
session = AgentSession(..., turn_handling={"turn_detection": detector})
detector.attach(session)        # follows conversation_item_added; or detector.set_assistant_said(text)

It needs a gateway from duplexjev>=0.6. On LiveKit eot-bench (English, DuplexJev-32B-Para v1.2, full run) this cuts false cut-offs at a 300 ms budget from 23.5% to 20.0%.

AgentSession(turn_detection=duplexjev.TurnDetector(), ...) works too (LiveKit marks that argument as deprecated). A complete agent is in examples/agent.py.

A VAD is required. LiveKit ignores a streaming turn detector when the session has no VAD. The plugin buffers the user's audio, and after about 200 ms of VAD silence LiveKit asks it for a prediction: the last 8 s of the current user turn go to DuplexJev, and P(finished) becomes LiveKit's end_of_turn_probability. Below unlikely_threshold the session waits max_delay instead of min_delay before replying. The VAD's min_silence_duration must be at least 0.25 s (Silero's default is 0.55 s).

STT is optional for the decision. If the session has an STT, LiveKit still waits for its transcript before it commits the turn, as with any turn detector, because the LLM needs the text.

Run the model yourself (recommended for real-time)

The default URL is our free trial API (rate limited to 30 requests per minute; the round trip from outside mainland China adds network latency; it runs the smaller DuplexJev-4B-Para without the v1.2 turn-taking update). For a live agent, serve the model next to it. For turn-taking, use DuplexJev-32B-Para v1.2 on one 80 GB GPU (the eot-bench numbers above):

pip install "vllm[audio]>=0.29" duplexjev-vllm "duplexjev[server]>=0.6"
vllm serve adventists-ai/DuplexJev-32B-Para --revision v1.2 --max-model-len 4096 --port 8000 &
duplexjev gateway --vllm http://localhost:8000/v1 --port 8420 --rate 1000000

On a smaller GPU, adventists-ai/DuplexJev-4B-Para needs about 10 GB; it is fine for emotion and the other questions, but its turn-taking has not had the v1.2 update. --rate 1000000 lifts the gateway's default limit of 30 requests per minute, which a live agent exceeds quickly.

Then pass api_url="http://localhost:8420" (or set DUPLEXJEV_API_URL).

Options

After each prediction detector.last_turn_class is finished, not_finished, backchannel or hesitating (reply, keep_listening, acknowledge with readout="act"), for your own routing.

License

The plugin code is BSD-2-Clause. DuplexJev model weights are CC BY-NC 4.0 (non-commercial); see the model cards. For commercial use, contact jiejin@adventists.ai.