livekit-plugins-rumik-ai

Rumik AI text-to-speech plugin for LiveKit Agents.

Streams low-latency 24 kHz speech from Rumik's Silk models over a reusable WebSocket session:

• muga — emotion-controlled via a leading [tone] tag (e.g. [happy], [sad]) plus optional <laugh>/<chuckle>/<sigh> events. Tuned for Romanized Hinglish.
• mulberry — steered by a natural-language voice description or a preset speaker, with optional pitch shift (f0_up_key).

Install

pip install livekit-plugins-rumik-ai

This depends on livekit-agents (1.5+). Set your key:

export RUMIK_API_KEY="your-rumik-api-key"

Quickstart

from livekit.agents import AgentSession
from livekit.plugins import rumik_ai

# muga: the LLM should start each reply with one tone tag, e.g. "[happy] ..."
session = AgentSession(
    stt=...,
    llm=...,
    tts=rumik_ai.TTS(model="muga"),
)

Mulberry, steered by a voice description (or a preset speaker):

tts = rumik_ai.TTS(
    model="mulberry",
    description="warm, gentle female friend",
    # speaker="ira",            # optional preset voice, overrides description
    # f0_up_key=2.0,            # optional pitch shift, -12..12 semitones
)

Changing the voice at runtime

description, speaker, f0_up_key, and the sampling params are sent on every request, so you can change mulberry's voice between turns without reconnecting — the pooled WebSocket is reused (only a model change re-mints the session):

tts.update_options(description="excited young man, fast and energetic")
# the next synthesis request uses the new voice

Latency vs. smoothness

The default is model-aware:

• muga buffers the full LLM reply and synthesizes it in one request, so its leading [tone] tag conditions the whole utterance (and there are no per-request TTFB gaps).
• mulberry streams sentence-by-sentence for lower time-to-first-word, since it has no tone tag to protect.

Override either with full_response_aggregation:

rumik_ai.TTS(model="muga", full_response_aggregation=False, tone="neutral")  # muga, lower latency
rumik_ai.TTS(model="mulberry", full_response_aggregation=True)               # mulberry, smoother

When you turn aggregation off for muga, set a fallback tone= so every sentence keeps a tone tag.

Barge-in & cancel

Built for the live "call with AI" case. When the caller talks over the agent, LiveKit interrupts the TTS and the plugin sends an explicit cancel to Rumik, so the in-flight generation stops immediately (and billing is finalized cleanly). The pooled WebSocket is kept warm across the interruption, so the next utterance doesn't pay a reconnect.

Configuration

Examples

See examples/ for a full voice agent (rumik_ai_agent.py) and a record-to-WAV demo (rumik_ai_tts.py).

License

Apache-2.0