Mulive

Build local-first, interactive voice and multimodal applications in Python.

Mulive is a lightweight real-time substrate that connects microphone, camera, models, application state, and web UI. Build a tiny voice interaction first, then grow it into a full product without replacing the runtime underneath.

Voice is an interface to your application—not the application itself.

Why Mulive

• Build applications, not just chatbots. Voice can operate a game, tutor, workflow, dashboard, or custom web UI.
• Keep product logic explicit. Voice, video, UI actions, and model results can be handled as typed events with deterministic state transitions.
• Run close to the product. Use direct WebRTC and local-capable STT/TTS; choose cloud models only where they add value.
• Make interactions testable. Test a single turn or a complete interaction, including interruption, cancellation, and application outcomes.

Start with a voice interaction

pip install mulive

# Browser audio + local voice stack
python -m mulive.quickstart.web --http --tts-browser

Open http://localhost:9000 and start streaming. localhost is trusted by modern browsers, so microphone access works without a certificate. The browser sends real-time media to Python and receives text and audio responses.

An audio-only browser pipeline is just a few composed stages:

audio = Stream.source(session.audio_input, name="audio")
turn = audio | turn_detector()
transcripts = turn.segments | stt(provider="faster_whisper", model_size="tiny")
user_text = transcripts | final_transcript_text()

add_text_sinks(user_text, session, role="user", subs=subs)
add_tts(user_text, session.audio_output, turn.signals, subs=subs)

Run the complete version with:

python -m mulive.quickstart.web --http --tts-browser

Choose a server setup

Use one of these public paths:

1. Standalone browser app. Run the WebRTC quickstart above for a complete browser client, signaling server, and local voice pipeline.

2. Existing FastAPI app. Mount voice into an app you already own:

   pip install "mulive[fastapi]"
   

   from fastapi import FastAPI
   from mulive import mount_voice
   
   app = FastAPI()
   mount_voice(app)
   

   Mulive serves its small browser client and its voice WebSocket transport. There is no separate WebSocket server setup to choose.

Default voice stack

Mulive includes portable local defaults: Faster-Whisper for speech recognition and Piper for speech output. They run on CPU and download their selected model weights on first use.

STT alternatives include MLX on Apple Silicon and cloud providers. TTS alternatives include in-process Kokoro ONNX (downloaded and cached on first use) or an external Kokoro-FastAPI server.

HTTPS for another device or deployment

Use HTTP only when the browser and Mulive run on the same machine. For a phone, LAN host, public hostname, or an HTTPS app embedding Mulive, serve it over HTTPS. For local device testing, create a trusted development certificate with mkcert:

mkcert -install
mkdir -p ~/.config/mulive/certs
mkcert -key-file ~/.config/mulive/certs/key.pem \
  -cert-file ~/.config/mulive/certs/cert.pem \
  localhost 127.0.0.1 ::1 <your-lan-hostname-or-ip>

python -m mulive.quickstart.web --tts-browser

For production, terminate TLS with the deployment platform or reverse proxy and set MULIVE_SSL_KEYFILE and MULIVE_SSL_CERTFILE when Mulive should serve TLS itself.

What you can build

The application model

Mulive separates the parts that benefit from models from the parts your product must control:

mic / camera / browser UI
            ↓
   streams and typed events
            ↓
  model calls + explicit app state
            ↓
 speech, UI updates, and safe effects

Use models for perception and conversation. Keep scoring, workflow state, permissions, retries, and external effects in normal application code. That is especially useful for tutoring, games, customer workflows, and any interaction where an answer must be checked before the app moves on.

Model architectures

Today, Mulive's examples use a cascaded pipeline:

audio → turn detection → STT → LLM/VLM → TTS

The same application model is intended to support direct streaming speech-to-speech models as they are added. The product state and UI should not need to change just because the voice model does.

What is next

Project structure

mulive/      Runtime, browser assets, and runnable quickstarts
mulive/core/ Real-time transport, streams, and media helpers
mulive/apps/ Application helpers
mulive/client/ Browser WebRTC client and UI
mulive/quickstart/ Small runnable applications
tests/       Pipeline, controller, transport, and application tests
docs/        Architecture and design notes

Current status

Mulive is actively evolving. The available quickstarts are useful foundations; the embed API, latency explorer, streaming-model adapters, and polished product demos are intentionally marked as planned rather than presented as shipped.