miragea-dub: multi-speaker AI video dubbing, open source

miragea-dub is an open-source command-line tool, MCP server and Agent Skill that dubs a video into another language while keeping a separate cloned voice for every speaker. It transcribes the video, works out who speaks when, lets you (or your AI agent) translate each line, clones every speaker's voice from the video itself with the Miragea voice API, fits each translated line into the original timing, ducks the original audio underneath, and exports a dubbed MP4, a stand-alone audio track for YouTube multi-language audio, and SRT/VTT subtitles. MIT licensed, Python 3.10+, works offline in demo mode.

Demo: a real two-speaker dub

An English clip with two speakers (Ana and Ben) dubbed into Spanish. Both voices were cloned from the clip itself, nothing was pre-recorded:

• Watch the dubbed MP4 (Spanish, burned-in subtitles, original English kept as 2nd audio track)
• Original English audio
• Spanish subtitles and timing report

Round-trip check: transcribing the Spanish track again returns the translated script almost word for word, and the diarizer assigned all 6 lines to the right speaker.

Quick start

pipx install miragea-dub          # or: uvx miragea-dub ...   (needs ffmpeg on PATH)
miragea-dub demo                  # offline end-to-end run with test tones, no key needed
export MIRAGEA_API_KEY=...        # create a key at https://miragea.ai/workspace/api
miragea-dub doctor                # checks ffmpeg, key, connectivity

# 1) transcribe + find speakers -> dub.json
miragea-dub prepare interview.mp4 --to es --speakers 2
# 2) translate: let your AI agent fill "translation" in dub.json, or use any OpenAI-compatible LLM
miragea-dub translate interview.dub/dub.json --translator openai
# 3) voice, fit, mix, export
miragea-dub render interview.dub/dub.json --model expressive
# or all in one:  miragea-dub dub interview.mp4 --to es --translator openai

Outputs next to the video: interview.es.mp4 (dub track + original track + subtitles), interview.es.m4a, .srt, .vtt and interview.es.report.json.

How it works

1. Transcribe: Miragea speech-to-text (no GPU), local faster-whisper, or your own SRT/VTT/JSON. Miragea returns subtitle cues cut at a fixed width, so cues are regrouped into whole sentences using the punctuated full text.
2. Diarize: a built-in, GPU-free speaker clustering on voice features (or pyannote, or an RTTM file). Speaker tags already in your transcript ([Ana] ..., S2: ...) are kept.
3. Translate: every line gets a length budget so the translation can be spoken inside the original time slot. Backends: agent (your AI agent edits dub.json), openai (any OpenAI-compatible endpoint), file, copy.
4. Clone and voice: an 8-20 second reference clip is cut per speaker from the video, hosted on a temporary public link, and each line is synthesized with that voice through the Miragea API. Requests are idempotent and cached, so a re-run never pays twice.
5. Fit and mix: lines are sped up (up to 1.35x by default) to fit, the original audio is ducked under speech (or removed, or replaced by your music stem, or separated with Demucs), then everything is muxed.

Real timing report

Lines that could not fit even at the speed limit are listed in lines_to_shorten; shorten them in dub.json and run render again (only changed lines are re-voiced).

Use it from an AI agent

MCP server (Claude Desktop, Claude Code, Cursor, VS Code, Windsurf):

{ "mcpServers": { "miragea-dub": { "command": "uvx", "args": ["--from", "miragea-dub", "miragea-dub-mcp"],
  "env": { "MIRAGEA_API_KEY": "your-key" } } } }

Tools: prepare_dub, save_translations, show_project, estimate_dub, render_dub, list_voices. The agent is the translator, so no second LLM key is needed.

Agent Skill: install video-dubbing from miragea-skills, or tell your agent: Install https://miragea.ai/skills/miragea-audio/install.md.

How it compares

Star counts at time of writing: VideoLingo 19.1k, pyvideotrans 19.7k. Those projects are bigger and have GUIs; miragea-dub is the small, scriptable, agent-friendly option.

FAQ

How do I dub a video into another language with AI? Install miragea-dub, run miragea-dub dub video.mp4 --to es --translator openai, and you get a dubbed MP4 plus subtitles.

Can I dub a video with two or more speakers and keep each voice? Yes. Speakers are separated by diarization (or by tags in your own transcript) and each one is cloned from their own lines.

Does it work for a YouTube video with multiple audio tracks? The .m4a output is the stand-alone dubbed track you can upload as an additional language audio track; the .mp4 already carries dub and original tracks.

Is there a free open-source alternative to ElevenLabs or HeyGen dubbing? The pipeline, diarization, timing fit and mixing are free and MIT; the voices come from the Miragea API, which needs a key.

Can an AI agent (Claude, Cursor, Codex) dub videos? Yes, through the MCP server or the video-dubbing skill. The agent writes the translations itself.

How do I keep the dubbed speech from running longer than the original? Every line gets a word or character budget before translation, long lines are sped up to the limit, and what still does not fit is reported for shortening.

Does it keep background music? By default the original audio is ducked under speech. Use --background file with a music stem, demucs to separate stems, or remove.

Which languages? Anything your translator and the Miragea voices handle; the repo's own example is English to Spanish.

Troubleshooting

• litterbox upload failed (HTTP 403): the default --ref-host auto already falls back to other free hosts; or use --ref-host "cmd:your-upload-command {path}", or prefix: with --ref-dir for your own server.
• Wrong speaker count: pass --speakers N, or supply --transcript with [Name] tags.
• A line sounds rushed: shorten its translation in dub.json and run render again.
• 401: create a key at https://miragea.ai/workspace/api and export MIRAGEA_API_KEY.

Development

uv venv && uv pip install -e ".[dev]"
python -m pytest        # 28 tests, offline: Miragea API is mocked with httpx.MockTransport
python docs/make_images.py

Base URL defaults to https://miragea.ai/v1 (override with MIRAGEA_BASE_URL). See CHANGELOG and llms.txt. MIT, see LICENSE.