vuesub 0.5.0

Whisper subtitles + CosyVoice 3 voice cloning, from the command line.

Vuesub

Audio-to-subtitle desktop app powered by Whisper. Tauri shell, React frontend, Python (faster-whisper + OpenAI SDK) backend running as a sidecar. Targets macOS, Windows, and Linux.

A vuesub-cli command ships alongside the app for headless use and AI agents — see Command-line use and AI agents.

┌──────────────┐   Tauri commands    ┌──────────────┐  line-JSON over   ┌───────────────────────┐
│ React + Vite │ ─────────────────►  │  Rust shell  │ ─────────────────►│  Python sidecar       │
│  (renderer)  │ ◄─── Tauri events ──│  (tauri-app) │ ◄─── stdout ──────│  faster-whisper       │
└──────────────┘                     └──────────────┘                   │  + OpenAI cloud path  │
                                                                        └───────────────────────┘

Supported audio / video files

Extension	Format	Type	Common source
.mp3	MPEG-1 Audio Layer III	Audio-only	Podcasts, downloads
.wav	Waveform Audio File	Audio-only	Recording sessions, lossless masters
.m4a	MPEG-4 Audio (AAC/ALAC)	Audio-only	Apple Voice Memos, Music exports
.flac	Free Lossless Audio Codec	Audio-only	Lossless archives
.ogg	Ogg Vorbis / Opus	Audio-only	Open-source apps, web
.mp4	MPEG-4 Part 14	Audio+video	Most cameras, Zoom recordings
.mov	QuickTime Movie	Audio+video	macOS screen recordings, iPhone video
.mkv	Matroska	Audio+video	Anime, archival, screen captures
.webm	WebM (Opus, AV1)	Audio+video	Browser-recorded video, OBS

For video files, Vuesub extracts the audio track via the underlying decoders (Web Audio API for the in-browser waveform, faster-whisper / PyAV for transcription) and ignores the video. WebKit on macOS doesn't natively play WebM; Vuesub transparently falls back to a Web Audio AudioBufferSourceNode path that uses the buffer it already decoded for the waveform.

Prerequisites

• Node.js 20+ and npm
• Rust (stable). Install via rustup.
• Python 3.10–3.12
• For end users on the Voice Clone feature: Python 3.10–3.12, uv, and git must be on PATH. They're not bundled — Vuesub stands up a CosyVoice runtime in ~/.vuesub/runtimes/cosyvoice/ on first model download (~6.5 GB, one-time). The Voice Clone page detects missing tools and shows install instructions inline.
• For end users on the TTS page (Qwen3-TTS): Python 3.10–3.12, uv, and SoX on PATH (qwen-tts shells out to sox for audio I/O). Vuesub stands up a Qwen3-TTS runtime in ~/.vuesub/runtimes/qwen3tts/ on first model download (~3.5 GB per variant: CustomVoice for preset voices, Base for zero-shot voice clone). No git required; the page is offline-capable once installed.
• Platform toolchain for Tauri:
  • macOS: Xcode Command Line Tools (xcode-select --install)
  • Windows: Microsoft C++ Build Tools + WebView2 (preinstalled on Win11)
  • Linux: webkit2gtk-4.1, libayatana-appindicator3, librsvg2, build-essential, curl, wget, file, pkg-config, libssl-dev, libxdo-dev, patchelf — see Tauri prerequisites.

First-time setup

# 1. Install JS deps
npm install

# 2. Create the Python venv and install backend deps
#    (faster-whisper, av, huggingface_hub, openai, pyinstaller)
npm run py:install

# 3. Create the dev sidecar shim so `tauri dev` can spawn the Python backend
npm run py:dev          # macOS / Linux
# Windows: run `python python/build.py` once to produce
# vuesub-backend-x86_64-pc-windows-msvc.exe

Develop

npm run tauri:dev

Vite serves the renderer, Tauri spawns the window, and the Python sidecar runs from your venv via the dev shim. Edits to React / TS hot-reload. Edits to Python require restarting tauri dev.

To talk to the backend directly without the GUI:

# in one terminal:
echo '{"id":1,"method":"ping","params":{}}' | python/.venv/bin/python python/main.py
echo '{"id":1,"method":"list_models","params":{}}' | python/.venv/bin/python python/main.py

Build a release app

You need a real, statically-bundled Python binary (not the dev shim) for the release bundle. PyInstaller produces one binary per host triple.

# Build the Python sidecar binary for the current host
npm run py:bundle

# Then build the Tauri app (.dmg, .msi/.exe, .deb)
npm run tauri:build

Artifacts land under src-tauri/target/release/bundle/. The CI workflow ships only the bundle types we actually distribute (mac DMG + .app, Windows NSIS .exe, Linux .deb) — see .github/workflows/release.yml.

Cross-platform bundling

PyInstaller can only target the OS it runs on. To ship all three platforms, build the sidecar (and ideally the whole Tauri app) on each:

Platform	Sidecar name produced	Bundle output
macOS	vuesub-backend-aarch64-apple-darwin (or x86_64)	.dmg, .app.tar.gz
Windows	vuesub-backend-x86_64-pc-windows-msvc.exe	NSIS .exe
Linux	vuesub-backend-x86_64-unknown-linux-gnu	.deb

CI runs three jobs (one per OS) that each call npm run py:install && npm run py:bundle && npm run tauri:build -- --bundles <list>, then collect the bundle artifacts and upload them to a release on the public Pages repo. Triggered by pushing a vX.Y.Z tag.

Project layout

vuesub/
├── src/                    React frontend (Vite + TS)
│   ├── App.tsx             Main UI: file picking, transcribe, transcript editor
│   ├── Settings.tsx        Settings dialog: model manager + OpenAI API key + font size + locale
│   ├── Timeline.tsx        Waveform timeline with zoom + Web Audio fallback
│   ├── VoiceClone.tsx      Voice Clone view (speaker registry, synthesis, history)
│   ├── Dropdown.tsx        Custom ChipSelect / MenuSelect (replaces native <select>)
│   ├── font-size.ts        Global font-scale provider (Small / Medium / Large / XL)
│   ├── languages.ts        Language list + persistence
│   ├── opencc.d.ts         Type shim for opencc-js
│   ├── segments.ts         Pure helpers for segment edits
│   ├── main.tsx            React root
│   └── styles.css
├── src-tauri/              Rust shell
│   ├── src/main.rs         Entry point
│   ├── src/lib.rs          Tauri command surface
│   ├── src/backend.rs      Sidecar IPC (line-JSON over stdio)
│   ├── tauri.conf.json     Bundle config
│   ├── capabilities/       Permission manifest
│   ├── icons/              App icons (regenerate via scripts/generate_icons.py)
│   └── binaries/           Sidecar binaries land here, named with target triple
├── python/                 Python backend
│   ├── main.py             JSON-RPC dispatcher over stdio
│   ├── transcribe.py       Whisper wrapper, model manager, OpenAI cloud path
│   ├── build.py            PyInstaller driver
│   └── requirements.txt    faster-whisper, av, huggingface_hub, openai
├── scripts/
│   ├── dev-sidecar.sh      Writes a dev shim into src-tauri/binaries/
│   └── generate_icons.py   Creates placeholder app icons
├── .publish-site/          Static GitHub Pages landing site (synced to Vuesub-Pro)
├── .github/workflows/
│   └── release.yml         Tag-triggered build matrix → public release
└── package.json

Command-line use

Every feature in the desktop app is also reachable from the shell via vuesub-cli — transcribe, manage Whisper models, register speakers, and synthesize voice clones.

vuesub-cli transcribe video.mp4 --output srt --out video.srt
vuesub-cli speakers add alice.wav --name "Alice" --json
vuesub-cli clone alice --text "Hello from the CLI" --out hello.wav

Run vuesub-cli --help for the full command surface. The complete contract (JSON output shapes, exit codes, speaker resolution rules) lives in AGENTS.md.

Install

Platform	Command	What you get
macOS (Apple Silicon)	brew install --cask opcify/tap/vuesub	Vuesub.app + vuesub-cli symlinked into $HOMEBREW_PREFIX/bin/ automatically.
macOS (Intel / manual)	Download .dmg from Releases	Run the app once, then Settings → General → "Install vuesub-cli" writes a shim to ~/.local/bin/vuesub-cli.
Linux (Debian / Ubuntu)	curl -LO …Vuesub-linux-x64.deb && sudo apt install ./Vuesub-linux-x64.deb	The .deb's postinst symlinks /usr/bin/vuesub-cli to the bundled sidecar automatically.
Windows	Run the NSIS .exe installer	The install directory is added to PATH automatically; vuesub-cli.cmd lives next to the main exe.
Headless (any OS)	pip install vuesub	CLI only, no desktop app. Faster-whisper installs at pip-install time; CosyVoice's 6 GB runtime lazy-installs on first clone.

Note: The Homebrew tap (opcify/homebrew-tap) is provisioned on the first release after the tap repo exists — see RELEASING.md. Until then, use the manual .dmg path on macOS.

The pip package and the desktop-bundled binary share the same Python code and accept the same commands.

AI agents

Drop AGENTS.md into the root of any project you want to give Vuesub access to. Most modern AI coding agents read it automatically:

Tool	What to do
Claude Code	Copy skills/vuesub/SKILL.md into ~/.claude/skills/vuesub/ (or commit it under .claude/skills/). The skill activates on captioning / voice-clone requests.
Cursor	AGENTS.md is read automatically — no extra step.
Codex CLI	Point at AGENTS.md (or place it in the project root — Codex picks it up).
OpenClaw	Place AGENTS.md in the project root.
OpenCode	Same — AGENTS.md in the project root.

The skill documents the exact CLI surface, JSON result shapes, speaker addressing rules, and the "auto-dub a transcript in a cloned voice" workflow that's the headline use case.

Backend protocol

The Rust shell talks to the Python sidecar with one JSON object per line on stdin / stdout. Topic names use a <scope>:<event> shape so multiple unrelated event streams can share the channel cleanly.

Requests (Rust → Python):

{"id": 1, "method": "transcribe", "params": {"path": "/abs/path.mp3", "model": "base", "language": null, "openai_api_key": null, "initial_prompt": "Sundar Pichai, GOOG, ..."}}
{"id": 2, "method": "list_models",     "params": {}}
{"id": 3, "method": "download_model",  "params": {"name": "small"}}
{"id": 4, "method": "delete_model",    "params": {"name": "small"}}
{"id": 5, "method": "import_model",    "params": {"name": "my-finetune", "zip_path": "/abs/model.zip", "description": "..."}}
{"id": 6, "method": "cancel_download", "params": {"name": "small"}}
{"id": 7, "method": "cancel_transcribe","params": {}}

{"id": 8,  "method": "tts_runtime_status", "params": {}}
{"id": 9,  "method": "list_tts_models",    "params": {}}
{"id": 10, "method": "download_tts_model", "params": {"name": "Fun-CosyVoice3-0.5B-2512"}}
{"id": 11, "method": "delete_tts_model",   "params": {"name": "Fun-CosyVoice3-0.5B-2512"}}
{"id": 12, "method": "cancel_voice_clone", "params": {}}
{"id": 13, "method": "register_speaker",   "params": {"name": "Alice", "audio_path": "/abs/ref.wav", "whisper_model": "base"}}
{"id": 14, "method": "list_speakers",      "params": {}}
{"id": 15, "method": "delete_speaker",     "params": {"id": "alice-3f2a"}}
{"id": 16, "method": "clone_voice",        "params": {"speaker_id": "alice-3f2a", "text": "...", "model": "Fun-CosyVoice3-0.5B-2512", "language": "en", "speed": 1.0, "instruct": "warm and conversational"}}
{"id": 17, "method": "list_clones",        "params": {"speaker_id": "alice-3f2a"}}
{"id": 18, "method": "delete_clone",       "params": {"output_path": "/abs/clones/alice-3f2a/2026-…wav"}}
{"id": 19, "method": "clear_speaker_clones","params": {"speaker_id": "alice-3f2a"}}

cancel_transcribe and cancel_voice_clone are best-effort: the transcribe loop checks the cancel flag at segment boundaries (and around the cloud HTTP call); the voice-clone path kills the helper subprocess and the next call restarts it lazily. The frontend treats both optimistically — UI unblocks immediately and any late result is discarded.

Responses (Python → Rust):

{"id": 1, "result": {"language": "en", "duration": 12.4, "segments": [{"start":0.0,"end":2.1,"text":"...","words":[{"start":0.0,"end":0.4,"text":"Hello"}]}]}}
{"id": 1, "error":  "FileNotFoundError: ..."}

Notifications (Python → Rust → frontend, fire-and-forget):

{"event": "ready",                "data": null}
{"event": "transcribe:progress",  "data": {"stage": "transcribing", "ratio": 0.42}}
{"event": "model:progress",       "data": {"name": "small", "stage": "downloading", "ratio": 0.31, "bytes": 152000000, "total_bytes": 488000000, "bytes_per_sec": 4200000, "eta_seconds": 80}}

{"event": "tts:install:progress", "data": {"stage": "installing-deps"}}
{"event": "tts:model:progress",   "data": {"name": "Fun-CosyVoice3-0.5B-2512", "stage": "downloading", "bytes": 1200000000, "total_bytes": 5200000000}}
{"event": "tts:progress",         "data": {"stage": "synthesizing"}}
{"event": "tts:helper:progress",  "data": {"stage": "complete", "duration_sec": 3.48, "elapsed_sec": 13.6, "rt_factor": 3.9}}

Topics with a : flow straight through to the webview as Tauri events; ready flips the backend:ready flag instead. The legacy bare progress event is still accepted for backwards compatibility (mapped to transcribe:progress).

Add new methods by extending Backend.methods in python/main.py, then calling them from the frontend with invoke("name", { ... }) after wiring a new #[tauri::command] in src-tauri/src/lib.rs. Tauri auto-converts camelCase JS arg names to snake_case Rust params.

Voice Clone (CosyVoice 3)

The left rail's Voice Clone view runs zero-shot voice synthesis using FunAudioLLM/CosyVoice 3 0.5B. The runtime is too heavy (~6.5 GB) to bundle, so Vuesub installs it on demand into ~/.vuesub/runtimes/cosyvoice/ the first time the user downloads the model. End users need three tools on PATH:

Tool	Why	Install
Python 3.10–3.12	CosyVoice's torch pin	brew install python@3.12 (mac) · python.org (Win) · apt install python3.12 python3.12-venv (Linux)
uv	venv + dep installer (10× faster than pip for our 1.5 GB tree)	curl -LsSf https://astral.sh/uv/install.sh | sh (mac/Linux) · irm https://astral.sh/uv/install.ps1 | iex (Win)
git	shallow-clones the CosyVoice repo + Matcha-TTS submodule	xcode-select --install (mac) · git-scm.com (Win) · apt install git (Linux)

The Voice Clone page surfaces missing tools with install snippets. Storage layout:

~/.vuesub/runtimes/cosyvoice/
  .venv/                                  uv-managed venv (~1.5 GB)
  repo/                                   shallow clone of FunAudioLLM/CosyVoice
  models/Fun-CosyVoice3-0.5B-2512/        ~5.2 GB (pruned from 9 GB)
  tts_helper.py                           the helper this sidecar talks to
~/.vuesub/speakers/<id>/                  reference.wav + meta.json (auto-transcript)
~/.vuesub/clones/<id>/                    generated WAVs

Performance on M-series Mac CPU: ~60s cold model load, ~3.5× realtime for synthesis, ~6 GB peak RAM. The helper process is killed after 5 min of idle to release that RAM.

Speaker registration & transcript alignment

CosyVoice 3 zero-shot needs both the reference audio AND the reference text to extract speaker characteristics. Vuesub auto-transcribes the uploaded clip with Whisper at registration time (defaults to the largest installed local model — large-v3 if available, falling back to smaller ones). The auto-transcript is stored in ~/.vuesub/speakers/<id>/meta.json and shown inline on the Voice Clone page with an Edit affordance — when synthesis quality drifts (poetic Chinese, accented speech, etc.), fixing the transcript is the highest-leverage repair.

If no Whisper model is installed, the new-speaker modal falls back to a manual transcript input where the user picks a language and types out what their reference says. The Tauri command surface accepts both shapes via optional transcript/language params on register_speaker.

Implementation notes

• Two-process design. python/voiceclone.py runs in the bundled main sidecar and proxies JSON-RPC to a long-running python/tts_helper.py worker spawned from the lazy venv. The proxy keeps one helper warm across calls and kills it on idle to release RAM.
• Audio playback. Generated WAVs are read via the read_clone_bytes Tauri command (path-restricted to ~/.vuesub/clones/) and wrapped in a Blob URL. WebKit's asset:// playback is flaky in some Tauri 2 builds; blob URLs work universally. Outputs are forced to PCM-16 mono (not CosyVoice's default float-32) so every webview's <audio> element can decode them.
• No language tag prepend. CosyVoice 2's <|en|>/<|zh|> tags don't exist in CosyVoice 3's tokenizer (only <|endofprompt|> + performance directives + phonemes are special-tokenised). Prepending them makes the model literally vocalise the tag as a prefix. We rely on auto-detection from tts_text content. The "Output language" picker is currently a no-op — a placeholder for future inference_instruct2 integration.

Whisper models

Built-in models download from Hugging Face via the Settings → Models panel with byte-level progress and ETA. Cache lives under ~/.cache/huggingface/hub. Imported .zip models go to ~/.vuesub/models/<name>/.

Model	Size	RAM	Notes
tiny	75 MB	~1 GB	Very fast, lower accuracy
base	145 MB	~1 GB	Good default
small	488 MB	~2 GB	Better accuracy
medium	1.5 GB	~5 GB	Strong accuracy on most audio
large-v3	3 GB	~10 GB	Best quality; requires a beefy machine
openai/whisper-1	—	—	Cloud option. Audio is auto-transcoded to 16 kHz mono Opus 16 kbps before upload to fit OpenAI's 25 MB API limit (≈ 7 MB / hour).

To use a GPU at runtime:

# Apple Silicon: faster-whisper does not support MPS yet — use CPU int8.
# NVIDIA:
export VUESUB_DEVICE=cuda
export VUESUB_COMPUTE_TYPE=float16

Export formats

The toolbar's format picker (src/App.tsx → FORMATS) writes one of four files via render() (line 1736). All four operate on the same in-memory Segment[], so edits in the transcript are always reflected.

Ext	Renderer	Use case
.srt	inline in render	Universal subtitle file. Works with VLC / YouTube / Vimeo / Premiere / Resolve / FCP / CapCut. Default — pick this if unsure.
.vtt	inline in render	WebVTT for HTML5 <video><track>. Same shape as SRT with . instead of , and a WEBVTT header.
.txt	inline in render	Plain transcript, no timestamps. One trimmed line per segment.
.fcpxml	renderFcpxml()	FCPXML 1.11 project for Final Cut Pro. Each segment is a native <caption> on lane 1 of a sequence.

SRT / VTT / TXT

Stateless renderers — no per-format settings. Time formatting:

• srtTime() — HH:MM:SS,mmm
• vttTime() — HH:MM:SS.mmm

Output is UTF-8 with \n line endings.

FCPXML

The FCPXML export opens a dialog (#FCPXML Export Settings) that lets the user pick a resolution (720p / 1080p / 1440p / 2160p) and frame rate (23.976 / 24 / 25 / 29.97 / 30 / 50 / 59.94 / 60). Both are persisted to localStorage under vuesub:fcpxml-prefs.

Implementation notes:

• Each segment becomes a <caption lane="1" …> with an inline <text-style-def> so it imports as a real, restyleable FCP caption (not burnt-in titles). Default style is Helvetica 48 bold, white, centered, bottom-placement, role iTT?captionFormat=ITT.en.
• Offsets and durations are rounded to integer-frame boundaries of the chosen rate via fcpxmlTime() so FCP imports without "caption is not on an edit frame boundary" warnings. The spine gap is rounded up (fcpxmlTimeCeil) so captions never spill past it.
• The 29.97 and 59.94 profiles emit tcFormat="DF" (drop-frame). NDF for everything else.
• Format names follow FFVideoFormat<res><rate> (e.g. FFVideoFormat1080p2997) which is what FCP looks up when it imports.

To add a new resolution or frame rate, extend RES_PROFILES / FPS_PROFILES. The dialog's options are derived from those tables, so a new entry shows up automatically.

Languages

Whisper-supported languages exposed in the UI: Auto, English, Chinese (Mandarin / Cantonese), Japanese, Spanish, Italian, German, Thai, Russian, Korean, Dutch, French, Hindi, Arabic, Hebrew. The toolbar's 简/繁 button appears whenever the active language is Chinese and converts the in-memory transcript via opencc-js. Language choice and Chinese variant are persisted in localStorage so they survive relaunches.

UI scaling

Settings → General → Font size exposes four presets (Small 0.85×, Medium 1×, Large 1.15×, Extra large 1.3×). The selection writes --fs-scale on <html> and persists to localStorage["vuesub:font-size"]. All the type tokens (--fs-xs … --fs-xl) and the transcript's fixed-width columns (#, timecodes, row actions) use calc(<base>px * var(--fs-scale)) so the whole interface scales without re-layout glitches. See src/font-size.ts.

The toolbar selects (Model / Lang / Format) and every modal select use the custom ChipSelect / MenuSelect from src/Dropdown.tsx — a portal-rendered listbox that opens flush below the trigger, with keyboard nav (Up / Down / Home / End / Enter / Tab / Esc) and outside- click dismissal. macOS's native pop-up button always centers the current item over the trigger, which obscures it; the custom widget gives a predictable drop-below behaviour across platforms.

Release flow

1. Bump version in package.json, package-lock.json, src-tauri/Cargo.toml, src-tauri/Cargo.lock, and src-tauri/tauri.conf.json.
2. Commit to main.
3. Push a tag matching vX.Y.Z (git tag -a v0.1.4 -m "..." && git push origin v0.1.4).
4. CI builds mac aarch64 + windows x64 + linux x64, uploads installers to a draft release on opcify/Vuesub-Pro, and flips it to live once at least one asset lands.

License

Proprietary — replace this section before publishing.