VoxWeave

BGM-robust subtitles for anime, film, and clips.

Vocal separation and song-skip so ASR never hallucinates on background music, OP/ED, or insert songs. Local-first Qwen3 ASR, forced alignment, and edit-and-resync — CJK-aware.

https://github.com/user-attachments/assets/e75b6dd3-fa37-4afe-89db-b6ee2c28f6bc

_{Sliced clip under heavy BGM · voxweave Test.mp4 · Qwen3-ASR-1.7B}

[!NOTE] Local-first. Separation, ASR, and forced alignment all run in-process on your GPU — no network endpoints, no audio leaves the machine. Runs on NVIDIA CUDA (PyTorch) and on Apple Silicon, where ASR + alignment use the native MLX Qwen3 models. Weights download once on first run. (Translation and ASR-correction are the only optional features that call an external LLM, and only when you invoke them.)

[!NOTE] Hardware. The default pipeline (Qwen3-ASR-0.6B, peak load strategy) runs in ~8 GB of VRAM — the separator is freed before ASR + alignment load, so peak ≈ max(stage), not their sum. --asr-model qwen3-asr-1.7B adds roughly +2 GB and still fits 8 GB under the default peak strategy. load_strategy = "sum" (concurrent, faster on big cards) makes peak the sum of the resident models — plan for 12 GB+. On Apple Silicon the MLX 8-bit weights roughly halve the Qwen footprint (so 1.7B fits comfortably in 16 GB unified memory). --hybrid loads a Whisper engine alongside Qwen to trade VRAM for accuracy — it does not save memory; the only knob that lowers it is staying on 0.6B + the peak strategy.

VoxWeave derives from the WhisperX "edit-and-resync" workflow: transcribe once, then edit the text and re-align it against the original audio for frame-accurate timestamps. Where it differs is the front end — vocal separation and song-skip keep background music out of the ASR, and a CJK-aware layout/alignment stack (MMS-300m for Japanese, BudouX/jieba for line breaks) handles Chinese/Japanese/English as first-class.

Contents

• Why VoxWeave
• Setup
• Quickstart
• Usage
  • Transcribe (voxweave <media>)
  • Label speakers (speakers)
  • Re-align after editing (align)
  • Re-layout offline (render)
  • ASR correction (correct)
  • Translate (translate)
  • Export (export)
  • Pack soft subtitles (pack)
  • Burn hard subtitles (burn)
• The edit-and-resync workflow
• Migration notes
• How it works
• Configuration
  • Performance knobs
• Data contract
• Testing
• Support
• License
• Acknowledgments

Why VoxWeave

• BGM removal before ASR. A Mel-Band Roformer vocal separator (pure torch, full-band 44.1k) strips music first, so ASR doesn't transcribe lyrics or hallucinate on score.
• Song-skip. PANNs detects singing/music on the separated vocals and skips OP/ED and insert songs before ASR — on by default, --no-skip-songs to keep them.
• Local Qwen3 ASR + forced alignment. Text and word-level timestamps in one pass, fully on-device — in-process PyTorch on NVIDIA, or the native MLX Qwen3 models on Apple Silicon. A Whisper hybrid engine is also available for when you prefer Whisper text (faster-whisper on NVIDIA, the native MLX Whisper port on Apple Silicon).
• Edit-and-resync. Fix the transcript by hand, then align re-derives timestamps from the audio — timestamps are never hand-written.
• CJK-aware. Japanese aligns with MMS-300m + uroman (zero-OOV, immune to the per-cue drift that breaks wav2vec2-xlsr on rare kanji); line breaks use BudouX phrase atoms + jieba.
• Optional LLM steps. correct cleans up ASR typos/garbled names before alignment; translate does whole-episode context-aware translation while preserving cue count (dual-speaker -line/-line cues are translated one speaker at a time and re-assembled).
• Ship the result. pack soft-muxes finished VTTs into the media as titled subtitle tracks (instant stream copy); burn hardcodes them at constant quality with NVENC / VideoToolbox acceleration, matching the source bit depth.

Setup

Two install variants: voxweave[cuda] (NVIDIA GPU — Blackwell sm_120 / cu128 by default) and voxweave[mps] (Apple Silicon / macOS). Both need ffmpeg on PATH.

# Ubuntu / Debian
sudo apt update && sudo apt install ffmpeg
# Arch Linux
sudo pacman -S ffmpeg
# macOS (Homebrew)
brew install ffmpeg

Install from PyPI (puts the global voxweave command on PATH):

# NVIDIA / Linux:
uv tool install --torch-backend=cu128 "voxweave[cuda]"   # full pipeline + faster-whisper hybrid
# Apple Silicon / macOS:
uv tool install "voxweave[mps]"                          # full pipeline + MLX Whisper hybrid

The full local pipeline — vocal separation, ASR, forced alignment (incl. MMS-300m for Japanese/CJK), layout, song-skip, and speaker-diarization support — plus CJK line-break and translation — is baked into the core dependencies. The variant selects the compute platform and the ASR/alignment backend:

• [cuda] (NVIDIA/Linux): the in-process PyTorch Qwen3-ASR + forced aligner (qwen-asr), GPU onnxruntime (CUDAExecutionProvider for MMS alignment), and the faster-whisper hybrid engine.
• [mps] (Apple Silicon/macOS): ASR runs on the native MLX Qwen3-ASR from mlx-audio (Metal kernels + quantization). Alignment keeps the same per-language stack as [cuda]: English on wav2vec2 CTC (torch, runs on MPS; the forced-align DP falls to CPU as torchaudio has no Metal kernel), Japanese/CJK on the ONNX MMS aligner (CoreML/CPU — onnxruntime has no Metal provider). Only the Qwen fallback (zh·yue, or any CTC failure) is served by the MLX Qwen3-ForcedAligner, since the torch qwen-asr aligner is absent here. The Whisper hybrid/fusion engines (--asr-model large-v3, --hybrid) run on the native mlx-whisper Metal port instead of faster-whisper (ctranslate2 has no Metal backend). Vocal separation (MelBandRoformer) + PANNs song-skip stay on torch-MPS. qwen-asr is excluded because its transformers==4.57.6 pin conflicts with mlx-audio, so [cuda] and [mps] are mutually exclusive — pick one per host.

Speaker diarization support ships by default on both variants but remains opt-in at runtime. The default model is pyannote/speaker-diarization-community-1, which separates and counts multiple speakers noticeably better than 3.1. It is gated separately on Hugging Face: before using --diarize, accept the conditions for the model you select, then authenticate once with hf auth login or set VOXWEAVE_HF_TOKEN / HF_TOKEN. Users who have accepted only the 3.1 gate can keep it with --diarize-model 3.1.

Phase-0 measurements used pyannote.audio 4.0.7, the same Japanese 16 kHz test waveform, and an RTX PRO 4000 24 GB. Peak allocated CUDA memory was identical in that run:

Community-1 generally improves speaker counting and separation; 3.1 stays selectable per run for users who have accepted only the existing 3.1 gate.

From source (for development or pulling new code):

make install       # auto-detects the platform: Apple Silicon -> [mps], anything else -> [cuda]
                   # (cu128 torch wheel with an NVIDIA driver, CPU wheel without)
make reinstall     # after pulling new code (same platform detection)
make uninstall

Override the detection per invocation: make install VARIANT=mps or make install TORCH_BACKEND=cpu.

Quickstart

# Transcribe a video to a timestamped VTT (+ a JSON source of truth)
voxweave episode.mkv

# Opt in to speaker diarization (requires gated-model access; see Setup)
voxweave interview.mkv --diarize

# Stay on the older 3.1 pipeline if that is the gate you have accepted
voxweave interview.mkv --diarize --diarize-model 3.1

# ...edit episode.vtt by hand (fix wording, line breaks)...

# Re-align the edited text against the original audio
voxweave align episode.vtt

# Optionally translate the aligned subtitles to Chinese
voxweave translate episode.vtt --target zh

Usage

All processing commands share numbered workflow steps, a segmented dot bar, and elapsed time. For example, while translating:

[2/3] translate cues
translate -> zh       [⣿⣿⣿⣿⣿⣿⣀⣀⣀⣀⣀⣀⣀⣀⣀⣀⣀⣀⣀⣀] 128/400 0:00:12

The denominator reflects enabled steps, not an estimate of elapsed time. Downloads and retries stay inside their current step. Cyan means active, green complete, yellow warning, and red failure; elapsed time is subdued. Unknown totals use a pulse instead of a fabricated percentage. Burn reports actual encoded frames/time. Non-terminal runs emit static step lines and occasional encoding updates to stderr, with no animation; stdout remains reserved for result paths or the speaker-service URL.

A run that finishes cleanly closes with one muted stderr line of wall-clock time per workflow step, so "where did the time go" is answerable from any run:

timing: inspect source 0.4s | prepare audio 1m17s | detect songs 31.6s | find speech 9.1s | transcribe and align 1m41s | detect shot changes 0.1s | layout subtitles 1.2s | write outputs 0.3s | total 3m41s

Anything under a minute is printed as seconds, anything above as NmSSs. A step entered more than once accumulates; total is the sum of the listed steps. Failed runs end in the error panel instead, and a run without a declared plan prints no summary. A transcription run with --debug additionally records the steps finished by mid-run in debug/meta.json under a timings key — that file is written while transcribe and align is still open, so it holds a prefix of the printed line, and only the transcribe path writes it.

Transcribe

voxweave transcribe <media> (or voxweave <media>) — separation → song-skip → VAD chunking → ASR + forced alignment → smart_split → writes <stem>.vtt (editable) + <stem>.json (word-level timestamp source of truth). Models load in-process (see voxweave.backend); the separator is released from VRAM before ASR+alignment load, so peak usage is ≈ max(sep, asr) rather than their sum.

voxweave episode.mkv
voxweave transcribe episode.mkv             # explicit form of the same command
voxweave clip.mp4 --no-separate          # clean speech (podcast/lecture): skip separation
voxweave episode.mkv --asr-model qwen3-asr-1.7B   # larger, more accurate ASR
voxweave episode.mkv --context "Ryland Grace, Astrophage, Hail Mary"   # bias names/terms

Unknown command words produce a command error. Bare subtitle or JSON paths are not transcribed: use align for edited VTT, export for subtitle format conversion, or render for layout from the sibling JSON. These inputs are never automatically routed to an in-place editing command. See the migration notes for old names.

Label speakers

After a diarized transcription, voxweave speakers serve <media> prepares an audition page in memory, serves it on 127.0.0.1, and opens it in your browser. The page embeds up to three clean, non-overlapping speech clips per diarizer id. Listen, enter names, then select Save to write them directly to the episode's speaker mapping. The server runs until Ctrl+C; use --no-open to print the URL without opening a browser, or --port N to choose its loopback port. No audition HTML is written to disk. voxweave speakers <media> is the supported shorthand for serve; --manual disables voice matching for that session.

voxweave episode.mkv --diarize
voxweave speakers serve episode.mkv
# Save names in the browser, stop the server with Ctrl+C, then render them
voxweave render episode.json

If pyannote merged two people under one diarizer id, select Split this speaker on that card. VoxWeave clusters the id's individual turns and lets you audition both proposed groups before applying the split. This action requires the episode to have been captured with --voiceprints; it refuses the proposal if the original embedding and audio provenance cannot be reproduced (for example, when a bound separated-vocals cache is missing or stale). A confirmed split rewrites speaker_turns and the bound voiceprint centroids, keeps a one-level undo snapshot, and asks you to restart voxweave speakers serve to audition and name the new id. Undo is refused after any rewritten input changes.

Voice matching across episodes is a separate, opt-in layer. Capture centroids with --diarize --voiceprints, review the ordinary empty mapping, then enroll only those human-entered names into an explicitly selected show store:

voxweave episode.mkv --diarize --voiceprints
voxweave speakers serve episode.mkv
# Save reviewed names in the browser first
voxweave speakers enroll episode.mkv \
  --voices ./example-show.voices.json --show "Example Show"

# Later episodes: suggestions appear in the served page and a regenerable cache record.
voxweave speakers serve episode-02.mkv --voices ./example-show.voices.json

A missing store is created only by speakers enroll with both explicit --voices and --show. Use speakers enroll --replace to replace this episode's prior contribution to that store. For reuse-only discovery, name it voxweave.voices.json beside the media and pass an equal normalized --show; discovery without --show reports the store but stays in manual mode. The shipped matching policy is suggest-only: VOXWEAVE_VOICES_ACCEPT defaults to off, so stored names appear as review buttons and never become authoritative mapping values. Even when an operator configures a finite accept threshold, a machine prefill exists only in the served input; the v1 mapping is still created with empty names and must be reviewed and saved by the operator.

The versioned mapping is intentionally small:

{"version": 1, "speakers": {"SPEAKER_00": "Aoi"}}

render renders mapped names as WebVTT voice tags while keeping transcript text and the sibling JSON clean. Empty or missing names remain unlabeled. An existing mapping is the normal edit flow: the server re-reads it for the page and replaces it only when you select Save. When burning a mapped SRT, recovered speaker identity is retained in the temporary ASS Name field while the visible dialogue remains prefix-free.

voxweave speakers purge episode.mkv removes that episode's voiceprint artifact and suggestion record from both supported legacy and cache locations. It also removes the cache-only one-level speaker-split undo snapshot and any legacy audition HTML while holding the episode transaction lock. It works after the media has been removed and deliberately keeps the human-edited mapping.

voxweave speakers list episode.mkv inspects that episode's speaker turns, reviewed mapping, and voiceprint state without changing them. Add --json for machine-readable output. It does not list or select a show-level voices store.

Re-align after editing

voxweave align <vtt> — takes the edited VTT text and re-runs forced alignment against the original audio, overwriting the timestamped VTT and updating the JSON. Does not re-run ASR or touch smart_split. Aligns on separated 16k vocals by default (prevents BGM interference); prefers the episode's cached vocals.32k.flac under the artifact cache, otherwise re-separates and caches it there. Existing media-adjacent cache/<stem>.vocals.32k.flac entries remain a legacy read/write-back lane. When a voiceprint pair requires source binding, both managed and legacy vocals caches are accepted only when their matching integrity companion validates.

voxweave align episode.vtt                 # finds episode.<ext> in the same dir
voxweave align episode.vtt --media original.mkv
voxweave align episode.vtt --no-separate   # align on the original audio (clean sources)

Re-layout offline

voxweave render <episode> — re-run deterministic layout from <stem>.json without any models (adjust line width / sentence breaks instantly).

voxweave render episode.json --max-line-length 14 --max-lines 1
voxweave render episode.json --no-timestamps   # plain-text editing draft

Pass the JSON, its VTT sibling, or the media path; each resolves to the same sibling JSON. This command rewrites the working VTT and JSON. Save or align manual VTT edits before rendering: the CLI rename does not add a backup or overwrite guard.

ASR correction

voxweave correct <vtt> — optional pre-align LLM pass that fixes obvious ASR typos, split words, and garbled proper nouns, producing a reviewable diff. Conservative substitution only (no completion/rewrite), gated by a code check that the matched text equals the original line-for-line. By default writes only an adjacent sidecar <stem>.asrfix.vtt plus an audit JSON in the episode artifact cache — the original VTT is untouched. Use --apply to overwrite, then run align to reassign timing.

voxweave correct episode.vtt --glossary names.json   # review the sidecar
voxweave correct episode.vtt --glossary names.json --apply
voxweave align episode.vtt

Translate

voxweave translate <subtitle> — after align, translate each cue with whole-episode context, preserving cue count, into <stem>.<target>.<ext> (the original is left unchanged). Accepts .vtt/.srt/.ass/.ssa; the output mirrors the input format (episode.srt → episode.zh.srt).

voxweave translate episode.vtt --target zh
voxweave translate episode.vtt --target en --context "sci-fi, formal register" --glossary terms.json
voxweave translate downloaded.srt -t zh               # foreign SRT in, SRT out
voxweave translate episode.vtt --target zh --reasoning-effort low
voxweave translate episode.vtt --target zh --concurrency 1   # one whole-episode request

Translation accepts any OpenAI-compatible Chat Completions endpoint, including a self-hosted Qwen server. Configure it once in ~/.config/voxweave.conf:

[llm]
base_url = "http://127.0.0.1:8000/v1"
model = "auto"              # or the exact served model ID from /v1/models
api_key_env = ""            # keyless server; otherwise name an environment variable
reasoning_effort = "low"    # translate only; omit to keep the server default
concurrency = 8             # translate windows in flight; 1 = single whole-episode request
window_cues = 100           # cues per window when concurrency > 1

By default the episode is translated as bounded windows (window_cues cues each) with several requests in flight (concurrency); each window sees the preceding source cues as context. This trades some cross-window stylistic continuity for throughput and resilience on a self-hosted server -- --glossary and --context are the consistency tools there. --concurrency 1 sends one whole-episode request (sequential windows with translated-tail continuity only past 800 cues), the best choice for a hosted API.

Every response is checked for completeness: an answer that does not finish with stop (length cap, a server aborting its structured-output grammar) is retried, and the last attempt for that window runs without response_format (plain JSON). Cues still untranslated after the retry stage fail the command and keep the progress file, so rerunning resumes; --allow-partial writes the file anyway with those cues in source text.

CLI options override environment variables, which override this configuration. --model accepts a served model name without a built-in model list. auto requires the endpoint to advertise exactly one model. For authenticated endpoints, set api_key_env = "MY_LLM_KEY" and put the secret in that environment variable.

--reasoning-effort sends the standard top-level reasoning_effort field in the request. Accepted values depend on the served model: for example, Qwen3.8-27B supports low, medium, and xhigh. Use --reasoning-effort default to override a configured value and omit the field. VoxWeave does not inject thinking template switches or silently replace an effort rejected by the endpoint. Reasoning output is kept out of subtitle text. Interrupted translations resume only when the input, endpoint, resolved model, effort, context, glossary, and target match.

Export

voxweave export <subtitle> — convert between subtitle formats: VTT/SRT/ASS/SSA in, SRT/ASS/VTT out (written next to the input; the VTT + JSON pair stays the source of truth for voxweave-produced subtitles). ASS output carries a Default style; lyric cues (♪ ... ♪) render italic. Named VTT cues become NAME: text in SRT and use the ASS Dialogue Name field. Foreign SRT/ASS files can be exported to VTT to enter the editing workflow.

voxweave export episode.vtt --format srt
voxweave export episode.vtt -f srt -f ass
voxweave export downloaded.ass --format vtt        # foreign ASS -> VTT for editing/translate

Repeat -f, --format to request multiple output formats in one run.

Pack (soft subtitles)

voxweave pack <subtitle>... — remux the source media with the subtitle file(s) (VTT/SRT/ASS) added as proper subtitle tracks. Pure stream copy (instant, lossless, reversible); each track is titled VoxWeave <Language> with the container language tag taken from the filename (episode.zh.vtt → chi / "VoxWeave Chinese"), and the first packed track is flagged default so players select it. ASS inputs keep their styling in mkv targets (mp4/webm store text-only codecs, so styling is dropped there).

voxweave pack episode.zh.vtt                    # finds episode.<ext>, keeps its container
voxweave pack episode.zh.vtt episode.ja.vtt     # several tracks at once
voxweave pack episode.zh.vtt --container mp4    # mov_text in mp4 (image subs are dropped)
voxweave pack episode.zh.vtt --media other.mkv -o out.mkv

mkv targets keep every source stream (including attachments); mp4/webm targets keep video+audio and existing text subtitle tracks only. HEVC video muxed into mp4 is tagged hvc1 for Apple players.

Burn (hard subtitles)

voxweave burn <subtitle> — render the subtitles (VTT/SRT/ASS) into the pixels and write a clean file with all subtitle tracks removed. For VTT/SRT input a styled ASS is generated at the actual frame size (same look as export, lyric cues italic); ASS/SSA input goes to libass as-is, keeping its own styling. The video is re-encoded at constant quality with hardware acceleration when available: NVENC on NVIDIA, VideoToolbox on macOS, libx264/libx265/libsvt-av1 software fallback. Audio is stream-copied (mp4 targets re-encode mp4-incompatible codecs to AAC).

voxweave burn episode.zh.vtt                          # hevc, auto hw encoder, -> episode.mp4
voxweave burn episode.zh.vtt --codec h264             # legacy-device compatibility
voxweave burn episode.zh.vtt --codec av1 --container mkv   # max compression, recent hardware
voxweave burn episode.zh.vtt --quality 20 --font "Noto Sans CJK SC"

Progress is rendered with rich: countable stages (demix windows / PANNs batches / per-chunk ASR+alignment / align per-cue / translate streaming per-line) show a real x/N bar with elapsed time; indeterminate stages (decode / file write) show a pulse bar. -v/--verbose enables DEBUG logging.

The edit-and-resync workflow

voxweave episode.mkv          # 1. transcribe  -> episode.vtt + episode.json
  └─ (optional) correct       # 2. LLM ASR fix -> episode.asrfix.vtt (--apply to commit)
edit episode.vtt by hand      # 3. fix wording / line breaks
voxweave align episode.vtt    # 4. re-derive timestamps from audio (overwrites VTT + JSON)
voxweave translate episode.vtt --target zh   # 5. context-aware translation
voxweave pack episode.zh.vtt             # 6. soft-mux into the media (or burn for hardsubs)

Timestamps are always derived from the audio by the forced aligner — you never hand-edit them. Edit the text freely; align puts the timing back.

How it works

Configuration

Precedence: CLI flag > env var > ~/.config/voxweave.conf > built-in default. A commented default config is written on first run (migrated automatically from a pre-rename qsub.conf).

# ~/.config/voxweave.conf  —  TOML
# Precedence: CLI flag > env var > this file > built-in default.

# Default ASR model (= --asr-model). Short name (qwen3-asr-0.6b | qwen3-asr-1.7b) or full HF id.
# Special value "hybrid" (= --hybrid) -> dual-ASR fusion (whisper text + Qwen punctuation).
asr_model = "Qwen/Qwen3-ASR-1.7B"        # built-in default: Qwen/Qwen3-ASR-0.6B

# Model load strategy:
#   "peak" (default) — serial peak-shaving: all-chunk ASR -> release -> all-chunk align;
#                      ASR and aligner never co-reside, peak VRAM = max(models). Works on 8 GB.
#   "sum"            — concurrent per-chunk ASR+align; peak VRAM = sum(models), but skips two
#                      model swap round-trips (faster on large-VRAM cards).
load_strategy = "sum"

# Inference batch sizes: windows per GPU forward (env: VOXWEAVE_SEP_BATCH / VOXWEAVE_CTC_BATCH /
# VOXWEAVE_MMS_BATCH / VOXWEAVE_ASR_BATCH). On an 8 GB-class card batch=1 already saturates
# compute — measured no speedup at 2/4, just ~+0.8 GiB VRAM per extra separation window — so the
# defaults stay at 1. Only worth raising on much wider GPUs, and only after measuring.
[batch]
separate = 1                             # vocal separation (MelBandRoformer) 8s windows
ctc      = 1                             # wav2vec2 CTC emission 30s windows (en aligner)
mms      = 4                             # MMS-300m emission batch (ja aligner)
asr      = 1                             # Qwen3-ASR chunks per decode call; 1 = the per-chunk call.
                                         # Opt-in: faster but its transcripts differ from batch 1 —
                                         # see "Performance knobs" below

# Vocal separation numerics (= env VOXWEAVE_SEP_AUTOCAST). autocast wraps only the model
# forward; the overlap-add accumulation stays fp32. CUDA only (ignored on CPU / MPS).
#   "off" (default) — fp32 forward, the reference output, byte-identical run to run.
#   "bf16" | "fp16" — mixed-precision forward: faster and slightly leaner, at the cost of tiny
#                     waveform differences in the stem. See "Performance knobs" below.
[separate]
autocast = "off"

# Diarization pipeline (= --diarize-model / env VOXWEAVE_DIARIZE_MODEL).
# Values: "community-1" (built-in default), "3.1", or any full Hugging Face pipeline id.
# Voiceprint stores are per-model: centroids captured under one pipeline do not match under
# the other (different embedding space), so switching models starts a fresh voiceprint store.
[diarize]
model = "community-1"

# Default on/off for the boolean pipeline flags. An explicit CLI flag always wins
# (e.g. separate = false here, --separate on the command line for one run).
[defaults]
separate   = true                        # vocal separation before ASR/alignment (--separate/--no-separate)
skip_songs = true                        # PANNs music detection + skip before ASR (--skip-songs/--no-skip-songs)
normalize  = false                       # loudnorm on the 16k input (--normalize/--no-normalize)
diarize    = false                       # pyannote speaker diarization (--diarize/--no-diarize; gated-model token required)
voiceprints = false                      # opt-in biometric centroid capture; requires diarize
timestamps = true                        # word-level timestamps in the VTT (--timestamps/--no-timestamps)
shot_snap  = true                        # snap cue boundaries onto shot changes (--shot-snap/--no-shot-snap)
vad_mask   = false                       # suppress CTC emissions outside speech (--vad-mask/--no-vad-mask)

# LLM for translate / correct: any OpenAI-compatible chat-completions endpoint.
# Precedence per key: CLI option > env (VOXWEAVE_TRANSLATE_MODEL / VOXWEAVE_FIX_MODEL,
# OPENAI_BASE_URL) > this section > built-in (gpt-5.5 on api.openai.com, key from OPENAI_API_KEY).
[llm]
model = "auto"                           # or a model name; "auto" = the endpoint's only served model
base_url = "http://127.0.0.1:8000/v1"    # e.g. a local vLLM; remove for api.openai.com
api_key_env = ""                         # "" = keyless endpoint; else the env var holding the key
# reasoning_effort = "low"               # translate only; accepted values depend on the served model
# concurrency = 8                        # translate windows in flight; 1 = single whole-episode request
# window_cues = 100                      # cues per window when concurrency > 1

# dual-ASR fusion sub-models — only consulted when running with --hybrid.
[fusion]
whisper = "large-v3-turbo"               # Whisper size: large-v3 (best) | large-v3-turbo (~5x faster); faster-whisper on cuda, mlx-whisper on mps
qwen    = "Qwen/Qwen3-ASR-1.7B"          # punctuation model; must emit punctuation -> 1.7B, not 0.6B

# Per-language forced-alignment model. Key = ISO-639-1 code; unlisted languages use Qwen3-ForcedAligner.
# Values:
#   "mms"   — MMS-300m + uroman, full-file single pass (immune to per-cue drift; the gold standard).
#   HF id   — wav2vec2 CTC via HF transformers; weights land in ~/.cache/voxweave/align (per-cue crop).
#   bundle  — torchaudio bundle name, e.g. "WAV2VEC2_ASR_LARGE_LV60K_960H" (same model, cached in ~/.cache/torch).
#   ""      — explicitly fall back to Qwen for that language.
[align]
en = "facebook/wav2vec2-large-960h-lv60-self"  # English: LV60K-self CTC, per-cue crop (HF hub)
ja = "mms"                                      # Japanese: MMS-300m + uroman full-file (= whisperx fork align_ctc)
# zh  = "mms"                                   # Chinese can also use MMS; default is Qwen (native CJK char-level)
# yue = ""                                      # force Qwen for Cantonese

Performance knobs

Two GPU throughput settings are opt-in and off by default, because both change the output. They are worth enabling only on a wide GPU, and only if you have re-checked the result on your own material — there is no truth ruler in the pipeline that can tell you whether the changed output is better or worse.

Neither has a CLI flag; precedence for both is env var > config file > built-in default. [batch].asr applies to the torch Qwen engine only — Whisper and the Apple Silicon MLX adapter take one chunk per ASR call whatever it says — and autocast applies to CUDA only, being ignored on CPU/MPS. Raising [batch].separate above 1 bought no speedup on the same GPU (the separator is already compute-bound), so it stays at 1 as well.

Batched ASR is guarded against qwen-asr #207, where a mixed-length batch can corrupt its shorter item into a lone !: chunks are grouped by duration to keep each batch's lengths close, and any batched result that comes back implausibly empty for its chunk's duration is re-run alone through the per-chunk call (thresholds: VOXWEAVE_ASR_BATCH_MIN_CPS, VOXWEAVE_ASR_BATCH_MIN_CHECK_SEC). A batch whose call raises is likewise redone chunk by chunk, so one poisoned chunk degrades alone.

Shot-change detection needs no setting: it is a CPU-only ffmpeg pass that now starts before transcription and is joined at its workflow step, so it overlaps the GPU stages and the detect shot changes entry in the timing line is normally ~0s.

Data contract

Each input keeps its editable delivery set beside the media:

• <stem>.json — the source of truth: word/character-level segments, language, VAD speech, plus optional replay data (shot_changes, sing_spans, speaker_turns) so render can redo shot snapping, lyric flagging, and speaker formatting without re-running any model.
• <stem>.vtt — editable subtitles. By default cues carry word-level timestamps (same precision as align output, ready to use); --no-timestamps writes a plain-text editing draft for hand-correction, which align re-times.
• Subtitle-family deliverables — translated .vtt/.srt/.ass files and derived files such as <stem>.sdh.vtt and <stem>.asrfix.vtt also stay beside the media.

Other episode state is stored beside the media, under <media directory>/cache/<stem>/. For episode.mkv, the layout is:

cache/episode/
├── source.json                    # source file-name claim (never an absolute path)
├── episode.episode.lock           # episode transaction lock
├── .episode-domain.lock           # same-directory stem publication lock
├── speakers.json                  # reviewed diarizer-id-to-name mapping
├── speakers.suggest.json          # regenerable match suggestions
├── voiceprints.json               # optional biometric centroids
├── speaker-split.undo.json        # guarded one-level split undo
├── vocals.32k.flac                # separated-vocals cache
├── vocals.32k.flac.meta.json      # source-bound integrity companion, when required
├── vocals.32k.flac.lock           # vocals-cache lock
├── episode.zh.progress.json       # interrupted translation resume state
├── episode.align-evidence.json    # durable alignment evidence
├── episode.asrfix.json            # correction audit
└── debug/                         # optional --debug bundle

Because the claim records only the source file name, moving or renaming the media directory (local disk to NAS, season folder reorganizations) keeps every artifact valid — the cache travels with the media. The marker normally claims the media file; for a supported standalone subtitle command with no discoverable sibling media, it claims that input file instead. If another same-stem file in the same directory already owns the plain stem (episode.mkv next to episode.mp3), VoxWeave uses <stem>--<sha1-of-file-name-first-8>/ for the second claim.

Existing adjacent machine sidecars remain compatible: when an adjacent <stem>.speakers.json, <stem>.speakers.suggest.json, <stem>.voiceprints.json, translation progress file, alignment-evidence file, correction audit, or old cache/ vocals entry exists, VoxWeave reads it first and writes updates back to that same legacy lane. New artifacts use the cache. An existing adjacent <stem>.episode.lock is joined to the cache lock so old and new VoxWeave processes still serialize the same episode. The adjacent <stem>.json transcript is not a legacy exception: it remains the current source of truth by design. The explicit/discovered show-level voxweave.voices.json is a user-managed database and also stays where the user placed it.

Sensitive and derived speaker data

Treat voiceprint artifacts, speaker-split undo snapshots, show-level voices stores, suggestion records, in-memory auditions, calibration reports, crash-temporary files, and media snapshots as sensitive or derived data. The served audition contains embedded audio but is never saved by VoxWeave; a snapshot contains private media bytes rather than biometrics. Snapshots live under ${VOXWEAVE_CACHE_ROOT:-~/.cache/voxweave}/snapshots, are mode 0600 in an owner-only directory, and inactive crash residue older than one hour is cleaned on a later snapshot creation. Active snapshots are lock-protected from that janitor.

The purge command removes only the episode voiceprints, suggestion record, split undo snapshot, and any legacy audition HTML; it preserves the reviewed mapping, transcript, subtitles, and unrelated cache state. Remove a show voices store, calibration report, inactive snapshot residue, or crash-temporary file manually when it is no longer needed. Do not remove an active snapshot. Backups, filesystem snapshots, synced folders, and manual copies are outside VoxWeave's control and are not erased by purge; delete them separately according to their retention policy.

Both VTT forms are accepted by align. The aligner strips punctuation as a hard constraint; ASR punctuation is re-injected by time so the final output has correct spacing and breaks without stray marks.

Testing

• Unit tests (models mocked, no network): make test (= uv run --extra $(VARIANT) pytest tests/ -v)
• Lint / format: make lint

Support

If VoxWeave saves you time, you can support development here:

License

MIT — see LICENSE.

Acknowledgments

• WhisperX — the forced-alignment + edit-and-resync workflow this project builds on; the Japanese MMS full-file alignment path is a faithful port of its ctc align backend.
• stable-ts — inspiration for timestamp post-processing and documentation structure.
• Qwen3-ASR / Qwen3-ForcedAligner (Alibaba) — local ASR + aligner.
• MMS-300m (Meta) via ctc-forced-aligner — zero-OOV CJK alignment.
• Mel-Band Roformer (lucidrains) + KimberleyJSN weights — vocal separation.
• BudouX, jieba, PySBD — CJK/sentence line-break.
• PANNs — song/music detection.
• Silero VAD — voice activity detection.