voxweave 0.4.1

BGM-robust subtitles for anime / film / clips: vocal separation + song-skip so ASR doesn't hallucinate on background music, OP/ED, or insert songs. Local-first Qwen3 ASR + forced alignment + edit-and-resync, CJK-aware.

VoxWeave

BGM-robust subtitles for anime, film, and clips.

Vocal separation and song-skip so ASR never hallucinates on background music, OP/ED, or insert songs. Local-first Qwen3 ASR, forced alignment, and edit-and-resync — CJK-aware.

https://github.com/user-attachments/assets/e75b6dd3-fa37-4afe-89db-b6ee2c28f6bc

_{Sliced clip under heavy BGM · voxweave Test.mp4 · Qwen3-ASR-1.7B}

[!NOTE] 100% local. Separation, ASR, and forced alignment all run in-process on your GPU — no network endpoints, no audio leaves the machine. Runs on NVIDIA CUDA (PyTorch) and on Apple Silicon, where ASR + alignment use the native MLX Qwen3 models. Weights download once on first run. (Translation and ASR-correction are the only optional features that call an external LLM, and only when you invoke them.)

VoxWeave derives from the WhisperX "edit-and-resync" workflow: transcribe once, then edit the text and re-align it against the original audio for frame-accurate timestamps. Where it differs is the front end — vocal separation and song-skip keep background music out of the ASR, and a CJK-aware layout/alignment stack (MMS-300m for Japanese, BudouX/jieba for line breaks) handles Chinese/Japanese/English as first-class.

Contents

• Why VoxWeave
• Setup
• Quickstart
• Usage
  • Transcribe (voxweave <media>)
  • Re-align after editing (align)
  • Re-layout offline (split)
  • ASR correction (correct)
  • Translate (translate)
  • Export (export)
  • Pack soft subtitles (pack)
  • Burn hard subtitles (burn)
• The edit-and-resync workflow
• How it works
• Configuration
• Data contract
• Testing
• Support
• License
• Acknowledgments

Why VoxWeave

• BGM removal before ASR. A Mel-Band Roformer vocal separator (pure torch, full-band 44.1k) strips music first, so ASR doesn't transcribe lyrics or hallucinate on score.
• Song-skip. PANNs detects singing/music on the separated vocals and skips OP/ED and insert songs before ASR — on by default, --no-skip-songs to keep them.
• Local Qwen3 ASR + forced alignment. Text and word-level timestamps in one pass, fully on-device — in-process PyTorch on NVIDIA, or the native MLX Qwen3 models on Apple Silicon. A Whisper hybrid engine is also available for when you prefer Whisper text (faster-whisper on NVIDIA, the native MLX Whisper port on Apple Silicon).
• Edit-and-resync. Fix the transcript by hand, then align re-derives timestamps from the audio — timestamps are never hand-written.
• CJK-aware. Japanese aligns with MMS-300m + uroman (zero-OOV, immune to the per-cue drift that breaks wav2vec2-xlsr on rare kanji); line breaks use BudouX phrase atoms + jieba.
• Optional LLM steps. correct cleans up ASR typos/garbled names before alignment; translate does whole-episode context-aware translation while preserving cue count.
• Ship the result. pack soft-muxes finished VTTs into the media as titled subtitle tracks (instant stream copy); burn hardcodes them at constant quality with NVENC / VideoToolbox acceleration, matching the source bit depth.

Setup

Two install variants: voxweave[cuda] (NVIDIA GPU — Blackwell sm_120 / cu128 by default) and voxweave[mps] (Apple Silicon / macOS). Both need ffmpeg on PATH.

Install ffmpeg

# Ubuntu / Debian
sudo apt update && sudo apt install ffmpeg
# Arch Linux
sudo pacman -S ffmpeg
# macOS (Homebrew)
brew install ffmpeg

CUDA / PyTorch notes

On the [cuda] variant the torch wheel is pinned to the cu128 build (Blackwell sm_120) and installed into an isolated uv tool venv. The CUDA toolkit does not need to be installed separately — the cu128 wheel bundles the required runtime libraries; only an NVIDIA driver is required on the host. The [mps] variant uses the default PyPI torch wheel (Metal/MPS built in). Override the torch index per-invocation: make install TORCH_BACKEND=cpu.

Install from PyPI (puts the global voxweave command on PATH):

# NVIDIA / Linux:
uv tool install --torch-backend=cu128 "voxweave[cuda]"   # full pipeline + faster-whisper hybrid
# Apple Silicon / macOS:
uv tool install "voxweave[mps]"                          # full pipeline + MLX Whisper hybrid

The full local pipeline — vocal separation, ASR, forced alignment (incl. MMS-300m for Japanese/CJK), layout, song-skip — plus CJK line-break and translation are baked into the core dependencies. The variant selects the compute platform and the ASR/alignment backend:

• [cuda] (NVIDIA/Linux): the in-process PyTorch Qwen3-ASR + forced aligner (qwen-asr), GPU onnxruntime (CUDAExecutionProvider for MMS alignment), and the faster-whisper hybrid engine.
• [mps] (Apple Silicon/macOS): ASR runs on the native MLX Qwen3-ASR from mlx-audio (Metal kernels + quantization). Alignment keeps the same per-language stack as [cuda]: English on wav2vec2 CTC (torch, runs on MPS; the forced-align DP falls to CPU as torchaudio has no Metal kernel), Japanese/CJK on the ONNX MMS aligner (CoreML/CPU — onnxruntime has no Metal provider). Only the Qwen fallback (zh·yue, or any CTC failure) is served by the MLX Qwen3-ForcedAligner, since the torch qwen-asr aligner is absent here. The Whisper hybrid/fusion engines (--model large-v3, --hybrid) run on the native mlx-whisper Metal port instead of faster-whisper (ctranslate2 has no Metal backend). Vocal separation (MelBandRoformer) + PANNs song-skip stay on torch-MPS. qwen-asr is excluded because its transformers==4.57.6 pin conflicts with mlx-audio, so [cuda] and [mps] are mutually exclusive — pick one per host.

From source (for development or pulling new code):

make cuda          # NVIDIA/Linux  — uv tool install --torch-backend=cu128 ".[cuda]"
make mps           # Apple Silicon — force-reinstall ".[mps]" (always picks up new source)
make reinstall     # after pulling new code (honours VARIANT, default cuda)
make uninstall

Extras & what each pulls

• The core pulls qwen-asr (hard-pins transformers==4.57.6 + accelerate==1.12.0) + a pure-torch Mel-Band Roformer vendored in voxweave.vendor (no onnx/onnxruntime — audio-separator is intentionally avoided because it eagerly imports onnxruntime at the top level) + MMS-300m forced aligner (ctc-forced-aligner) + layout (pysbd) + song-skip (panns-inference) + CJK break (budoux + jieba) + translation (openai).
• [cuda] (NVIDIA/Linux): qwen-asr + onnxruntime-gpu + faster-whisper. [mps] (Apple Silicon/macOS): mlx-audio + plain onnxruntime. Declared conflicting in [tool.uv] (incompatible transformers pins), so uv resolves each in its own fork — pick one per host (make dev VARIANT=mps on Apple Silicon).
• The device is auto-detected at runtime (cuda → mps → cpu); override with VOXWEAVE_DEVICE. On mps the MLX backend is selected automatically; force it either way with VOXWEAVE_BACKEND=mlx|torch.
• Development: make dev (= uv sync --extra cuda --dev; on Apple Silicon use make dev VARIANT=mps — [cuda]/[mps] are conflicting extras and can't be synced together).

Quickstart

# Transcribe a video to a timestamped VTT (+ a JSON source of truth)
voxweave episode.mkv

# ...edit episode.vtt by hand (fix wording, line breaks)...

# Re-align the edited text against the original audio
voxweave align episode.vtt

# Optionally translate the aligned subtitles to Chinese
voxweave translate episode.vtt --to zh

Usage

Transcribe

voxweave <media> — separation → song-skip → VAD chunking → ASR + forced alignment → smart_split → writes <stem>.vtt (editable) + <stem>.json (word-level timestamp source of truth). Models load in-process (see voxweave.backend); the separator is released from VRAM before ASR+alignment load, so peak usage is ≈ max(sep, asr) rather than their sum.

voxweave episode.mkv
voxweave clip.mp4 --no-separate          # clean speech (podcast/lecture): skip separation
voxweave episode.mkv --model qwen3-asr-1.7B   # larger, more accurate ASR
voxweave episode.mkv --context "Ryland Grace, Astrophage, Hail Mary"   # bias names/terms

Options

Option	Description
--language	Force language (ISO code or full name); default auto-detect.
--no-separate	Skip vocal separation (for clean speech) to save GPU time.
--no-skip-songs	Keep lyrics / transcribe purely musical content (song-skip is on by default).
--model	Local ASR model (default Qwen3-ASR-0.6B; qwen3-asr-1.7B is more accurate).
--context	ASR bias prompt: names/terms likely to appear (comma or newline separated). Bare term lists are auto-framed as Proper nouns: ... for Qwen — a bare list actually regresses accuracy (details); prose or pre-framed text passes through.
--hybrid	Dual-ASR fusion: Whisper text + Qwen punctuation. Whisper's error bias is the opposite of Qwen's (it hallucinates rather than omits), so use this when Qwen drops uncertain words.
--normalize	Apply loudness normalization (loudnorm) to the 16k ASR input — helps when quiet words get dropped; off by default since it also amplifies noise.
--timestamps/--no-timestamps	VTT carries word-level timestamps (default on); --no-timestamps writes a plain-text editing draft.
--keep-lyrics	Transcribe detected songs instead of skipping them; sung cues are wrapped ♪ ... ♪ (italic in ASS export).
--sdh	Also write <stem>.sdh.vtt: PANNs non-speech event tags ([explosion], [phone ringing], ...) in speech-free gaps.
--diarize	pyannote speaker diarization: two-speaker cues become dual-speaker events (-line per speaker). Needs voxweave[diarize] + an HF token (VOXWEAVE_HF_TOKEN) for the gated checkpoint.
--no-shot-snap	Disable shot-change detection/snapping (cue boundaries otherwise land on cuts per the Netflix zone rules).
--debug	Write intermediate artifacts (full-band / vocals / per-chunk VAD + ASR + alignment) to debug/<stem>/.

Re-align after editing

voxweave align <vtt> — takes the edited VTT text and re-runs forced alignment against the original audio, overwriting the timestamped VTT and updating the JSON. Does not re-run ASR or touch smart_split. Aligns on separated 16k vocals by default (prevents BGM interference); prefers a cached cache/<stem>.16k.flac, otherwise re-separates and caches.

voxweave align episode.vtt                 # finds episode.<ext> in the same dir
voxweave align episode.vtt --media original.mkv
voxweave align episode.vtt --no-separate   # align on the original audio (clean sources)

Options

Option	Description
--media	Source media path (default: same-name file in the same directory).
--language	Force language (ISO code or full name); default: read from JSON.
--no-separate	Align on the original audio instead of separated vocals.
--normalize	Apply loudnorm to the 16k alignment input.

Re-layout offline

voxweave split <json> — re-run smart_split from <stem>.json without any models (adjust line width / sentence breaks instantly).

voxweave split episode.json --max-line-length 14 --max-lines 1
voxweave split episode.json --no-timestamps   # plain-text editing draft

ASR correction

voxweave correct <vtt> — optional pre-align LLM pass that fixes obvious ASR typos, split words, and garbled proper nouns, producing a reviewable diff. Conservative substitution only (no completion/rewrite), gated by a code check that the matched text equals the original line-for-line. By default writes only a sidecar <stem>.asrfix.vtt + audit JSON — the original VTT is untouched. Use --apply to overwrite, then run align to reassign timing.

voxweave correct episode.vtt --glossary names.json   # review the sidecar
voxweave correct episode.vtt --glossary names.json --apply
voxweave align episode.vtt

Options

Option	Description
--glossary	Term/name glossary (.json → mapping; other → raw prompt). Strongly recommended for ambiguous proper nouns.
--apply	Overwrite the original VTT (default: sidecar only, for review).
--model	Correction model (default VOXWEAVE_FIX_MODEL env or gpt-5.3-chat-latest).
--base-url / --api-key-env	OpenAI-compatible endpoint + which env var holds the key.

Translate

voxweave translate <vtt> — after align, translate each cue with whole-episode context, preserving cue count, into <stem>.<to>.vtt (the original is left unchanged).

voxweave translate episode.vtt --to zh
voxweave translate episode.vtt --to en --context "sci-fi, formal register" --glossary terms.json

Options

Option	Description
--to	Target language code, written to <stem>.<to>.vtt (default zh).
--context	Show/tone context injected into the prompt.
--glossary	Term/name glossary (.json → mapping; other → raw prompt).
--model	Translation model (default VOXWEAVE_TRANSLATE_MODEL env or gpt-5.3-chat-latest).
--base-url / --api-key-env	OpenAI-compatible endpoint + which env var holds the key.

Export

voxweave export <vtt> — render an aligned VTT into SRT and/or ASS next to it (the VTT + JSON pair stays the source of truth). ASS carries a Default style; lyric cues (♪ ... ♪) render italic.

voxweave export episode.vtt --to srt
voxweave export episode.vtt --to srt --to ass

Pack (soft subtitles)

voxweave pack <vtt>... — remux the source media with the VTT(s) added as proper subtitle tracks. Pure stream copy (instant, lossless, reversible); each track is titled VoxWeave <Language> with the container language tag taken from the VTT filename (episode.zh.vtt → chi / "VoxWeave Chinese"), and the first packed track is flagged default so players select it.

voxweave pack episode.zh.vtt                    # finds episode.<ext>, keeps its container
voxweave pack episode.zh.vtt episode.ja.vtt     # several tracks at once
voxweave pack episode.zh.vtt --to mp4           # mov_text in mp4 (image subs are dropped)
voxweave pack episode.zh.vtt --media other.mkv -o out.mkv

mkv targets keep every source stream (including attachments); mp4/webm targets keep video+audio and existing text subtitle tracks only. HEVC video muxed into mp4 is tagged hvc1 for Apple players.

Burn (hard subtitles)

voxweave burn <vtt> — render the subtitles into the pixels and write a clean file with all subtitle tracks removed. A styled ASS is generated at the actual frame size (same look as export, lyric cues italic), then the video is re-encoded at constant quality with hardware acceleration when available: NVENC on NVIDIA, VideoToolbox on macOS, libx264/libx265/libsvt-av1 software fallback. Audio is stream-copied (mp4 targets re-encode mp4-incompatible codecs to AAC).

voxweave burn episode.zh.vtt                          # hevc, auto hw encoder, -> episode.mp4
voxweave burn episode.zh.vtt --codec h264             # legacy-device compatibility
voxweave burn episode.zh.vtt --codec av1 --to mkv     # max compression, recent hardware
voxweave burn episode.zh.vtt --quality 20 --font "Noto Sans CJK SC"

Burn options & encoding policy

Option	Meaning
--codec	hevc (default: 10-bit capable, ~40% smaller than h264, plays everywhere as hvc1 mp4) / h264 / av1.
--encoder	Force a specific ffmpeg encoder (default: auto-probe with a test encode).
--quality	Constant quality: NVENC -cq / software -crf (lower = better); VideoToolbox -q:v (higher = better).
--to	mp4 (default, maximum compatibility) or mkv.
--font	Subtitle font family (fontconfig resolves fallbacks; e.g. Noto Sans CJK SC).
--font-size	Override the default 72-at-1080p scaled size.

Bitrate is never targeted: pure constant-quality (-b:v 0 on NVENC) lets the encoder spend bits where the content needs them, with no overshoot against the source rate. Output bit depth follows the source dynamically (8-bit stays 8-bit, 10-bit stays 10-bit; 12-bit is kept on libx265 and clamped to 10 on NVENC/VideoToolbox/SVT-AV1, which top out there) — except on h264 paths, which are always 8-bit for player compatibility (NVENC h264 cannot encode 10-bit at all).

Progress is rendered with rich: countable stages (demix windows / PANNs batches / per-chunk ASR+alignment / align per-cue / translate streaming per-line) show a real x/N bar with elapsed time; indeterminate stages (decode / file write) show a pulse bar. -v/--verbose enables DEBUG logging.

The edit-and-resync workflow

voxweave episode.mkv          # 1. transcribe  -> episode.vtt + episode.json
  └─ (optional) correct       # 2. LLM ASR fix -> episode.asrfix.vtt (--apply to commit)
edit episode.vtt by hand      # 3. fix wording / line breaks
voxweave align episode.vtt    # 4. re-derive timestamps from audio (overwrites VTT + JSON)
voxweave translate episode.vtt --to zh   # 5. context-aware translation
voxweave pack episode.zh.vtt             # 6. soft-mux into the media (or burn for hardsubs)

Timestamps are always derived from the audio by the forced aligner — you never hand-edit them. Edit the text freely; align puts the timing back.

How it works

Stage	What runs
Separation	Mel-Band Roformer (full-band 44.1k stereo, vendored pure-torch) isolates vocals; downsampled to 16k afterwards.
Song-skip	PANNs (route ii) flags singing/music on the separated vocals before ASR.
Chunking	Silero VAD splits speech into ≤120s chunks (longer risks ASR repetition-loop collapse).
ASR + align	Qwen3-ASR (default, text + units in one pass) / Whisper hybrid (faster-whisper on cuda, mlx-whisper on mps) / dual-ASR fusion — the pipeline is engine-agnostic.
Alignment	ja → MMS-300m + uroman (full-file single pass, WhisperX-gold); en → wav2vec2-LV60K CTC per-cue; zh·yue → Qwen.
Layout	gap-aware smart_split: word-level gaps + BudouX phrase atoms + line-length, on a shared timeline forked per language.

Configuration

Precedence: CLI flag > env var > ~/.config/voxweave.conf > built-in default. A commented default config is written on first run (migrated automatically from a pre-rename qsub.conf).

Environment variables

Models

• VOXWEAVE_ASR_MODEL (default Qwen/Qwen3-ASR-0.6B; same as --model)
• VOXWEAVE_ALIGNER_MODEL (default Qwen/Qwen3-ForcedAligner-0.6B)
• VOXWEAVE_DEVICE (default: auto-detect cuda:0 → mps → cpu)
• VOXWEAVE_BACKEND (mlx | torch; default: mlx on mps, else torch) — picks the ASR/alignment backend
• VOXWEAVE_OFFLINE (1 to enable) — once all models are cached, sets HF_HUB_OFFLINE/TRANSFORMERS_OFFLINE so loading skips the per-file HEAD revalidation + optional-file probing huggingface_hub/transformers otherwise do on every run (no network on a cache hit). Leave off for the first download.
• VOXWEAVE_MLX_ASR_REPO / VOXWEAVE_MLX_ALIGNER_REPO / VOXWEAVE_MLX_WHISPER_REPO — MLX backend repos. By default the ASR repo tracks --model size (--model 1.7b → mlx-community/Qwen3-ASR-1.7B-8bit) and the Whisper repo tracks the Whisper size (--model large-v3 → mlx-community/whisper-large-v3-mlx); set the matching var to hard-pin a specific quant (e.g. a 4-bit build) regardless of --model.

All model weights (torch + MLX) are cached under ~/.cache/voxweave/{asr,align,audio} (auto-downloaded on first use; override the root with VOXWEAVE_CACHE_ROOT), so a container only needs to bind-mount that one directory. Each model exposes an env override to swap the HF repo, or to point at an explicit local file (which, if it exists, skips the HF download):

• VOXWEAVE_SEPARATOR_REPO / VOXWEAVE_SEPARATOR_REPO_FILE (default KimberleyJSN/melbandroformer / MelBandRoformer.ckpt), or VOXWEAVE_SEPARATOR_CKPT / VOXWEAVE_SEPARATOR_CONFIG for explicit weights + matching yaml
• VOXWEAVE_PANNS_REPO / VOXWEAVE_PANNS_REPO_FILE (default thelou1s/panns-inference / Cnn14_mAP=0.431.pth), or VOXWEAVE_PANNS_CKPT for an explicit checkpoint (song-skip CNN)
• VOXWEAVE_MMS_REPO / VOXWEAVE_MMS_REPO_FILE (default deskpai/ctc_forced_aligner / 04ac86b67129634da93aea76e0147ef3.onnx), or VOXWEAVE_MMS_MODEL for an explicit onnx path (Japanese/CJK MMS-300m aligner)

Tuning

• VOXWEAVE_MAX_CHUNK_SEC (default 120; shorter chunks reduce ASR repetition loops on long segments)
• VOXWEAVE_LOUDNORM (default loudnorm=I=-16:TP=-1.5:LRA=11; the -af filter for --normalize)
• VOXWEAVE_MIN_CUE_SEC (default 0.8; minimum cue display duration in align)
• VOXWEAVE_SNAP_VAD_THRESHOLD (default 0.25; sensitive VAD used when repositioning zero-duration units against the original audio)

Config file (~/.config/voxweave.conf, TOML)

Every key below is optional — delete a line to fall back to its built-in default. The values shown are a usable starting point, not the defaults (the auto-written template has everything commented out).

# ~/.config/voxweave.conf  —  TOML
# Precedence: CLI flag > env var > this file > built-in default.

# Default ASR model (= --model). Short name (qwen3-asr-0.6b | qwen3-asr-1.7b) or full HF id.
# Special value "hybrid" (= --hybrid) -> dual-ASR fusion (whisper text + Qwen punctuation).
asr_model = "Qwen/Qwen3-ASR-1.7B"        # built-in default: Qwen/Qwen3-ASR-0.6B

# Model load strategy:
#   "peak" (default) — serial peak-shaving: all-chunk ASR -> release -> all-chunk align;
#                      ASR and aligner never co-reside, peak VRAM = max(models). Works on 8 GB.
#   "sum"            — concurrent per-chunk ASR+align; peak VRAM = sum(models), but skips two
#                      model swap round-trips (faster on large-VRAM cards).
load_strategy = "sum"

# dual-ASR fusion sub-models — only consulted when running with --hybrid.
[fusion]
whisper = "large-v3-turbo"               # Whisper size: large-v3 (best) | large-v3-turbo (~5x faster); faster-whisper on cuda, mlx-whisper on mps
qwen    = "Qwen/Qwen3-ASR-1.7B"          # punctuation model; must emit punctuation -> 1.7B, not 0.6B

# Per-language forced-alignment model. Key = ISO-639-1 code; unlisted languages use Qwen3-ForcedAligner.
# Values:
#   "mms"   — MMS-300m + uroman, full-file single pass (immune to per-cue drift; the gold standard).
#   HF id   — wav2vec2 CTC via HF transformers; weights land in ~/.cache/voxweave/align (per-cue crop).
#   bundle  — torchaudio bundle name, e.g. "WAV2VEC2_ASR_LARGE_LV60K_960H" (same model, cached in ~/.cache/torch).
#   ""      — explicitly fall back to Qwen for that language.
[align]
en = "facebook/wav2vec2-large-960h-lv60-self"  # English: LV60K-self CTC, per-cue crop (HF hub)
ja = "mms"                                      # Japanese: MMS-300m + uroman full-file (= whisperx fork align_ctc)
# zh  = "mms"                                   # Chinese can also use MMS; default is Qwen (native CJK char-level)
# yue = ""                                      # force Qwen for Cantonese

Data contract

Each input produces two sibling files:

• <stem>.json — the source of truth: word/character-level segments, language, VAD speech, plus optional replay data (shot_changes, sing_spans, speaker_turns) so split can redo shot snapping, lyric flagging, and speaker formatting without re-running any model.
• <stem>.vtt — editable subtitles. By default cues carry word-level timestamps (same precision as align output, ready to use); --no-timestamps writes a plain-text editing draft for hand-correction, which align re-times.

Both VTT forms are accepted by align. The aligner strips punctuation as a hard constraint; ASR punctuation is re-injected by time so the final output has correct spacing and breaks without stray marks.

Testing

• Unit tests (models mocked, no network): make test (= uv run pytest tests/)
• Lint / format: make lint

Support

If VoxWeave saves you time, you can support development here:

License

MIT — see LICENSE.

Acknowledgments

• WhisperX — the forced-alignment + edit-and-resync workflow this project builds on; the Japanese MMS full-file alignment path is a faithful port of its ctc align backend.
• stable-ts — inspiration for timestamp post-processing and documentation structure.
• Qwen3-ASR / Qwen3-ForcedAligner (Alibaba) — local ASR + aligner.
• MMS-300m (Meta) via ctc-forced-aligner — zero-OOV CJK alignment.
• Mel-Band Roformer (lucidrains) + KimberleyJSN weights — vocal separation.
• BudouX, jieba, PySBD — CJK/sentence line-break.
• PANNs — song/music detection.
• Silero VAD — voice activity detection.