subresync-app 0.1.2

Per-cue subtitle resynchronization: neural VAD + piecewise alignment + optional ASR word anchors, with a batch GUI.

SubResync

Fix out-of-sync subtitles - every sentence, not just one global shift.

SubResync re-times subtitle files against the actual speech in your video files. Unlike most tools, which find a single time-delta and shift everything, SubResync builds a piecewise correction map - it finds ad-break jumps, framerate mis-speeds and gradual drift, then fine-tunes each cue individually onto the detected speech. It works with any subtitle language against any audio language (Hebrew or Russian subtitles on English audio are first-class citizens), because the core engine matches the rhythm of speech, not the words.

Highlights

• Per-sentence correction - global offset + framerate fix, then piecewise segments (ad breaks / re-edits), then a per-cue snap onto speech onsets. Optional word-level refinement via speech recognition.
• Batch a whole folder - drop a season directory into the GUI; videos and subtitles are paired automatically (name stems, S01E03 patterns, language suffixes like .he.srt).
• Any formats - MKV/MP4/AVI/TS/… via FFmpeg; SRT/ASS/SSA/VTT/… via pysubs2, with automatic encoding detection (Hebrew cp1255, Cyrillic cp1251, UTF-8/16 all handled).
• Can't make it worse - every computed variant is scored against the audio; if nothing beats the original file, the original is left untouched and the file is flagged for review.
• SDH-aware - [music], (sighs), speaker labels are excluded from alignment (they have no audio correlate) but still re-timed in the output.
• Dual-audio aware - releases with an untagged dub track (e.g. Russian dub + English original in one AVI) are handled: when the detected audio language doesn't match the subtitle, each track is probed with Whisper language-ID and the matching one is used for word-level features.
• GPU-accelerated ASR - Max-precision mode and check --asr use the NVIDIA GPU automatically when available (~20-40× realtime on an RTX 3070), falling back to CPU otherwise.
• See what it's doing - the GUI has a live activity log (every stage of every file), a CPU/GPU compute indicator, a per-file Match score and a Sync column (verdict + residual in ms). Double-click any finished row to preview the before→after retiming, apply a manual nudge, or undo. See docs/SCORING.md.
• Zero setup - FFmpeg and the Silero VAD model download automatically on first run. No admin rights needed.

Install & run

Just want the app, no Python? Download SubResync-Setup-<version>.exe from the latest release, run it (no admin rights needed - it installs for your user only), and launch SubResync from the Start Menu like any other program. That's the whole setup - Max-precision mode works out of the box on CPU; if a GPU is detected, the app offers a one-time ~2 GB download (status bar - "Enable GPU acceleration") to speed it up. Building it yourself: build_exe.ps1 + build_installer.ps1 (needs Inno Setup).

Developing / running from source:

# from the project folder
py -3.11 -m venv .venv
.venv\Scripts\python -m pip install -e .[gui,asr,dev]

# optional: GPU acceleration for Max mode / --asr (no CUDA Toolkit needed)
.venv\Scripts\python -m pip install -e .[cuda]
# ...or, from a plain [asr] install, let the app fetch the same libs itself:
.venv\Scripts\subresync setup --gpu

# GUI
.venv\Scripts\subresync-gui        # or double-click SubResync.bat

# CLI
.venv\Scripts\subresync setup                                # pre-download FFmpeg/model, show GPU status
.venv\Scripts\subresync sync "D:\TV\Show Season 1"           # whole folder
.venv\Scripts\subresync sync video.mkv subs.srt              # single pair
.venv\Scripts\subresync sync "D:\TV\Season 1" --mode auto    # Balanced, escalate hard files
.venv\Scripts\subresync sync . --mode max --json             # always run ASR
.venv\Scripts\subresync pairs "D:\Movies"                    # preview pairing
.venv\Scripts\subresync check "D:\TV\Show Season 1"          # verify sync quality
.venv\Scripts\subresync check . --synced                     # verify the outputs
.venv\Scripts\subresync check video.mkv subs.srt --asr       # word-level check

check measures the local agreement between the subtitle and the actual speech in every ~60 s window of the timeline and grades each window (ok / loose / off), so you can see exactly which minutes of a file are out of sync and by how much. With --asr it additionally transcribes the audio and verifies that each cue's words are spoken where the cue says (same-language subtitles only) - the strictest possible ground-truth check.

Output goes next to the video as <video-name>.synced.<lang>.srt (original files are never touched unless you enable overwrite, which keeps a .bak). Re-running a folder never deletes previous outputs - a second pass writes …synced.<lang>.v2.srt, …v3.srt, and so on, so you can compare a stronger mode against an earlier run. Each run writes one timestamped HTML report (subresync-report-<time>.html) with every file's offsets, segments, scores and residuals - open it in the app (the Report button) or any browser, or ignore it. (No more one .sync.json per episode.)

Modes

Mode	Stages	When
Fast	global + piecewise	Quick pass, large libraries
Balanced	+ per-cue snap	Solid default, no speech recognition
Auto (recommended)	Balanced, then Max only on still-off files	Best quality-per-time on a library
Max precision	always run speech-recognition word anchors	Same-language subs, worst cases

Auto runs Balanced on everything, then automatically re-runs only the files that are still out of sync in Max - so you get Max quality on the hard files without paying the speech-recognition cost on the easy ones. Max (and the Max pass inside Auto) automatically falls back to Balanced when the subtitle language doesn't match the audio language (e.g. Hebrew subs on English audio) - that is by design; cross-language sync uses speech rhythm only.

How it works

1. Extract & detect speech. FFmpeg decodes the best audio track to 16 kHz mono; Silero VAD (neural, ONNX) produces a speech-probability signal on a 10 ms grid. Cached by content hash - re-runs are instant.
2. Global fit. FFT cross-correlation finds the dominant offset and tests framerate ratios (23.976↔25 etc.), with golden-section refinement.
3. Piecewise fit. Sliding windows measure the local residual offset; confident windows become anchors; jumps open new segments. Segment boundaries are re-optimized over cue-order splits with a collision penalty - this is what fixes ad-break / director's-cut discontinuities.
4. Per-cue snap. Each sentence is nudged (±0.6 s) to maximize speech coverage inside the cue and minimize it just outside, with outlier clamping against neighbors.
5. (Max mode) Word anchors. faster-whisper transcribes with word timestamps (GPU when available); cue texts are fuzzy-matched to the transcript; matched cues pin to the exact spoken word, the rest interpolate. If the audio language doesn't match the subtitle and the file has more audio tracks, each track is probed and the matching one is used.
6. Verify & write. All variants are scored against the audio signal; the best one is written in the original subtitle format (UTF-8).

Project layout

For a full guided tour (every file's responsibility, the data flow through the pipeline, a "where do I change X" index), see ARCHITECTURE.md.

src/subresync/
  config.py       settings & mode presets
  pipeline.py     stage orchestration, quality gates, auto-escalation
  verify.py       sync-quality checker (windowed + word-level)
  report.py       one consolidated HTML report per run
  pairing.py      video<->subtitle discovery (stems, S01E01, languages)
  batch.py        parallel batch engine
  cli.py          command line
  gpu.py          CUDA DLL registration, GPU detection + on-demand cuBLAS/cuDNN download
  certs.py        TLS fix for antivirus HTTPS interception (Avast etc.)
  media/          ffmpeg locate/download/extract, signal cache
  subtitles/      format I/O, encoding detection, SDH classifier
  vad/            Silero ONNX VAD + energy fallback
  align/          global FFT fit, piecewise segments, per-cue snap,
                  ASR word anchors, final warp
  gui/            PySide6 app (drag-drop, activity log, CPU/GPU chip, scores)
tests/            synthetic-warp recovery tests + unit tests
scripts/          bench_modes.py - score all modes on one pair
docs/PLAN.md          architecture & design decisions
docs/SCORING.md       what the match score means
docs/COMPARISON.md    modes vs each other, and vs other tools
LICENSE               MIT
THIRD_PARTY_NOTICES.md dependencies + prior-art acknowledgments

Development

.venv\Scripts\python -m pytest          # test suite

The synthetic tests generate known desyncs (offset, framerate, mid-file break, per-cue jitter) and assert the engine recovers them to within tens of milliseconds. An end-to-end harness with real (TTS) speech lives in the dev scratchpad; median recovery error there is ~64 ms with a 9 s mid-file break correctly detected and placed.

Real-world validation

Measured on a full TV season (23 × 43-min AVI episodes, dual-audio release with a Russian dub + English original, English subtitles from a different edit):

• Every episode required piecewise correction: per-episode global offsets from −9.9 s to +3.7 s plus 2–15 local segments each. One episode's desync was a staircase (−10 s → −2 s in ~2 s steps at commercial-break cuts) - unfixable by any single-delta tool.
• check --synced on all outputs: median residuals 0–25 ms, every file graded in-sync / mostly-in-sync.
• Word-level ground truth (check --asr, GPU): 514/588 cues found verbatim in the audio at their subtitle position, median timing error 206 ms - for files synced by rhythm alone.

How it compares

On S02E01, the best possible single global correction (what global-only tools do) reaches match 0.66 and leaves 13 segments still 1–2 s out; SubResync's piecewise + per-cue reach 0.75 / 20 ms median / 94 % of windows in sync. SubResync is the only open-source tool that combines global + framerate + piecewise + per-cue correction with optional ASR anchoring and a "can't-make-it-worse" guarantee. Full breakdown and a capability matrix vs ffsubsync / alass / subaligner / WhisperX / AutoSubSync / subsyncarr: docs/COMPARISON.md.

License

MIT - see LICENSE. All engine code is original; the alignment approach was inspired by prior open-source work (ffsubsync, alass, WhisperX, subaligner) but no code was copied, so the GPL terms of alass/AutoSubSync do not apply. Dependency licenses and acknowledgments: THIRD_PARTY_NOTICES.md.