ffsubsync 0.5.0

No project description provided

FFsubsync

Language-agnostic automatic synchronization of subtitles with video, so that subtitles are aligned to the correct starting point within the video.

Turn this:	Into this:

Helping Development

Please consider supporting Ukraine rather than donating directly to this project. That said, at the request of some, you can now help cover my coffee expenses using the Github Sponsors button at the top, or using the below Paypal Donate button:

Install

First, make sure ffmpeg is installed. On MacOS, this looks like:

brew install ffmpeg

(Windows users: make sure ffmpeg is on your path and can be referenced from the command line!)

Next, grab the package (compatible with Python >= 3.6):

pip install ffsubsync

If you want to live dangerously, you can grab the latest version as follows:

pip install git+https://github.com/smacke/ffsubsync@latest

Usage

ffs, subsync and ffsubsync all work as entrypoints:

ffs video.mp4 -i unsynchronized.srt -o synchronized.srt

There may be occasions where you have a correctly synchronized srt file in a language you are unfamiliar with, as well as an unsynchronized srt file in your native language. In this case, you can use the correctly synchronized srt file directly as a reference for synchronization, instead of using the video as the reference:

ffsubsync reference.srt -i unsynchronized.srt -o synchronized.srt

ffsubsync uses the file extension to decide whether to perform voice activity detection on the audio or to directly extract speech from an srt file.

If you omit -i, ffsubsync auto-detects input subtitles sitting next to the reference that share its name, so you can simply run:

ffs video.mp4

This picks up files like video.srt and video.en.srt from the reference's directory and writes the synced result for each to a <name>.synced.srt alongside it (e.g. video.synced.srt), leaving the originals untouched. Previously-synced *.synced.srt outputs are skipped, so re-running is safe. Add --overwrite-input to overwrite the detected file(s) in place instead. Auto- detection is skipped when subtitles are piped in on stdin.

The reference can also be a remote URL instead of a local file. Anything ffmpeg can read directly works as a video / audio reference, and remote subtitle files work as a reference too:

ffs "https://example.com/video.mp4" -i unsynchronized.srt -o synchronized.srt
ffs "https://example.com/reference.srt" -i unsynchronized.srt -o synchronized.srt

Supported protocols are http(s)://, rtmp://, rtsp://, and ftp://. Processing streams the reference over the network, so it depends on connection stability; for large or flaky sources, downloading first is more reliable. Sibling-subtitle auto-detection (the no--i form above) is local-only and is skipped for remote references.

To speed up long references, --max-duration-seconds N processes only the first N seconds (measured from --start-seconds). This is especially helpful for remote references, since ffmpeg stops reading—and therefore downloading—once that duration is reached:

ffs "https://example.com/video.mp4" -i unsynchronized.srt -o synchronized.srt --max-duration-seconds 600

On flaky connections, --extract-audio-first can be more reliable: instead of holding a network stream open throughout speech detection, it first copies the remote audio track to a local temp file (no re-encode) and runs detection on that. It is ignored for local references and composes with --max-duration-seconds:

ffs "https://example.com/video.mp4" -i unsynchronized.srt -o synchronized.srt --extract-audio-first

For long references where --max-duration-seconds would miss desync that only shows up later, --multi-segment-sync instead samples several short segments spread across the whole reference and runs speech detection on just those:

ffs "https://example.com/video.mp4" -i unsynchronized.srt -o synchronized.srt --multi-segment-sync

Only the sampled audio is extracted (and, for remote references, downloaded), but because each segment keeps its true position on the timeline, the usual framerate-ratio and offset search is unchanged—so a framerate mismatch is still detected and corrected. Tune it with --segment-count N (default 8), --skip-intro-outro (skip the first 30s / last 60s, which often lack dialogue), and --parallel-workers N (overlap segment downloads, default 4). It applies to video / audio references only.

Docker

Prebuilt images are published to the GitHub Container Registry. Pull the latest release with:

docker pull ghcr.io/smacke/ffsubsync:latest

Run it by mounting the directory with your video and subtitles into /video:

docker run --rm -v "$PWD":/video ghcr.io/smacke/ffsubsync:latest \
  video.mp4 -i unsynchronized.srt -o synchronized.srt

You can also build the image yourself. The multi-stage Dockerfile defaults to installing from the current working tree:

docker build -t ffsubsync .

To install a specific version from PyPI instead, set FFSUBSYNC_VERSION:

docker build -t ffsubsync --build-arg FFSUBSYNC_VERSION=0.4.31 .

Using as a Library

ffsubsync can be driven programmatically via ffsubsync.run, which accepts an argparse.Namespace (build one with make_parser). To surface progress in your own UI, pass a progress_handler callback; it is invoked repeatedly while the reference's audio is being decoded with a ProgressInfo describing how far along the extraction is:

import ffsubsync
from ffsubsync.ffsubsync import make_parser

def on_progress(info: ffsubsync.ProgressInfo) -> None:
    # info.processed_seconds / info.total_seconds (total may be None);
    # info.fraction is a 0.0-1.0 ratio (None if the total is unknown).
    if info.fraction is not None:
        print(f"{info.fraction:.0%}")

args = make_parser().parse_args(["ref.mkv", "-i", "in.srt", "-o", "out.srt"])
result = ffsubsync.run(args, progress_handler=on_progress)

The handler is called only for the video / audio reference path (the dominant cost of a sync). Exceptions it raises are logged and swallowed, so a buggy handler can never abort syncing.

Sync Issues

If the sync fails, the following recourses are available:

• Try to sync assuming identical video / subtitle framerates by passing --no-fix-framerate;
• Try passing --gss to use golden-section search to find the optimal ratio between video and subtitle framerates (by default, only a few common ratios are evaluated);
• Try a value of --max-offset-seconds greater than the default of 60, in the event that the subtitles are out of sync by more than 60 seconds (empirically unlikely in practice, but possible).
• Try --vad=auditok since auditok can sometimes work better in the case of low-quality audio than WebRTC's VAD. Auditok does not specifically detect voice, but instead detects all audio; this property can yield suboptimal syncing behavior when a proper VAD can work well, but can be effective in some cases.
• Try --vad=fused, which combines WebRTC with the neural silero VAD and can be more robust on noisy audio. The strategy can be tuned: --vad=fused:intersection (conservative -- speech only where both agree), --vad=fused:union (aggressive -- speech where either fires), or --vad=fused:weighted (the default). These options require the optional silero dependency, which itself requires PyTorch; install both with pip install ffsubsync[torch] (or just pip install torch). torch is not installed with ffsubsync by default.

When syncing in bulk, a bad sync can be worse than none. Passing --skip-sync-on-low-quality leaves the subtitles unmodified when the alignment looks untrustworthy—an anti-correlated score (--min-score, default 0.0) or an implausibly large offset (--quality-max-offset-seconds, default 30). There is also a --max-framerate-deviation check (default 0.1, which permits every framerate correction ffsubsync makes); tighten it only when you know the framerate should not change.

If the sync still fails, consider trying one of the following similar tools:

• sc0ty/subsync: does speech-to-text and looks for matching word morphemes
• kaegi/alass: rust-based subtitle synchronizer with a fancy dynamic programming algorithm
• tympanix/subsync: neural net based approach that optimizes directly for alignment when performing speech detection
• oseiskar/autosubsync: performs speech detection with bespoke spectrogram + logistic regression
• pums974/srtsync: similar approach to ffsubsync (WebRTC's VAD + FFT to maximize signal cross correlation)

Speed

ffsubsync usually finishes in 20 to 30 seconds, depending on the length of the video. The most expensive step is actually extraction of raw audio. If you already have a correctly synchronized "reference" srt file (in which case audio extraction can be skipped), ffsubsync typically runs in less than a second.

How It Works

The synchronization algorithm operates in 3 steps:

1. Discretize both the video file's audio stream and the subtitles into 10ms windows.
2. For each 10ms window, determine whether that window contains speech. This is trivial to do for subtitles (we just determine whether any subtitle is "on" during each time window); for the audio stream, use an off-the-shelf voice activity detector (VAD) like the one built into webrtc.
3. Now we have two binary strings: one for the subtitles, and one for the video. Try to align these strings by matching 0's with 0's and 1's with 1's. We score these alignments as (# video 1's matched w/ subtitle 1's) - (# video 1's matched with subtitle 0's).

The best-scoring alignment from step 3 determines how to offset the subtitles in time so that they are properly synced with the video. Because the binary strings are fairly long (millions of digits for video longer than an hour), the naive O(n^2) strategy for scoring all alignments is unacceptable. Instead, we use the fact that "scoring all alignments" is a convolution operation and can be implemented with the Fast Fourier Transform (FFT), bringing the complexity down to O(n log n).

Limitations

In most cases, inconsistencies between video and subtitles occur when starting or ending segments present in video are not present in subtitles, or vice versa. This can occur, for example, when a TV episode recap in the subtitles was pruned from video. FFsubsync typically works well in these cases, and in my experience this covers >95% of use cases. Handling breaks and splits outside of the beginning and ending segments is left to future work (see below).

Future Work

Besides general stability and usability improvements, one line of work aims to extend the synchronization algorithm to handle splits / breaks in the middle of video not present in subtitles (or vice versa). Developing a robust solution will take some time (assuming one is possible). See #10 for more details.

History

The implementation for this project was started during HackIllinois 2019, for which it received an Honorable Mention (ranked in the top 5 projects, excluding projects that won company-specific prizes).

Credits

This project would not be possible without the following libraries:

• ffmpeg and the ffmpeg-python wrapper, for extracting raw audio from video
• VAD from webrtc and the py-webrtcvad wrapper, for speech detection
• srt for operating on SRT files
• numpy and, indirectly, FFTPACK, which powers the FFT-based algorithm for fast scoring of alignments between subtitles (or subtitles and video)
• Other excellent Python libraries like argparse, rich, and tqdm, not related to the core functionality, but which enable much better experiences for developers and users.

License

Code in this project is MIT licensed.