av-drift-detector (Python)

Measure audio/video sync to a fraction of a millisecond, frame by frame.

Python bindings to av-drift-detector, the Rust library behind the av-drift command. Generate a test video whose every frame, and every frame's worth of audio, carries its own frame number. Play it through whatever you want to test (an encoder, a streaming pipeline, a TV, a video call), record the result, and measure how far the sound has drifted from the picture at every frame.

import av_drift_detector as avd

avd.generate("sync.mp4", fps="29.97", duration=10, audio_offset=0.040)
# ... play sync.mp4 through the system under test and record it as captured.mp4 ...
report = avd.analyze("captured.mp4")
print(report["summary"]["mean_ms"])  # positive means the audio is late

Install

pip install av-drift-detector

Python 3.11 or later. Wheels cover macOS (Apple silicon and Intel), Linux (x64 and arm64, manylinux) and Windows (x64); elsewhere pip builds from source, which needs Rust.

Reading and writing files runs the ffmpeg and ffprobe programs, so install FFmpeg 5.1 or later (brew install ffmpeg, sudo apt install ffmpeg, winget install Gyan.FFmpeg) and keep them on your PATH, or set AV_DRIFT_FFMPEG and AV_DRIFT_FFPROBE to their paths. Writing H.264 needs an FFmpeg with libx264, as those packages have. The in-memory functions below don't need FFmpeg.

Files

• generate(output, *, fps="30", duration=10.0, width=1280, height=720, sample_rate=48000, video_codec="h264", audio_codec="aac", audio_offset=0.0, music=True, on_progress=None) writes a test video and returns a summary. audio_offset (seconds) deliberately delays the audio; negative makes it early. music plays a tune under the tones that doesn't affect analysis. on_progress(frames_done, frames_total) is called about once a percent. FFV1 video and PCM audio need .mov or .mkv.
• analyze(path, *, offset=0.0, duration=None, fps=None, use_qr=True, capture="auto") measures every frame of a capture. offset and duration choose a window in seconds, and capture is "auto", "file" (a digital copy) or "camera" (a phone filming a screen).

Reports are the dicts av-drift analyze --json writes (format), typed as DriftReport, so json.dump(report, f) saves the CLI's format:

s = report["summary"]
print(f"{s['mean_ms']:+.2f} ms ± {s['stddev_ms']:.2f}, skew {s['skew_ppm']:+.1f} ppm")
for f in report["frames"]:
    if f["status"] == "ok":
        print(f["time"], f["video_frame"], f["drift_ms"])

Frames and audio from elsewhere

If you already have decoded frames and audio (a capture card, WebRTC, a camera), measure them without files:

reader = avd.StampReader()
for time, luma in my_frames():           # 8-bit luma (Y) planes: bytes, numpy uint8, ...
    reader.read(luma, width, height, time)

audio = {"sample_rate": 48_000, "start_time": t0, "samples": mono_float32}  # array("f"), numpy float32, ...
report = avd.analyze_frames(reader.finish(), audio, width=width, height=height)

Frame times and start_time must be on the same clock, in seconds.

• StampReader(*, try_qr=True) reads the frames of one capture in order. read(luma, width, height, time, stride=None, *, exposure=None, readout=None) returns the frame ({"time", "stamp", "qr_located", "exposure", "readout"}); finish() returns every frame read, with any that could only be identified from later frames filled in. It follows the QR code from frame to frame, which is faster than reading each frame alone, and it reads pictures a camera caught while the screen switched frames, measuring how far through the switch each was (the stamp's blend). Use it for camera footage. For a camera, time is when the picture's top row began its exposure, as phones' camera APIs record it; if the camera reports each frame's exposure time and rolling-shutter readout (seconds), pass them too, and a camera locked to the pattern's rate is timed to a millisecond or two.
• read_frame(luma, width, height, stride=None, *, try_qr=True) reads one frame's stamp (barcode, then QR code), or returns None.
• PatternRenderer(*, fps="30", width=1280, height=720, sample_rate=48000, duration=None).render(frame_id) draws a test-pattern frame as luma bytes. With duration (seconds), the info line shows the file's length.
• synthesize_audio(count, *, fps="30", sample_rate=48000, delay=0.0, start=0, music=True) makes the test audio as an array("f"), with the music unless music is false.

Errors raise AvDriftError. Slow calls release the GIL, so they run in parallel from threads.

More

The repository has the guides: measuring with a camera, the test pattern specification, and how the analysis works. Licensed under the BSD-2-Clause Plus Patent License.