mosaic-media

Media probing, transcode planning, and frame reading. Distribution name mosaic-media, import name mosaic_media. Python 3.12 or newer, Apache-2.0.

Source: https://github.com/EcodylicScience/mosaic-media

The package answers two questions about a video file, then constructs and runs the ffmpeg work that follows from the answers:

1. Does it play well in a browser, and does it scrub quickly?
2. Is it usable for per-frame analyses like tracking?

A file can need a transcode for one and not the other, so the two verdicts are independent. What counts as playable is supplied by the caller, not decided here.

Installation

pip install mosaic-media          # probe, verdict, transcode, thumbnails
pip install "mosaic-media[io]"    # + in-process frame reader and writer
pip install "mosaic-media[cli]"   # + the mosaic-media command line app

ffmpeg and ffprobe must be on PATH for the probe, the transcode, and the CLI: version 5.1 or newer at runtime (-fps_mode), 6.0 or newer to run the test suite (-display_rotation). The [io] reader decodes in process and needs no ffmpeg binary.

Platforms: developed and tested on Linux. Nothing is platform-specific except hardware encoding, which requires the relevant ffmpeg encoder and a device.

Quick start

Probe once. MediaFacts is the authoritative metadata from then on: pass it onward rather than re-measuring, because a file that is already clean is never rewritten and so cannot be assumed to have canonical bytes.

from mosaic_media import CHROME_149, DEFAULT_THRESHOLDS, derive, probe_media

facts = probe_media("recording.mp4")
print(facts.frame_count, facts.fps, facts.video_uuid)

verdict = derive(facts, CHROME_149, DEFAULT_THRESHOLDS)
print(verdict.analysis_transcode, verdict.stream_transcode)
print(sorted(verdict.analysis_reasons), sorted(verdict.stream_reasons))

Both policy objects are yours to supply. CHROME_149 is the one profile shipped, measured against Chrome 149 on Linux. Describe a different target by constructing your own:

from mosaic_media import PlaybackProfile, Thresholds

profile = PlaybackProfile(
    # ffprobe format_name strings, never file extensions
    containers=frozenset({"mov,mp4,m4a,3gp,3g2,mj2"}),
    codecs=frozenset({"h264", "av1"}),
    client_dependent_codecs=frozenset({"hevc"}),  # plays only on some clients
    baseline_pixel_formats=frozenset({"yuv420p"}),
)
thresholds = Thresholds(drift_frame_periods=0.5)  # the rest keep defaults

Thresholds carries every limit the verdict applies, and DEFAULT_THRESHOLDS is Thresholds(): drift_frame_periods=0.5 (how far a frame may sit from its uniform-grid position before the file counts as variable rate), max_gop_bytes=524288, max_keyframe_interval_frames=200, truncation_duration_ratio=0.95, start_time_frame_periods=0.5, and frame_exact_codecs=FRAME_EXACT_CODECS -- the codecs whose decoders are measured to emit one frame per packet, the one limit that is a set rather than a number. A codec outside it gets an analysis re-encode, so a caller that has measured one this package omits injects a wider set rather than patching it.

Run the work the verdict calls for:

from mosaic_media.transcode import ANALYSIS_ENCODING, run_transcode

result = run_transcode(
    "recording.mp4", "derived.mp4", "analysis", facts, verdict,
    profile=CHROME_149, thresholds=DEFAULT_THRESHOLDS,
    encoding=ANALYSIS_ENCODING,
)
if result.performed:
    print(result.operation, result.output_path)
    print(result.output_facts.video_uuid, result.source_video_uuid)

run_transcode accepts on_progress and cancel_check callbacks and a timeout. It returns without writing when the verdict asks for nothing.

Read frames, with the [io] extra:

from mosaic_media.io import VideoReader

# A no-op transcode writes nothing, so read the source in that case.
path = result.output_path if result.performed else "recording.mp4"
known = result.output_facts if result.performed else facts

with VideoReader(path, facts=known) as reader:
    for index, frame in reader:   # frame index, then a BGR numpy array
        ...

output_path, output_facts, and output_verdict are None on a no-op; performed is what distinguishes the two branches, and source_video_uuid is populated either way because it describes the input. Passing facts skips a probe the caller has already paid for. MultiVideoReader does the same across a list of files, presenting them as one sequence.

Everything in the core is importable from mosaic_media itself; the two extras keep their own namespaces, mosaic_media.transcode and mosaic_media.io.

Compare two files for duplication:

from mosaic_media import compare_for_duplicate

print(compare_for_duplicate(facts, probe_media("other.mp4")).verdict)

Thumbnails need no extra:

from mosaic_media import (
    downscale_to_jpeg, extract_first_frame, thumbnail_dimensions,
)

width, height = thumbnail_dimensions(facts.width, facts.height, cap=320)
extract_first_frame("recording.mp4", "frame.png")
downscale_to_jpeg("frame.png", "thumb.jpg", width=width, height=height)

thumbnail_dimensions returns the (width, height) that fits the long edge to cap while preserving aspect.

Layering and optional dependencies

The core is standard library only so the transcode runner can start on a machine that has ffmpeg and nothing else: a minimal container, or a recording box without an analysis stack. A test enforces it, so installing the core pulls neither numpy nor typer.

Decoding

The probe and the transcode run through the system ffmpeg binaries; the CLI is the terminal interface to both. The probe decodes no frame at all -- it reads demultiplexed packets, and returns each one's time, size, and keyframe flag in decode order. Only the [io] extra decodes and encodes in process, through av.

That packet index is what makes seeking exact. VideoReader resolves the target's preceding keyframe from the index and seeks the container to that keyframe's timestamp. A landing at or before that keyframe is expected -- container seek granularity is coarser than the keyframe list on some formats -- so the reader resolves where the decoder actually landed against the index and counts frames forward from there. A landing after the requested keyframe means the target's own references were skipped, and raises. Every step uses the frames' own recorded timestamps; nothing converts an index to a timestamp through an average rate.

VideoReader holds to one contract:

• A video the probe found analysis-ready reads cleanly: the reader does not fail on it, and every index within frame_count returns the frame that belongs to it.
• A video it cannot faithfully deliver raises MediaProbeError naming the frame, rather than returning a neighboring one.

The verdict does not reach the reader, which carries no policy of its own, so choosing whether to open one is the caller's step: probe, read the analysis verdict, and open the derivative rather than the original where the verdict asks for one.

Neither path uses OpenCV, for two reasons that matter if you are replacing a cv2.VideoCapture decode path:

• CAP_PROP_POS_FRAMES converts a frame index to a timestamp through one average rate, which is exact at constant rate and inexact otherwise.
• Current opencv-python wheels decode zero frames from an AV1 file while reporting a plausible frame count and raising nothing, and the derivatives this package produces are AV1 (tests/io/test_reader_cv2_equality.py).

By default av installs as a wheel carrying its own FFmpeg build, so the reader's codec table is that build's rather than the system one. Where a single decode stack matters, pip install av --no-binary av builds av against the system libav, after which the reader and the subprocess paths share one FFmpeg.

A performance gate measures the reader against cv2.VideoCapture on metadata open, sequential decode, strided decode, and cold random seek, among others. It is excluded from the default suite and run explicitly (pytest -m bench), on an idle machine, because contended measurements swing enough to make the comparison meaningless.

A VideoReader is not thread-safe: it holds an open decoder with position state. Use one per thread.

Transcode semantics

Two targets, one codec, differing in rate control, in whether audio is kept, and in whether they run at all. The analysis derivative carries no audio track and no keyframe-interval cap; the playback derivative keeps audio and caps the interval so scrubbing does not decode long runs. The original is preserved in every case; derivatives are separate artifacts.

Which containers and codecs count as supported is the caller's PlaybackProfile, not a constant here. Verdict.stream_transcode is Literal["required", "recommended"] | None and HARD_STREAM_REASONS is the set that makes it required; Verdict.analysis_transcode is Literal["required"] | None, because a defect that invalidates per-frame results has no advisory tier.

The verdict selects the minimum operation rather than a blanket re-encode. A header that lies about timing gets a -c copy remux with corrected timestamps, a tail moov gets -movflags +faststart, and a supported stream in an unopenable container gets a rewrap. A re-encode is what remains when no copy can clear the reasons: a defect in the pixel grid or the frame clock, a stream the browser cannot decode or cannot stream economically, or a codec whose frame correspondence is unverified. A copy that would clear its reasons is escalated to a re-encode anyway when it would not survive as a copy -- because the output container cannot carry the source codec, or because it would carry forward packets the source itself does not decode, leaving the derivative missing the same frames. Encoding runs on SVT-AV1, or on NVENC when the caller permits hardware and this machine can open av1_nvenc. A listing proves only that the encoder was compiled in, so permission is checked against a probe that encodes a frame; a build carrying the encoder on a device that cannot run it encodes on the CPU instead of failing at encoder startup.

Variable frame rate is the hard measurement. ffprobe and OpenCV both report it for containers that merely quantize timestamps to milliseconds, so the probe instead fits a uniform grid across all frame timestamps and measures the worst deviation in frame periods, against the drift limit in Thresholds. The fit covers the whole file: a bounded window misclassifies recordings that drop frames partway through rather than at the start.

The transcoded output is re-probed as its acceptance test, since a variable-rate source resampled to a constant rate can still carry residual drift, and that probe mints the derivative's authoritative MediaFacts. If the output's own verdict is not clean the converter raises rather than retrying: the input is unchanged and the command is unchanged, so a second attempt has nothing new to work with.

A transcode changes the pixels and therefore every measured fact, so the derivative's identity shares nothing with its source's and no hash recovers the link. TranscodeResult carries source_video_uuid for that reason -- the source's video_uuid, recorded on the result, including on the no-op branch where it describes the input either way.

Video identity

The probe derives two values per file, both MediaFacts fields, so they travel with the rest of the metadata and are minted once. They answer different questions and are not interchangeable.

Both are built from a per-packet payload hash the packet scan reads through ffprobe's -show_data_hash, not from a decoded frame and not from a file checksum. content_digest hashes the codec-level facts and every packet's size, keyframe flag, and payload hash; video_uuid folds the packet timestamps on top and is emitted as an RFC 9562 UUIDv8. The payload hash is CRC32, which guards against accidental collision, not against a crafted one.

video_uuid is the only value safe to compare for identity or to name anything. Because it hashes the timestamps, a container change moves it, and a directory named from it is not recoverable across one -- not even by repacking back: Matroska quantizes timestamps to milliseconds and MPEG-TS rebases onto the first program clock reference, and returning to the original container inherits that quantization rather than undoing it.

content_digest survives those container rewrites, but only where they preserve the elementary stream. It is not invariant across a bitstream reframing: an mp4 to MPEG-TS repack applies Annex B conversion, which changes the coded bytes themselves, so the digest changes with them.

To find duplicates, group by content_digest and call compare_for_duplicate on the members of a group. It compares the timing with a duration-scaled tolerance -- a longer file is allowed proportionally more absolute drift -- and returns one of five verdicts: duplicate, different timing, timing unknown, unminted, or distinct. It is exported so the tolerance rule has one implementation rather than one per caller.

Reading a packet's payload is what this costs: the scan reads every packet's bytes rather than only its header, once, at the probe.

identity_scheme and prober_version are recorded on every probe -- the scheme version that minted the two values, and the ffprobe build whose demuxer output the digest is defined against. Neither is hashed.

The digest folds in each packet's payload hash as libavformat hands it over, so an ffmpeg upgrade is the one event outside your control that could re-mint stored values. Measured, it does not: both values are byte-identical across FFmpeg 6.1, 7.1, and 8.1 for every committed fixture, including raw elementary streams. tests/probe/test_identity_across_ffmpeg_builds.py re-checks it whenever alternate builds are configured, and prober_version is what makes a future re-mint detectable rather than silent.

Versioning

Semantic versioning, with the pre-1.0 convention that a minor bump is a breaking change and a patch bump is compatible. Pin a range (mosaic-media>=0.3.0,<0.4.0); a bare floor would not exclude the next breaking release. Release notes are on the repository's releases page.

IDENTITY_SCHEME is a second, independent number. It moves only when the bytes hashed into video_uuid or content_digest change, and a move re-mints every stored value. The two are coupled in one direction only: a scheme bump always forces a breaking version bump, and a version bump never implies a scheme bump. A pinned range therefore protects stored identity values.

Command line

The [cli] extra installs a typer application, usable standalone or mounted as a sub-application:

import typer
from mosaic_media.cli import app as media_app

app = typer.Typer()
app.add_typer(media_app, name="media")

mosaic-media probe video.mp4
mosaic-media transcode video.mp4 --target playback --output media/
mosaic-media compare left.mp4 right.mp4

--output takes a file path or an existing directory, in which case the filename derives from the source stem and is always .mp4. The converter refuses to overwrite the source and refuses a file destination not ending in .mp4; re-running the same transcode replaces its output atomically. The package knows no dataset layout; where originals and derivatives live is the caller's decision.

compare prints the DuplicateComparison as JSON, and its verdict is also the exit code, so it works as a shell test without parsing stdout:

Exit 2 is left to the CLI framework's usage error, so a mistyped option is never mistaken for a verdict. --fps-tolerance and --duration-tolerance override the derived defaults; the left file is the reference the tolerances are computed from.

For programmatic use prefer the Python API: structured exceptions, no argv escaping, no output parsing.

Errors

Two exception types, both RuntimeError subclasses.

MediaProbeError comes from the probe: no video stream, a missing file, or ffprobe failing or returning output that cannot be parsed.

It also comes from VideoReader, which raises it rather than hand back a frame that is not the one asked for. A read loop can raise, not only the probe call before it. The reader's cases are a seek landing that cannot be resolved against the index, a delivered frame the index does not place at the index it is being returned under, a gap between consecutive decoded frames of a constant-rate source, two consecutive decoded frames at one timestamp, and a read that ends before delivering every frame its window declares.

The last four report the frame model and the decode disagreeing about how many pictures the file has, and each names the frame: three reach a shortfall from different directions -- packets that do not all decode -- while two frames at one timestamp is the excess, a picture the model does not count. A seek landing is different, and covers two raises: the container landed past the keyframe the index named, so the target's own references were skipped and counting forward would decode from the wrong prefix; or it landed outside the index's span altogether, and the decoded time matches no entry. Neither is a disagreement about the picture count, and the second names a time rather than a frame. A source the analysis verdict accepted raises none of them.

TranscodeError comes from the converter: ffmpeg failing, the run exceeding its timeout, a cancel callback asking it to stop, a destination that is refused, or the output failing its acceptance probe.

It also comes from build_command, before any command is built, for a source that states no frame rate in either its container or its bitstream. Such a file carries no timing at all, and neither a remux nor a re-encode can supply one without inventing it.

Container image

Dockerfile assembles a prebuilt LGPL FFmpeg -- a third-party release pinned by release tag and asset digest -- and compiles PyAV against it rather than against the FFmpeg its own wheel bundles. The FFmpeg stage rejects a download whose configuration carries --enable-gpl, --enable-libx264, or --enable-libx265; the test and runtime stages then run scripts/verify-ffmpeg-lgpl.py, which inspects the library the process actually links and fails if a GPL encoder is reachable or if the CLI on PATH is a different build from that library. Each is a build step, so a stage that fails its gate produces no image. The test stage also runs the suite and pins video identity against that exact FFmpeg. --target ffmpeg yields the gated prefix with the verifier installed but not yet run -- that stage has no PyAV to run it against -- and --target runtime adds PyAV on top.

License

Apache License 2.0 -- see LICENSE.

The package shells out to the system ffmpeg and ffprobe binaries and, for the [io] extra, uses av. Those components are not distributed with this package and carry their own licenses: FFmpeg is LGPL-2.1-or-later, or GPL if built with GPL-only components, and av is BSD-3-Clause.

Nothing in the distributed package names a GPL-only encoder. av links FFmpeg into the calling process, so an encoder named here would become a dependency of this package; the default av wheel also carries its own FFmpeg build, whose codec set is chosen by whoever built the wheel rather than by this package.

Redistributors bundling any of these are combining separately licensed works and should establish their own obligations rather than relying on this summary.