cymedia

Low-latency, group-aligned media processing graphs on FFmpeg's libraries, for Python.

A graph decodes a source once and feeds any number of branches. Each branch is a libavfilter chain ending in an encoder, and its output goes to a sink. Source group numbers, object numbers and timestamps pass through unchanged, and every branch starts a keyframe at each source group start, so equally numbered groups line up across all outputs. The first graph is an ABR ladder.

Status: alpha. H.264 and AV1 ladders on the NVIDIA path (NVDEC, scale_cuda, NVENC) and in software, and H.264 ladders on Apple's VideoToolbox, on Linux and macOS with Python 3.12 and 3.13. The API can change before 1.0.

Install

cymedia builds against the system's FFmpeg, 6.1 or later. It needs FFmpeg's development files, pkg-config and a C compiler:

sudo apt-get install pkg-config libavcodec-dev libavfilter-dev libavutil-dev   # Ubuntu 24.04
brew install ffmpeg pkg-config                                                 # macOS
pip install cymedia

Prebuilt wheels with an LGPL FFmpeg are planned.

Packaged FFmpeg builds differ in what they include; Homebrew's has no libaom, for example. scripts/build-ffmpeg.sh builds one with what the profiles need, in a directory of its own:

scripts/build-ffmpeg.sh
source ~/.local/cymedia-ffmpeg/env.sh
pip install --no-cache-dir --no-binary cymedia cymedia

That is an LGPL FFmpeg with libaom, plus NVENC and scale_cuda on Linux or VideoToolbox on macOS. It has no software H.264 encoder: --gpl adds libx264 and makes the build GPL.

Example

from cymedia import Rung, ladder

rungs = [Rung(1280, 720, 3_750_000), Rung(640, 360, 900_000)]
with ladder(rungs, profile="nvenc", codec="h264", fps=(60, 1)) as g:
    g.push(frame, group, obj, pts_us, key)  # one H.264 access unit
    for out in g.read(timeout=0.1):
        ...  # out.branch, out.group, out.object, out.pts_us, out.key, out.group_start, out.data

• Input is one H.264 access unit per push(): Annex B, or 4-byte length prefixes with the avcC passed as extradata=. key marks a group's first frame. push() returns False when the frame was dropped.
• Output is one Output per encoded frame, carrying its source frame's group, object and timestamp: Annex B for H.264, a temporal unit of OBUs for AV1. g.fileno() is readable while outputs are queued, for an event loop: loop.add_reader(g.fileno(), lambda: handle(g.read(0))).
• flush(timeout) ends input and waits for every branch's last frame; stats() and delays(branch) report counters and recent push-to-output delays.

Profiles

The NVENC encoders run p1, ultra low latency, CBR with a one-frame buffer and no B-frames; H.264 filler NAL units and AV1 padding OBUs are stripped. Every profile forces a keyframe at each group start. VideoToolbox runs in its low-latency mode with no B-frames. SVT-AV1 is not used: version 1.7 ignores forced keyframes.

Any other graph is a list of branches:

from cymedia import Branch, Graph

g = Graph([Branch("scale=640:360,format=yuv420p", "libx264", {"preset": "veryfast"})],
          decoder="h264", fps=(30, 1))

How it runs

• One decode thread decodes each source frame once, then runs every branch's filter chain.
• Each branch encodes on its own thread. Python's GIL is released throughout.
• In realtime mode (the default) a full queue drops instead of blocking. Input resumes at the next group start; a branch that falls behind abandons the rest of that group, so a group is cut short and never has a hole in it. While a branch's encoder first opens, frames wait rather than drop. With realtime=False, queues block instead.
• Outputs go to a sink with a C ABI (cymedia_sink.h). The default sink queues them for Graph.read(); a native consumer passes its own sink as a cymedia.sink capsule and keeps frames out of Python entirely.

Measured delay

An RTX 4070, a 2160p60 H.264 source to six NVENC rungs (1440p to 180p), from push() to the encoded output, the 4K decode included (about 4.4 ms of it):

Under a light load a GeForce card drops to a low power state and the delay doubles; no frames are held.

An Apple M4, a 1080p60 source to four VideoToolbox rungs (720p to 180p): p50 12 ms on the first rung and 75–80 ms on the others, in realtime with no drops. It cannot keep up with a 2160p60 source to six rungs.

bench/ladder_delay.py measures it on your hardware.

Used by

aiomoqt's abr_ingest sits in front of a MoQ relay and publishes a broadcast plus a cymedia ladder under one catalog.

Develop

python -m venv .venv && .venv/bin/pip install "Cython>=3.0.10" "setuptools>=77" pytest
.venv/bin/pip install --no-build-isolation -e .
.venv/bin/python -m pytest -q

GPU tests skip unless CUDA, h264_nvenc and scale_cuda are available. The delay bench:

.venv/bin/python bench/ladder_delay.py clip.mkv --profile nvenc --seconds 90

Contributor rules are in AGENTS.md.

License

MIT. This software uses code of FFmpeg licensed under the LGPLv2.1 and its source can be downloaded here. cymedia ships no FFmpeg: it links the system's, and cymedia.ffmpeg_info() reports that build's license. Redistributing FFmpeg with cymedia is subject to FFmpeg's terms.