sqlmpeg 0.24.0

SQL in; ffmpeg out

You write a SELECT statement, sqlmpeg compiles it into a -filter_complex invocation, and ffmpeg does the actual pixel-pushing. This tool never decodes a single frame: it's a compiler, and ffmpeg is the executor.

Why does this exist?

Look, ffmpeg is a marvel - one that I've used for over a decade - but I still need to lookup the syntax every single time I want to do something non-trivial. And AI barely helps - it often just gives you the same wrong answer from Reddit that you could have Googled yourself.

SQL, meanwhile, has been describing dataflow DAGs for fifty years, and it's the language every developer (and every LLM) already speaks.

This project connects the two so you can bring your preexisting knowledge and create declarative, composable, and generalizable ffmpeg formulae.

Install

Python 3.10+.

pip install sqlmpeg

Or run it without installing anything:

uvx sqlmpeg

ffmpeg and ffprobe are required and, optionally, handled for you. A preexisting ffmpeg install on PATH always wins. On a machine without one, the bundled provisioner (static-ffmpeg) fetches both binaries on first use.

Ask before you act

run is the default subcommand, so a query is the whole invocation - and a query with no COPY ... TO is a metadata query: the answer is probed metadata, fully known the moment compilation ends, so sqlmpeg prints it as a table and never runs ffmpeg at all. This works on anything ffprobe can read, remote manifests included:

$ sqlmpeg "SELECT t.index, t.language, t.codec \
FROM input(:'src') f, unnest(f.audio) t \
WHERE t.codec = 'aac'" \
-v src=https://storage.googleapis.com/shaka-demo-assets/angel-one/dash.mpd
 index | language | codec
-------+----------+-------
 1     | es       | aac
 2     | de       | aac
 3     | en       | aac
 7     | fr       | aac
 10    | it       | aac
(5 rows)

COPY ... TO STDOUT WITH (format 'csv') is the scriptable spelling of the same thing - stock Postgres COPY, header true optional, or TO 'tracks.csv' to write a file:

$ sqlmpeg "COPY ( \
  SELECT t.language, t.codec \
  FROM input(:'src') f, unnest(f.audio) t \
  WHERE t.codec = 'aac' \
)TO STDOUT WITH (format 'csv', header true)" \
-v src=https://storage.googleapis.com/shaka-demo-assets/angel-one/dash.mpd
language,codec
es,aac
de,aac
en,aac
fr,aac
it,aac

Media only moves when you ask for a file: using the same COPY ... TO 'out.mkv' inside the query.

PiP demo

Imagine you wanted to shrink commentary.mkv into the corner of film.mkv, and duck the commentary under the main mix. And both files carry two audio tracks - an English and a French language.

COPY(
WITH pip AS (
  SELECT scale(c.frame, 'iw/4', -2) AS frame, c.audio AS sound
  FROM input('commentary.mkv') c
)
SELECT overlay(f.frame, pip.frame, 20, 20),
       amix(volume(f.audio, 0.65), volume(pip.sound, 0.35))
FROM input('film.mkv') f, pip
) TO ('pip.mkv')

$ sqlmpeg compile -f query.sql
ffmpeg -i commentary.mkv -i film.mkv -filter_complex '
  [0:v:0]scale=width=iw/4:height=-2[n1];
  [1:v:0][n1]overlay=x=20:y=20[out0];
  [1:a:0]volume=volume=0.65[n3];
  [1:a:1]volume=volume=0.65[n4];
  [0:a:0]volume=volume=0.35[n5];
  [0:a:1]volume=volume=0.35[n6];
  [n3][n5]amix=inputs=2[out1];
  [n4][n6]amix=inputs=2[out2]' \
  -map '[out0]' -map '[out1]' -metadata:s:1 language=eng -map '[out2]' -metadata:s:2 \
  language=fra pip.mkv

Check out all that work you didn't need to do! No pad labels or bookkeeping. You never even said how many audio tracks there were: c.audio is the whole array, volume broadcasts over it (one node per language), and amix zips the two arrays elementwise, English with English, French with French. Each mixed track keeps its language tag, because both parents agreed. (compile shows the command; drop it - sqlmpeg -f query.sql - and the default run executes it instead.)

Encoding

The query above describes the edit and says nothing about codecs, so ffmpeg picks its defaults. When you care about the encode, wrap the query in COPY ... TO ... WITH (...) - stock Postgres syntax - and the destination and codec settings ride along inside the query:

COPY (
  WITH pip AS (
    SELECT scale(c.frame, 'iw/4', -2) AS frame, c.audio AS sound
    FROM input('commentary.mkv') c
  )
  SELECT overlay(f.frame, pip.frame, 20, 20),
         amix(volume(f.audio, 0.65), volume(pip.sound, 0.35))
  FROM input('film.mkv') f, pip
) TO 'pip.mkv' WITH (
  video_codec 'libx264', crf 20, audio_codec 'aac', audio_bitrate '192k'
)

$ sqlmpeg compile -f query.sql
ffmpeg -i commentary.mkv -i film.mkv -filter_complex '
  [0:v:0]scale=width=iw/4:height=-2[n1];
  [1:v:0][n1]overlay=x=20:y=20[out0];
  [1:a:0]volume=volume=0.65[n3];
  [1:a:1]volume=volume=0.65[n4];
  [0:a:0]volume=volume=0.35[n5];
  [0:a:1]volume=volume=0.35[n6];
  [n3][n5]amix=inputs=2[out1];
  [n4][n6]amix=inputs=2[out2]' \
  -map '[out0]' -map '[out1]' -metadata:s:1 language=eng -map '[out2]' -metadata:s:2 \
  language=fra -c:0 libx264 -crf:0 20 -c:1 aac -c:2 aac -b:1 192k -b:2 192k pip.mkv

Views and multiple outputs

A CREATE VIEW name AS <query>; followed by one or more COPY (...) TO '<path>' WITH (...); is a script - the ABR-ladder shape, one decode feeding several encodes. It still compiles to ONE ffmpeg invocation, one output group per COPY:

CREATE VIEW main AS
  SELECT scale(f.video[1], 1920, -2) AS v, volume(f.audio[1], 0.9) AS a
  FROM input('film.mkv') f;

COPY (SELECT scale(m.v, 1280, -2) AS v, m.a FROM main m) TO '720.mp4'
WITH (video_codec 'libx264', crf 21, audio_codec 'aac');

COPY (SELECT scale(m.v, 640, -2) AS v, m.a FROM main m) TO '360.mp4'
WITH (video_codec 'libx264', crf 26, audio_codec 'aac');

COPY (SELECT m.a FROM main m) TO 'audio.m4a'
WITH (audio_codec 'aac', audio_bitrate '128k')

$ sqlmpeg compile -f query.sql
ffmpeg -i film.mkv -filter_complex '
  [0:v:0]scale=width=1920:height=-2[n1];
  [0:a:0]volume=volume=0.9[n2];
  [n1]split=2[n1_split0][n1_split1];
  [n1_split0]scale=width=1280:height=-2[out0];
  [n1_split1]scale=width=640:height=-2[out2];
  [n2]asplit=3[out1][out3][out4]' \
  -map '[out0]' -map '[out1]' -c:0 libx264 -crf:0 21 -c:1 aac 720.mp4 \
  -map '[out2]' -map '[out3]' -c:0 libx264 -crf:0 26 -c:1 aac 360.mp4 \
  -map '[out4]' -c:0 aac -b:0 128k audio.m4a

A view is to statements what a CTE is to branches: main decodes and filters film.mkv exactly once - scale and volume each appear a single time in the graph above - and the split pass hands out however many pads its readers need (split=2 for the two video consumers, asplit=3 for the three audio ones).

There's much more - watermarks, GIFs, subtitle muxing, multiband compression, generated test media - and it all lives in the cookbook: nearly sixty real tasks, simple to complex, every shown output rerun and byte-checked by the test suite - and most of them parameterized with -v variables, so they run against your files as-is.

CLI reference

sqlmpeg <command> [-h] [-f FILE] [-v NAME=VALUE] [--timeout TIMEOUT] [-y] [query]

run is the default subcommand: any invocation that doesn't start with a subcommand name is run's, so sqlmpeg "SELECT ..." and sqlmpeg -f query.sql just work. All four query commands take the SQL as text right on the command line, or from a file with -f query.sql (-f - reads stdin). Exactly one of the two. All four also take -v name=value (repeatable - psql's flag, psql's syntax): :'name' in the query becomes the value as an escaped string literal, bare :name becomes it raw, and an undefined variable is a compile-time error. A query file plus -v is a reusable program; queries/ collects ready-made ones.

The ideas, briefly

• Streams are columns. Every input exposes <alias>.video, <alias>.audio, <alias>.subtitle, <alias>.data (1-based subscripts; <alias>.frame is sugar for video[1]), and the SELECT list is the result set - in a media query (one with a COPY destination), one column is one -map, in order, nothing implicit. A bare subscript no function touches stays a stream copy. input() takes per-input options (loop => true keeps a still image alive). SELECT * keeps everything.
• Bare arrays broadcast. atempo(v.audio, 1.25) fans out one node per track, each output keeping its language tag. Two arrays in one call zip elementwise.
• Tracks are rows when you need them. unnest(f.audio) turns a track array into a compile-time table whose columns are the probed metadata, so picking a track is WHERE t.language = 'eng' and aligning two files' tracks is a real SQL JOIN - inner, left, or full outer, with generated silence (or an empty caption track) standing in for what a file lacks. Selecting a computed column next to a track edits its tags (CASE ... END AS language retags a whole library in one expression), and chapters(f) is a table too - readable, and writable from a VALUES list. Rows combine only when written: array_agg gathers them into one file, a TO (expression) writes one file per row, and a multi-row query into a single path is a compile error - what the table preview shows is exactly what a COPY serializes. Every join is decided at compile time; ffmpeg only sees the wiring. docs/rows.md has the whole story.
• A SELECT with no COPY prints a table. The result set was fully known at compile time, so sqlmpeg "SELECT * FROM input('film.mkv') f, unnest(f.audio) t" prints the tracks as rows - ffprobe you can read, joins included - and COPY (...) TO STDOUT WITH (FORMAT csv) makes it scriptable. ffmpeg only runs when a COPY names a media destination.
• Trims are seeks. WHERE a.t BETWEEN 5 AND 60 (or either bound alone, open-ended) becomes -ss/-to on that alias's -i: fast, all stream types at once, stream-copy still possible. Decoded streams cut frame-accurate; copied ones snap to a keyframe. The measurements, and the caption caveat, are in docs/trimming.md.
• Every filter, one convention. All ~450 filters in your ffmpeg build are callable: streams first, then options - positionally in the exact order ffmpeg -help filter=<name> prints them, by name (unsharp(a.frame, luma_amount => 1.5)), or both. Every option is type-checked against what the binary reports. ffmpeg.<name>(...) always means the raw filter, including the eleven names Postgres grammar would otherwise eat; sqlmpeg.<name>(...) holds exactly four macros for jobs no single filter does (delay, speed, blur_regions, and loudnorm2, which measures a stream's loudness and corrects it in a second pass). A few multi-output filters (channelsplit, acrossover, extractplanes) return arrays. docs/filters.md has the whole story.
• Generated sources live in FROM. ffmpeg.sine(frequency => 440, duration => 1) s is a table function, not a file - the compiled command has no -i at all.
• enable and expressions. gblur(a.frame, 12, enable => 'between(t,10,20)') windows an effect in time; expression strings like '(W-w)/2' do per-frame geometry in any string-typed option.
• Captions ride along, untouched. Subtitle and data streams select, extract and mux like anything else, but they're passthrough-only - a filtergraph has no subtitle pads.
• Errors are a feature. Every rejection is a typed, line-anchored JSON object with a hint, documented with captured examples in docs/errors.md.

Use with an AI

sqlmpeg ships the system prompt. Bring whatever model you like.

$ sqlmpeg prompt > system.txt      # the dialect, the calling convention, your filters

Pipe that in as the system prompt, ask for the edit in English, and put the reply through the validator:

$ sqlmpeg validate --json -f query.sql
{"line": 1, "col": 8, "code": "UDF_ARG_TYPE", "message": "...", "hint": "..."}

The prompt's filter reference is rendered from the same registry the compiler resolves against - your installed ffmpeg - so it cannot drift, and the model works with your actual machine rather than a platonic ideal of one.

More docs: cookbook · filters · row shapes · trimming · error contract · known gaps