shortsmaker 0.1.0

Turn a long interview into publish-ready vertical clips: cut on sentence boundaries, reframed to 9:16, karaoke subtitles burned in.

🎬 shortsmaker

A long interview goes in. Publish-ready vertical clips come out.

Cut on sentence boundaries · reframed to 9:16 by body tracking · karaoke subtitles burned in

shortsmaker run interview.mp4

A 15-minute extract of a podcast becomes four or five clips you could post, in about five minutes, on a laptop with no GPU. French by default (-l fr); also handles English, Spanish, German, Italian, Portuguese and Dutch.

What a finished clip looks like

▶ Watch an example clip — 21 seconds, straight out of shortsmaker run, not retouched.

It shows the three things the pipeline exists to get right. The clip opens and closes on a whole sentence, because its bounds were looked up in a word-level transcript rather than guessed. The crop holds on the speaker and jumps only where the show's own editor cut — it never drifts, which would invent a camera move that isn't in the source. And the caption highlights each word as it is spoken, timed from the same transcript, with the title written by the model at the top.

---

Install

Needs Python 3.12+, uv, and FFmpeg built with libass (the distro package is fine — apt install ffmpeg).

Quick start (from PyPI)

# Run directly without installation
uvx shortsmaker run interview.mp4

# Or install globally
uv tool install shortsmaker
shortsmaker run interview.mp4

One install, both uses: the shortsmaker command, and an importable pipeline. Importing it does not turn it into a CLI — nothing writes to stdout, and the stages log through the standard logging module, silent until your application configures a handler:

from shortsmaker import Paths, Settings
from shortsmaker.stages import analyze, transcribe

transcript = transcribe.transcribe(paths, Settings())
segments = analyze.analyze(paths, transcript, Settings(), clips=3)

See the package docstring for the full loop — and note the if __name__ == "__main__": guard it insists on, which the reframe's worker processes require.

The font and the pose model are vendored inside the package, so the install runs offline with nothing else to fetch.

Development install (from source)

git clone … && cd shortsmaker
uv sync
cp .env.example .env      # then fill in the two keys

API Keys

Key	For
OPENAI_API_KEY	Whisper transcription
GEMINI_API_KEY	Choosing the segments

Both are metered, paid APIs — transcribe and analyze are the only two stages that call out to them, everything else runs locally and free.

Testing without API keys? Use --mock to simulate LLM responses:

shortsmaker run interview.mp4 --mock  # No API calls, no cost

---

Use

shortsmaker run interview.mp4                    # everything → clips/
shortsmaker run interview.mp4 -n 3               # keep only the 3 best passages
shortsmaker run interview.mp4 -f 1:1 -l en       # square, English
shortsmaker run interview.mp4 --mock             # test without API calls

-f --format	9:16	9:16 TikTok/Reels/Shorts · 4:5 Instagram feed · 1:1 LinkedIn/X · 16:9 keep the original shape (nothing is cropped, and the tracking is skipped entirely)
-l --language	fr	The language spoken — and the language the titles and summaries are written in.
-n --clips	all	Keep only the N best-scoring passages.
--min-score	none	Keep only the passages scoring at least this. Combines with -n.
-j --jobs	0 (auto)	Clips processed at once. 0 picks a sensible number from the machine's core count.
-w --work-dir	.shortsmaker/ beside the video	Where intermediates (transcript, segments, crop plans) are kept.
-o --out	clips/ beside the video	Where the finished clips land.
-v --verbose	off	Show the underlying ffmpeg commands and the raw API traffic.
--mock	off	Simulate LLM responses for testing without API keys or costs.

--force has no short flag on purpose: it re-runs the paid stages, so it should cost a moment's thought.

Or one stage at a time. Only the first two cost money, and they cache — so you can re-run the last two as often as you like while tuning the look:

shortsmaker transcribe interview.mp4        # Whisper   → .shortsmaker/transcript.json   (the clock)
shortsmaker analyze    interview.mp4        # Gemini    → .shortsmaker/analysis.json     (paid, kept whole)
shortsmaker show       interview.mp4        # just print what was found, with scores
shortsmaker reframe    interview.mp4        # MediaPipe → .shortsmaker/reframe/
shortsmaker render     interview.mp4        # ffmpeg    → clips/

shortsmaker render interview.mp4 --size 96 --accent "#00E5FF"   # restyle, free
shortsmaker render interview.mp4 --only 3                       # re-render just clip 3, leave the rest
shortsmaker analyze interview.mp4 --prompt-file mine.txt        # your own prompt instead of the built-in one

There is no cut command. Cutting the clips out of the source is free, fast and required before anything else can run, so it is not a decision to make: reframe and render do it themselves, and skip it when the clips are already there.

Choosing which passages to keep costs nothing. analyze writes everything it found to analysis.json and never narrows it; -n and --min-score select from that into segments.json, which is what the rest of the pipeline reads. So changing your mind is free — no API call is made:

shortsmaker analyze interview.mp4 -n 3            # keep the 3 best…
shortsmaker analyze interview.mp4 --min-score 0.8 # …no, the ones that scored 0.8+ — still free

render does not ask for the format: the crop plan records what it was built for, so it reads it back. Pass -f only to re-target.

shortsmaker --help on anything. The defaults carry their reasoning: --pop explains why 100 means no zoom, --shadow warns that ASS's shadow is a hard offset copy rather than a soft one.

shortsmaker run and shortsmaker analyze print a token table after the Gemini call — prompt, output and total tokens for the run.

---

Offline testing with --mock

You can test the pipeline offline without making any API calls by using the --mock flag. This replaces LLM responses with realistic mock data:

shortsmaker run interview.mp4 --mock              # full pipeline with mock responses
shortsmaker transcribe interview.mp4 --mock       # mock Whisper transcription
shortsmaker analyze interview.mp4 --mock          # mock Gemini analysis

When to use it:

• Testing the pipeline structure without spending credits
• Developing locally without API access
• Verifying the downstream stages (cut, reframe, render) work correctly
• CI/CD pipelines and automated testing

What gets mocked:

• Whisper transcription: generates realistic word-level timestamps and punctuated sentences
• Gemini analysis: produces segment proposals with scores, labels, and quotes from the mock transcript

The mock data is deterministic but randomized enough to test edge cases. Once transcription and analysis are cached (or mocked), the remaining stages run exactly as they would with real API data.

---

How it works, and why it looks like this

→ SPEC.md — the flowchart, and the reasoning, stage by stage. Four spikes were built and measured before any of this was written, and nearly every rule in the code replaced one that made the clips worse.

The three decisions worth knowing:

Gemini decides what to cut. Whisper decides when. Gemini reliably picks the right moments and cannot place them in time — its timestamps drift by ~2 s, and one run answered in MM.SS while the prompt demanded seconds, producing five 1-second clips that passed every check in silence. So it is never asked for timestamps. It quotes the words, and the quote is looked up in Whisper's word-level transcript. Bounds land on real word boundaries by construction. Which is why transcription runs first: it is the clock, not a subtitle afterthought.

Gemini gets the audio, never the video. The source is a talking head. The frames carry nothing the audio doesn't, but at ~1 fps they dominate the token bill and are the only reason the context window would ever overflow. The signal that decides whether a Short lands — a laugh, the pause before a punchline, a voice dropping — is in the audio, and Gemini hears it natively. 56 MB of video becomes 4.5 MB of audio.

The reframe tracks bodies, and never guesses who is speaking. Face detection was tried and failed exactly where it mattered: on a wide shot it missed the guest (in profile, head down) and hallucinated a box on the background. Bodies don't vanish in profile. And when two people are in frame, we don't pick a speaker — we show both, on a blurred fill. A question you decline to ask has no wrong answer.

---

Performance

Measured on a 16-core CPU, no GPU:

Reframe (MediaPipe)	0.71× realtime
Render (crop + blur + libass + H.264)	0.45× realtime
Local processing	~1.2× realtime

Transcription and analysis are network-bound: about 90 seconds for a 15-minute video.

---

Development

uv sync                                            # runtime deps + pytest
uv run pytest                                      # unit tests

uvx ruff@0.15.21 check .                           # lint
uvx ty@0.0.58 check                                # type check
uvx bandit@1.9.4 -c pyproject.toml -r src          # security

These are, verbatim, the four commands CI runs — see ci.yml. Green here means green there.

The three tools are not in pyproject.toml's dev group, and that is deliberate: they read the source, they are never imported by it, and locking them would pin their versions in a second place that nothing keeps in step with CI. The pins live in ci.yml, once. pytest is a dev dependency, because the test code imports it.

Rules deliberately off: E501 (the formatter is not run — the ffmpeg argument blocks are aligned by hand and ruff format would explode them one argument per line) and RUF001-003 (the prose uses ×, not x, and that is not a typo). Bandit skips four checks, each justified in pyproject.toml — chiefly the subprocess calls that are the product. B602 (shell=True) stays on.

To publish a release, see PUBLISHING.md. The short version: push a tag.

---

Licence and credits

The code is MIT — see LICENSE.

The example clip in docs/ is not. It is a short excerpt of an interview with Bernard Arnault from the show Legend (Guillaume Pley), reproduced here for the sole purpose of illustrating what the tool outputs. All rights to the footage remain with its owners; it is not covered by this project's licence, and it is not redistributable as part of it.

The vendored assets carry their own licences: Lato (Łukasz Dziedzic, SIL Open Font License 1.1) and MediaPipe's pose_landmarker (Google, Apache 2.0).