frameproof

Your coding agent did not watch that video. It guessed.

Ask Claude Code to "watch this tutorial" and it samples frames on a scene-change threshold. On a screencast that threshold cannot fire. A tool may warn that coverage is sparse, but it will not tell you WHERE the hole is — so the agent cannot tell "few frames" from "no frames for twenty minutes straight".

Measured on a real 38-minute tutorial, the most popular tool in this niche extracted 17 frames by default, with a 20-minute 51-second gap. frameproof extracted 220 with a 14-second maximum gap — first command, no flags.

The full table, including their best mode where they lead on coverage, is in bench/RESULTS.md. Hiding it would be less interesting.

pip install frameproof
frameproof index "https://youtube.com/watch?v=..." --ocr

Why the threshold cannot work

ffmpeg's scene filter measures the mean delta across the whole frame. Measured on ffmpeg 8.0.1 with real terminal colours (#cccccc on #1e1e1e, 640×360):

Real glyphs cover 10–15 % of a line's area, so a typed command scores around 0.006 — off by a factor of about 40. Lowering the threshold does not help: what rescues a screencast buries a fast-cut video under thousands of frames.

frameproof measures the fraction of changed pixels per grid cell, calibrated against each cell's own baseline. A cell that moves constantly — the presenter's webcam, a running timer, a cursor — is suppressed automatically. A cell that is quiet most of the time and then changes is an event.

The guarantee

No stretch of the timeline is left without a frame for longer than --max-gap seconds (15 by default). When detectors stay silent, frames are placed on a grid. Coverage is not a matter of picking a lucky threshold.

And when the guarantee cannot be met, the tool says so:

покрытие: 97 % — 2 участка без кадров (57 с). НЕ утверждай, что показано на экране в них.
    БЕЗ КАДРА  25:30 – 25:59   (29 с)

Silent blindness is worse than an honest "I did not look here".

Three commands, on purpose

Search tells you where it could not look

A hit is an answer. It is not the whole answer if part of the recording has no frames at all. search now ends with the gaps that sit near the hit:

[12:30 / seg#1] speech: цена подписки двадцать долларов

1 совпадений. Ни одной картинки не загружено.

⚠ рядом с найденным 1 участок без кадров: 15:00–18:00
  Ответ мог быть и там.

The gaps were always computed at index time; only report printed them, and nobody runs report before answering. The distance is measured to the nearest hit, so this is a caveat about YOUR answer, not general statistics: a gap forty minutes away from everything you found stays out of the way.

If search could return pictures, the savings would vanish on the first query. A frame at 1280×720 costs about 1196 visual tokens; the transcript of an hour is about 50 KB. Most questions are answered without loading a single image.

frameproof search "openrouter" --out ~/.frameproof/hermes
# [9:57 / f0050] screen: ... OpenRouter • дешевле напрямую ...

frameproof frames --at 18:38 --out ~/.frameproof/hermes
# [18:38 / f0097] .../frames/f0097.jpg  (1196 токенов)

Two speed tiers

frameproof index <url> --fast     # 1 second
frameproof index <url>            # 32 seconds, frames land better

--fast takes candidates from keyframes instead of decoding the whole video. Measured on a 38-minute tutorial:

The fast tier returns 85 % of the information for 3 % of the time. The trade is honest: frames land where the encoder put a keyframe, not where the thought on screen finished.

The frame budget scales with duration instead of being a constant: a one-minute clip gets 40, a 38-minute tutorial 231, a three-hour lecture 600.

A citation you can check

[18:38 / f0097] is not decoration. It points at a row of the index, and arithmetic checks it:

frameproof verify answer.md --out ~/.frameproof/hermes

✗ [20:00 / f9999] The memory architecture diagram is on screen.
      FAIL  FRAME_NOT_FOUND: no frame f9999 in the index — the reference is invented
✗ [5:00 / f0097] Here he opens the router settings.
      FAIL  TIME_MISMATCH: the tag says 5:00, frame f0097 was taken at 18:38
?  [29:31 / f0160] A list of ten skills is shown.
      WARN  NEVER_OPENED: the frame exists but was never requested —
            the claim was made without looking

Six checks, zero model calls: does the frame exist · does the timestamp match · does the moment fall in a coverage gap · was the frame ever served to the agent · does the quoted string appear in the frame's OCR · does it appear in nearby speech.

A blind second look

Meaning is beyond arithmetic. For that there is a separate subagent that sees only the frame and the claim — not the user's question, not the author's reasoning, not the rest of the answer. Its job is to refute.

frameproof verify answer.md --out <index> --plan   # tasks carrying no context at all

It runs only when explicitly asked. Refuted claims are flagged, not hidden: measured adversarial panels raise false alarms on up to a third of correct claims, so the call stays with the human.

Install

pip install frameproof          # core
pip install "frameproof[net]"   # + yt-dlp for links
pip install "frameproof[mlx]"   # + fast local transcription on Apple Silicon

frameproof doctor               # check what is available
frameproof install              # install the skill into Claude Code

npx skills add edvardgrishin27/frameproof -g   # Codex, Cursor, Copilot, others

We have not verified this outside Claude Code. The SKILL.md format is portable and the manifests are in place, but we will not claim support we did not test — see CLAIMS.md.

Requires ffmpeg. Everything else is optional and degrades gracefully. No API keys, ever. Subtitles come free from yt-dlp; when there are none, transcription runs locally.

Kinescope

A Russian video host carrying courses and webinars. yt-dlp cannot fetch it: the extractor request has been open since 2022 and the page returns "Unsupported URL". Handing it the manifest directly does not help either — it lists the formats but downloads the wrong thing: 1243 "segments" point at one file through byte ranges, and the downloader ignores the ranges. Measured on an 82-minute lecture: yt-dlp estimated 96 GiB for a video that weighs 121 MB (yt-dlp#12687).

So the fetch is our own, and it is simpler: the server honours any range asked of it, so the whole file is addressable directly instead of through 1243 pseudo-segments.

frameproof index "https://kinescope.io/embed/<id>" --ocr

Some videos are behind a signed link: without expires and sign the manifest returns 403, in DASH and HLS alike. Pass the URL whole, parameters included; if it has none, they are looked up on the player page. Downloads run in chunks with resume — the server drops a single large request, and a partial file survives both the drop and a restart.

Audio sits in the manifest under a different shape — BaseURL pointing straight at the file plus byte ranges — and is fetched alongside the video. DASH carries no subtitles, but for some videos a ready track exists in the HLS manifest of the same video: when one is found it is used and the audio is not downloaded at all. Auto-generated tracks are flagged as such — the host serves ASR, and ASR is wrong sometimes.

ClearKey-encrypted videos are refused out loud rather than half downloaded: decryption needs mp4decrypt from Bento4, a separate binary we do not ship.

Off the Mac

Two places grew up on a MacBook: text recognition went through Apple Vision, and local transcription through mlx-whisper on Apple Silicon. Both doors now open outward, with the core untouched.

# your own recognizer: takes image paths, prints "path<TAB>text"
frameproof index video.mp4 --ocr --ocr-command "python ocr_windows.py"

# your own subtitles instead of transcription — .vtt, .srt or .json3
frameproof index video.mp4 --subs speech.srt

--ocr-command is the same contract the internal Swift binary already speaks, simply exposed. On Windows 10 and 11 the built-in offline Windows.Media.Ocr fits it directly: no keys, no install. Reply in UTF-8; a system-ANSI reply is accepted too, but UTF-8 is the contract.

On resolution. Display frames are scaled down to --width (1280 is a token-cost decision), and small interface text does not survive that: the same frame of a GitHub page yielded one word at 1280 and full filenames and commit lines at 2560. Recognition therefore runs on a separate full-resolution copy that is deleted right after, controlled by --ocr-width. What you show stays cheap.

Use in Claude Code

After frameproof install, just ask: "watch this video and tell me which command he shows at 4:12". The skill enforces one rule the agent cannot skip:

Never claim what was on screen without having seen a frame. Every statement about the screen carries a [MM:SS / fNNNN] tag so a human can check it.

Honest limits

The full list is in CLAIMS.md. The short version: this tool guarantees coverage, not that no change was ever missed; OCR is for finding frames, not for reading code verbatim; and the benchmark is one video of the class where the gap is widest.

Russian documentation: README.ru.md

MIT