vidqa

Local, deterministic video QA — for humans, CI gates, and AI agents.

Every command prints one compact JSON object (sorted keys, ≤4KB) and exits 0 (pass) / 1 (gate failed) / 2 (error — a crash never masquerades as a gate failure). Deterministic checks are byte-identical run-to-run, so you can diff them and gate on them. Nothing is uploaded anywhere: analysis is ffmpeg + OpenCV, and the optional semantic layer talks to a local Ollama model.

Built for game/build QA — "is this capture smooth?", "did rendering break vs the golden frame?", "what does this clip actually show?" — at zero API cost, but nothing about it is game-specific.

Demo

A checkout flow recorded by a test runner, analyzed by vidqa:

$ vidqa strip run.mp4 --out sheet.png --every 1.5   # the image above

$ vidqa when run.mp4 "Payment failed"               # when did the error show?
{"first_s":8.0,"found":true,"intervals":[{"end_s":10.5,"start_s":8.0}],"mode":"text","query":"Payment failed","sampled":31,"step_s":0.5}

$ vidqa shot run.mp4 --out evidence.png --at-text "Payment failed"
{"at_s":8.25,"found":true,"height":800,"out":"evidence.png","query":"Payment failed","width":1280}

$ vidqa ci run.mp4 --rules checkout.rules.json      # gate it in CI
{"pass":true,"rules":[{"detail":{"first_s":10.5},"pass":true,"rule":{"by_s":15,"text":"Order confirmed","type":"expect_text"}},{"detail":{},"pass":true,"rule":{"type":"no_blank_frames"}}],"step_s":0.5}

Requirements

• Python 3.11+
• ffmpeg/ffprobe on PATH (installed separately; not bundled)
• Optional, for ask: Ollama with a vision model (default qwen3-vl:8b)
• Optional, for live (Windows only): Intel PresentMon — a copy is bundled in tools/
• Optional, for speech: faster-whisper (pip install vidqa-cli[speech]; runs on CPU, model downloads on first use)

Install

pip install vidqa-cli

(The PyPI name is vidqa-cli; the module and the command are both vidqa.)

From source, for development:

git clone https://github.com/DSmereski/vidqa
cd vidqa
python -m venv .venv
.venv/Scripts/pip install -e .[dev]   # Windows; use .venv/bin/pip elsewhere

Commands

Example:

$ vidqa timing capture.mp4
{"dup_ratio":0.0,"frame_count":3600,"frame_time_ms":{"p50":16.6667,"p95":16.6667,"p99":16.6833},"freeze_count":0,...}

$ vidqa ask capture.mp4 "What color is the health bar?" --enum red,yellow,green --expect green
{"answer":"green","confidence":"high","evidence":"...","frames_used":6,"model":"qwen3-vl:8b"}

Test recordings (web / Android / iOS)

Point vidqa at whatever your runner saved — a Playwright video, an adb screenrecord, an XCUITest attachment, an OBS capture:

vidqa when run.webm "Payment failed"                      # when did it appear?
vidqa locate run.webm "Payment failed" --shots evidence/  # last-good + first-bad frames
vidqa shot run.webm --out evidence.png --at-text "Payment failed"
vidqa strip run.webm --out sheet.png                      # whole run at a glance
vidqa ci run.webm --rules checkout.rules.json             # gate it in CI

Automated runners move fast: a state a human would watch for seconds can be on screen for under half a second, because the next click fires the moment the assertion resolves (measured on real Playwright runs: an error panel visible for 0.36 s). When gating on such flashes, tighten the sampling with --step 0.25. Also, the recording starts at the first painted frame — a form the test fills instantly may barely appear in the video at all. The video is evidence of what happened after the page rendered; VLM rubrics (judge) should ask about the states that dominate it, not about flashes.

A rules file turns a recording into a pass/fail check:

{"rules": [
  {"type": "expect_text", "text": "Order confirmed", "by_s": 20},
  {"type": "forbid_text", "builtin": "error_pages"},
  {"type": "max_freeze_s", "seconds": 3},
  {"type": "no_blank_frames", "after_s": 5}
]}

Design contract

• One JSON object on stdout. Sorted keys, 4-decimal floats, no timestamps or hostnames — reruns of deterministic commands are byte-identical.
• Exit codes mean one thing each. 1 always means "the check genuinely failed"; internal errors are always 2.
• Gate on determinism. VLM answers are schema-constrained (Ollama structured outputs at temperature 0). Enum answers are argmax-stable; free-prose evidence is best-effort context, never something to compare byte-wise. Automated gates should use deterministic fields or --enum/--expect only.
• Cheap first. For agents: deterministic metrics → OCR/template → local VLM. Most questions never need a model at all.
• timing freezes measure static content. Menus, loading screens, and an idle player all count — that is all recorded footage can show (verified against real game captures). For true performance hitches of a running build, use vidqa live (ETW ground truth).

MCP server

vidqa-mcp exposes every command as an MCP tool over stdio, for clients that speak Model Context Protocol:

{"mcpServers": {"vidqa": {"command": "/path/to/.venv/Scripts/vidqa-mcp"}}}

Live frametimes (Windows)

vidqa live wraps Intel PresentMon (bundled, MIT — see tools/PresentMon-LICENSE.txt) to measure the real present-to-present frametimes of a running process via ETW. ETW capture needs elevation once: run from an admin terminal, or add your user to the "Performance Log Users" group and sign back in.

Testing

.venv/Scripts/python -m pytest -q

151 tests; fixtures are synthesized on the fly with ffmpeg (injected freezes, dropped frames, seeded corruption, silence, clipping, drawn text) plus Windows text-to-speech for the speech tests, so no test media is checked in. eval/run_eval.py --runs 3 exercises the VLM lane end-to-end against a local Ollama model.

License

MIT (see LICENSE). The repository bundles the Intel PresentMon binary under its own MIT license (tools/PresentMon-LICENSE.txt). ffmpeg is a separate install and is not distributed here.