framelock 0.6.1

Synchronized multi-camera recording for multi-view pose estimation: OAK-D + iPhone RTSP + mic on one sub-ms clock, with rig registration, corresponded exports, and 3D playback

framelock

Synchronized multi-camera recording for multi-view pose estimation. Point an OAK-D, a handful of iPhones, and a microphone at the same scene; framelock records everything into one session folder on one sub-millisecond shared clock, solves where every camera is in a single world frame, and hands you corresponded, GVHMR-ready exports.

• One clock. Hardware-locked shutters inside an OAK, a verified re-implementation of DepthAI's sync across devices, acoustic chirp calibration for phones — every sample from every sensor lands on one host-monotonic timeline, and every claim is cross-checked by an independent estimator before it's trusted.
• One world frame. Press b mid-take: the laptop shows a fiducial board with a live per-camera status strip, and the whole rig — static cameras, phones via ARKit odometry, all three OAK cameras fused through factory extrinsics — is solved automatically at session end (~1 cm / 1° per camera, gravity-leveled). Printed wall tags extend it anywhere the board can't reach.
• Beyond pixels. iRTSP phones stream ARKit pose, IMU, GPS, compass, barometer, and LiDAR on the video's own clock (no offset estimation — the app's clock design makes it exact); OAK-Ds contribute IMU, stereo depth, feature tracks, and thermals.
• Verified, not hoped. Gyro-vs-pixels sync checks, gravity cross-checks between accelerometers and ARKit, stereo-metric size votes, pose-noise grading — --verify tells you what the take is actually worth.

pip install framelock            # or: uv add framelock
pip install 'framelock[viz]'     # + 3D playback (viser)

One CLI, sensible subcommands — framelock help lists them:

framelock / framelock record	record a session (bare flags work: framelock --shutter-sync ...)
framelock register	solve rig registration (board wave / tags / acoustic)
framelock tags	export printable wall fiducials at true physical scale
framelock correspond	corresponded exports + end-to-end verification
framelock align	acoustic label-bias / clock-drift measurement
framelock preview	synchronized side-by-side playback with transport
framelock viz	fly through the take in 3D (viser)
framelock report	session health report

Beyond video: iRTSP phones stream odometry (fused IMU, ARKit 6-DOF pose, GPS, compass, barometer, intrinsics, LiDAR depth) and framelock records it on the same timeline as the frames — no offset estimation, the app's clock design makes it exact (see docs/OUTPUTS.md). OAK-Ds contribute their own IMU, stereo depth, feature tracks, and thermals via OakD(data=("imu", "depth", ...)). Everything lands in the corresponded export together, framelock correspond --verify checks the odometry against the pixels, and framelock viz flies you through the take in 3D.

The whole rig is registered into one world frame (registration.json). The primary ritual costs one keypress: press b during a take and the laptop shows a fiducial board fullscreen with a live per-camera status strip; show it to each camera (the solver verifies the laptop's stillness from the data) and the rig is solved automatically at session end — every camera ~1 cm / 1°, phones via their ARKit odometry with timed drift anchors, all OAK cameras fused through factory extrinsics, gravity-leveled z-up world (or pick your own origin: world_origin="board" / any device / its Source object). Board size is cross-voted by independent estimators (ruler > stereo triangulation / ARKit motion > screen EDID), and every claim is verified by an independent witness — gravity cross-checks between the OAK accelerometer and each phone's ARKit measure the solved rotation's error board-free. The witness rules are strict about who may testify: only a camera that stayed put (its own IMU vouching for it) can convict the laptop of moving, because a phone cannot tell board motion from its own ARKit drift — measured on a live take, a phone's frame wandered 15.8 cm between two board showings while the parked OAK proved the board still to 0.7 cm. That difference is reported as the drift it is, and absorbed by timed anchors. Every take writes report.json: the numbers behind every line the console printed — the witness table, each phone's drift, how many frames each camera actually solved. Printed wall tags (docs/TAGS.md) extend it: framelock tags grid 0-11 --cols 4 --mm 40, pin anywhere fixed, and localized devices survey them automatically — every later sighting pins phone drift or rescues a camera the board never reached. The acoustic stop-and-chirp ritual (g) remains as a no-line-of-sight fallback. Registered devices share a metric world cell in the viz, with the board and tags drawn at their solved poses.

# The classic single-OAK setup (all flags from the original recorder):
uv run framelock --shutter-sync --exposure 8000 --iso 800 --warmup auto -n take1

# Add an iPhone (any RTSP-server camera app):
uv run framelock --shutter-sync --exposure 8000 --iso 800 \
    --rtsp iphone=rtsp://192.168.1.23:8554/live

# Digital zoom ramp 1x -> 2x between t=5s and t=10s of the recording:
uv run framelock --zoom-ramp 5:10:1.0:2.0

The CLI is a thin veneer — everything is a first-class Python API (see examples/ for runnable scripts covering each feature):

from framelock import OakD, RtspCamera, Session, Warmup, ZoomRamp

oak = OakD(shutter_sync=True, shutter=1/125, iso=800,
           automations=[ZoomRamp(1.0, 2.0, start=5, end=10)])
phone = RtspCamera("rtsp://192.168.1.23:8554/live", name="iphone")

with Session("recordings", name="take1", warmup=Warmup.AUTO) as session:
    session.record(oak, phone)

Options are typed StrEnums (VideoFormat.MP4, Transport.TCP, ExternalSync.MASTER, Control.ZOOM, ChirpKind.VELVET, Warmup.AUTO) — IDE-completable and validated, while plain strings keep working everywhere. Dynamic control is first-class too: Func("zoom", lambda t: ...) drives a control with any function of time, and session.record(..., on_tick=...) runs arbitrary per-tick logic — apply() is rate-safe (per-control resolution filtering built into every source), so both can emit values naively (see examples/04_live_control.py).

Previews are optional (ui=False, or preview=False per source) and render as one CCTV-style composite window — labeled tiles, REC indicator, "no signal" placeholders. Start/stop is console-driven: warmup=Warmup.MANUAL streams and settles until you press r in the terminal; q stops (with the end chirp when end_chirp=True); session.request_start() / request_stop() are the programmatic equivalents (thread-safe).

Output layout

recordings/take1/
  session.json               manifest: sources, clock map, settings, accuracy notes
  sync_groups.csv            cross-device alignment (per group: each stream's seq + ts)
  registration.json          every device's pose in ONE world frame (+ board, tags)
  audio_alignment.json       measured per-source label bias / clock drift
  report.json                the evidence behind every console line (read this first)
  corresponded/              equal-frame-count videos + pose-per-frame exports
  oak/
    CAM_A.mp4  CAM_A.mcap  CAM_A.intrinsics.json  CAM_A.timestamps.csv
    CAM_B.*    CAM_C.*     calibration.json
    depth.mkv  features.csv  imu_*.csv  system.csv    with data=(...)
  iphone/
    video.mp4  video.timestamps.csv
    imu.csv  pose.csv  gnss.csv ...   when the iRTSP app streams them

Every video has a *.timestamps.csv (file frame index ↔ sequence number ↔ host-monotonic + epoch time) and sync_groups.csv joins them across devices, so post-hoc alignment never depends on the live estimates. Full column reference for every file: docs/OUTPUTS.md.

uv run framelock viz recordings/take1   # fly through the take in 3D:
# camera frusta with their images, phone trajectories from ARKit odometry,
# play/pause/scrub/step/speed, click any camera for its full details
# (needs the viz extra: uv sync --extra viz)

Synchronization: what you actually get

Link	Mechanism	Accuracy
OAK, all three cameras (one device)	hardware FSIN (shutter_sync=True): the colour camera drives the pulse, both monos follow	0.039 ms colour↔mono, 0.013 ms mono↔mono (measured, 1080p30)
OAK mono pair only (shutter_sync="mono")	FSIN CAM_C→CAM_B; colour free-runs	monos ~0.01 ms; colour ±½ frame (3–16 ms)
Two OAK devices, one host	dai XLink timesync → host clock	timestamps <1 ms comparable; shutters free-run (±½ frame) unless FSIN-wired (PoE M8 / FFC hardware only — not the USB OAK-D)
iPhone / RTSP	software only (no hw path exists into a phone)	live: tens of ms (arrival-anchored); post-hoc with sidecars + clap calibration: ~1 frame

The host-side Synchronizer re-implements DepthAI's dai.node.Sync algorithm (verified against depthai-core v3.7.1 source) for any number of streams, with one deliberate improvement: nearest-match grouping instead of first-fit, so same-rate streams can't lock into a stable one-frame-off pairing.

Camera controls & automations

OakD accepts initial focus/exposure at construction (zoom=True enables the zoom chain, which starts at 1x), live control via oak.apply(zoom=1.5), or a timeline:

from framelock import FocusStep, ZoomRamp
OakD(automations=[
    ZoomRamp(1.0, 2.0, start=5, end=10),    # smooth digital zoom
    FocusStep([(0, 120), (8, 200)]),         # refocus at t=8s
])

Every built-in control has all three shapes — ZoomRamp/ZoomStep/ZoomFunc, FocusRamp/FocusStep/FocusFunc, ShutterRamp/ShutterStep/ShutterFunc, IsoRamp/IsoStep/IsoFunc — as conveniences over the generic Ramp(control, ...) / Step(control, ...) / Func(control, fn) / Automation(control, keyframes, mode), which drive any control a source supports.

Digital zoom is a runtime ImageManip center-crop on the color camera (output size constant, H.264-safe). The intrinsics sidecar automatically switches to per-frame fx/fy/cx/cy arrays that track the zoom exactly; distortion coefficients are crop-invariant and stay valid throughout.

Automatic offset calibration (audio)

The laptop's microphone doubles as an acoustic sync reference: record it as a source (a wideband marker plays automatically at recording start — a pleasant pseudo-random "tss" by default; --chirp-tone mls|velvet|sweep), then cross-correlate it against the phone's audio track — any shared sound works (the chirp, a clap, speech). Yields the phone's timeline offset at ~1 ms precision plus its clock drift:

uv run framelock --rtsp iphone=rtsp://... --mic -n take1
uv run framelock align recordings/take1
#   label bias +43.10 ms (drift +3.2 ppm ...) -> RtspCamera(offset=-0.0431)

Sound is slow (~2.9 ms/m): if the sound source is at different distances from the phone and the laptop, pass --distance-phone / --distance-mic (meters) and optionally --temperature (°C, refines the speed of sound) and the propagation difference is corrected. --chirp-period N repeats the chirp for drift markers across long, quiet takes. Audition all the marker sounds in docs/chirps/ (one .wav per tone) before choosing via --chirp-tone / Microphone(chirp=...).

Going deeper

• docs/SYNCHRONIZATION.md — the full sync model: clock domains (including the macOS steady_clock trap), hardware vs software tiers with measured accuracies, the Synchronizer algorithm, and the phone-calibration workflow.
• docs/OUTPUTS.md — every output file and column: sidecars, manifests, and how to join them for offline alignment.
• docs/TAGS.md — printed wall fiducials: export, print at true scale, affix, and let the rig survey them.
• docs/CAMERA_METADATA.md — the GVHMR intrinsics-sidecar spec the OAK sidecars target.
• examples/ — runnable scripts: single OAK, OAK+iPhone with acoustic calibration, zoom automations, live on_tick control, custom sync chirps, offline analysis, mobile rigs (end chirp), air conditions / speed-of-sound models (Gavioso 2025 vs Cramer 1993), and multi-iPhone rigs (per-phone alignment + pixel-level verify).
• docs/MEASUREMENTS.md / docs/CHIRP_BENCHMARK.md — reference-rig measurement snapshots; regenerate for any session or machine with framelock report session <dir> / framelock report chirps --live.
• framelock --help / framelock align --help / framelock preview --help / framelock report --help — full CLI reference.

Development

uv sync                      # deps (+ dev group: pytest, ruff, pre-commit)
uv run pytest                # full suite (~70 s, synthetic-physics GT tests)
uv run ruff check .          # lint (CI enforces both)
uv run pre-commit install    # once per clone: lint on every commit
uv run framelock preview     # side-by-side playback of the newest session