Stabilo Benchmark

Stabilo Benchmark evaluates the complete registration pipeline of reference-based UAV video stabilization (feature detector, descriptor, matcher, robust estimator and object masking) on static and dynamic urban scenes. It is the benchmark of the paper "Consensus Capture in Reference-Based UAV Video Stabilization: A Benchmark on Dynamic Urban Scenes": the protocols, the metrics and the commands that regenerate every benchmark table and figure from the paper's data record. Pipelines are run through Stabilo 🌀, and objects are annotated with HBB2OBB 📦.

🎬 Both protocols on the Songdo data, one step per trial and per temporal offset. Top, Part I: a static 4K scene against a synthetic warp of itself, whose homography is known exactly. Bottom, Part II: a hover's reference frame against later frames of the same hover, from one frame to a minute and a half later. Blue lines are a sample of ORB's inlier matches, orange the oriented-box masks that keep the vehicles out of feature detection.

Why Stabilo Benchmark

• 🚗 Built for scenes that move. Feature-matching benchmarks score correspondences on scenes in which only the camera moves. Over an intersection, vehicles moving together supply mutually consistent correspondences that can draw the homography toward their own motion: consensus capture.
• 🎯 Scored on the ground, never on the inliers. Pipelines are ranked by how well they align the static world: against exact homographies in Part I, against an audited reference estimator and by the background's photometric alignment in Part II. Inlier statistics are diagnostics only.
• 🧭 Capture made observable. The registration error is decomposed along the dominant motion axis of the annotated objects, and a capture fraction measures how far an estimate follows the largest group of objects that moved together.
• 📍 An audited reference. Part II's pseudo-ground truth is checked against manually annotated ground control points, which also score every candidate directly.
• 🎭 Masking as a controlled axis. No masking, horizontal boxes (HBB) and oriented boxes (OBB), applied before feature detection to every pipeline alike.
• ⚖️ Pipelines on equal footing. Paired trials, per-pipeline tuning on a site-disjoint split, a matched resolution and feature budget, failures counted rather than dropped, and a pre-specified analysis plan (cluster bootstrap with an equivalence margin set by the audit).
• 📦 Every number from a command. The data record holds the inputs and every campaign's results; fetch downloads it and report regenerates the tables and figures.

Consensus Capture

🔍 One pair of a Songdo hover, a third of a second apart, registered by ORB at the real-time budget. Without masking (left), the estimate registers the platoon in the middle of the crop back onto where it stood: many of its inliers lie on the vehicles that moved (orange dots), and its error, drawn as black arrows magnified 11 times, carries their motion across the crop. Under oriented-box masks (right), the same pipeline finds its inliers on the ground and aligns it.

A homography that describes the moving foreground is as self-consistent as one that describes the camera, so the statistics a robust estimator reports cannot tell the two apart:

The benchmark therefore measures alignment from outside the estimator. It applies to any footage in which annotated objects move independently of the camera; vehicles at intersections are the paper's application.

Benchmark Design

Part I: static protocol. Each scene is warped by a random homography with photometric degradation (brightness, saturation, blur, haze, sensor noise, JPEG compression), and every pipeline recovers it. The ground truth is exact, so the protocol establishes what each pipeline can do when nothing moves, which of its settings matter, and what masking costs. Every pipeline is tuned on scenes from sites disjoint from the evaluation scenes, at a resolution and feature budget matched across pipelines, and faces the same perturbations in each trial.

Part II: dynamic protocol. Real hovers are registered frame by frame onto one reference frame, over temporal offsets from one frame to two minutes. The cases and their reference frames are chosen from the video alone to span how much of the foreground moves and how coherently, with a stationary control. Real footage has no true homography: a reference estimator supplies a pseudo-ground truth whose error is measured against ground control points, and the reference-free measures score the background only, outside every object and outside above-ground structure. Every pipeline runs at two points of Part I's budget grid: offline (half resolution, 2000 features, for registration after the flight) and real-time (the grid's cheapest point, for registration at video rate).

Pipelines. Ten detector and matcher combinations of Stabilo, each with its own tuned matcher and estimator thresholds: ORB, SIFT, RootSIFT, BRISK, AKAZE, XFeat, DISK, DeDoDe and KeyNet (the last three with LightGlue), and the detector-free LoFTR. The robust estimator is a further axis of Part I. Named presets are in stabilo_benchmark/cfg/pipelines.yaml; any Stabilo configuration can be benchmarked.

Metrics.

Object boxes are carried corner by corner through the homography and compared as quadrilaterals. OCE is the position error a trajectory inherits from registration; unlike IoU, it does not depend on object size.

Install

It is recommended to work from a clone, which holds the paper's configs, scripts and the sample data (Python 3.11 to 3.13):

git clone https://github.com/rfonod/stabilo-benchmark.git
cd stabilo-benchmark
python3.11 -m venv .venv && source .venv/bin/activate  # Windows: .venv\Scripts\activate
pip install -e .

HBB2OBB, which writes the object annotations, is installed with it. pip install stabilo-benchmark installs the package and its command alone, to run your own configs on your own data.

uv venv --python 3.11 && source .venv/bin/activate && uv pip install -e .   # uv
conda create -n stabilo-benchmark python=3.11 -y && conda activate stabilo-benchmark && pip install -e .
pip install -e '.[dev]'                                                      # adds pytest and ruff

Quick Start

data/sample/ holds a subset of the paper's data record, so every command runs from a fresh clone without a download, video, GPU or detector weights. Part I, three of the paper's 29 static scenes, those with the fewest, the median and the most vehicles:

stabilo-benchmark static configs/sample/static.yaml -v 1

Static benchmark 'smoke': 6 pipeline(s) x 3 scene(s) x 5 trial(s) = 90 evaluations
  orb+none      median CTE  0.129 px | failure rate 0.0% | n=15
  orb+hbb       median CTE  0.155 px | failure rate 0.0% | n=15
  ...
  sift+obb      median CTE  0.070 px | failure rate 0.0% | n=15

Part II, ten frames of one of the paper's cases, a hover over an intersection, with their labels, ground control points and exclusion polygons, in the protocol's order: the reference estimator is audited before any candidate is scored.

case=data/sample/dynamic/2022-10-06_D9_AM3_S2
stabilo-benchmark oracle  $case -c configs/sample/dynamic.yaml
stabilo-benchmark audit   $case
stabilo-benchmark dynamic $case -c configs/sample/dynamic.yaml -v 1
stabilo-benchmark audit   $case --results $case/results/smoke

Reference estimator audit against ground control points
  ...
  pooled over 117 point(s): median 0.12 px, p95 0.45 px, max 1.01 px
  ...
Dynamic benchmark 'smoke': 6 pipeline(s) x 9 frame pair(s) = 54 registrations, plus the identity and reference baselines
  orb+none      median CTE 0.81 px | SSIM 0.914 | align 0.61 | inlier ratio 0.57 | failures 0.0%
  orb+hbb       median CTE 0.40 px | SSIM 0.935 | align 0.61 | inlier ratio 0.78 | failures 0.0%
  orb+obb       median CTE 0.40 px | SSIM 0.935 | align 0.59 | inlier ratio 0.78 | failures 0.0%
  ...

Then the LaTeX tables and PDF figures, into reports/ unless --output says otherwise:

stabilo-benchmark report data/sample/static/results/smoke
stabilo-benchmark report $case/results/smoke

Results land in results/smoke/ inside each part. The sample is a smoke test, never evidence: three scenes and nine frame pairs are too few to rank pipelines. data/sample/README.md describes where each file comes from and how to try the annotation tools on it.

Usage

static and dynamic take --results-dir <dir> to write the campaign to <dir>/<name>, -o to overwrite and -r to resume. stabilo-benchmark <command> -h lists every flag; python -m stabilo_benchmark is equivalent.

Static campaign

# 1. Annotate <scenes>/images with hbb2obb (see "Annotating images")

# 2. Tune each pipeline on scenes held out from the evaluation set
stabilo-benchmark static configs/songdo/static/tuning.yaml --scenes <scenes> \
    --scenes-list <tuning_scenes.txt> -v 1

# 3. Score the tuned pipelines on the evaluation scenes, then build the tables and figures
stabilo-benchmark static configs/songdo/static/detectors.yaml --scenes <scenes> \
    --scenes-list <eval_scenes.txt> -v 1
stabilo-benchmark report <scenes>/results/detectors --kind static --output <out>

--scenes-list restricts a scenes directory to the names in a text file (one per line), which keeps the tuning and evaluation scenes disjoint. docs/songdo_static.md describes the paper's campaigns, with its scene split in configs/songdo/static/.

Dynamic campaign

# 0. Choose which hovers become cases, and their reference frames, from the recordings alone
stabilo-benchmark survey <videos>/*.mp4 -c configs/songdo/dynamic/offline_masking.yaml \
    --anchor scan --max-per-video 1 -k 2 --controls 1

# 1. Build a case at the reference frame the survey chose, then annotate <case>/images with
#    hbb2obb. --cell records the survey cell of this segment.
stabilo-benchmark prepare <videos>/intersection_a.mp4 -o <cases>/intersection_a \
    -c configs/songdo/dynamic/offline_masking.yaml --reference 1800 --cell "high moving / coherent"

# 2. Cache the reference estimator's homographies
stabilo-benchmark oracle <cases>/intersection_a -c configs/songdo/dynamic/offline_masking.yaml

# 3. Outline above-ground structure (buildings) on the reference frame, in LabelMe; the config's
#    metrics.region.exclusion leaves it out of the evaluation region (null excludes nothing)
./scripts/draw_exclusion.sh <cases>/intersection_a

# 4. Place ground control points: by hand in the reference frame; in later frames each opens on a
#    proposal (a local image fit, independent of the reference estimator) that Enter accepts and a
#    click corrects (--no-propose places every point by hand; --repeat is a second, unaided session;
#    --labels re-places single points in the given frames), then audit
stabilo-benchmark annotate-gcp <cases>/intersection_a
stabilo-benchmark annotate-gcp <cases>/intersection_a --frames 0 --labels p02
stabilo-benchmark annotate-gcp <cases>/intersection_a --repeat
stabilo-benchmark audit <cases>/intersection_a

# 5. Benchmark, score the candidates against the ground control points, and report
stabilo-benchmark dynamic <cases>/intersection_a -c configs/songdo/dynamic/offline_masking.yaml -v 1
stabilo-benchmark audit <cases>/intersection_a --results <cases>/intersection_a/results/offline_masking
stabilo-benchmark report <cases>/intersection_a/results/offline_masking <cases>/intersection_b/results/offline_masking

audit precedes dynamic, so the reference estimator's error is measured before any candidate number is read. Its 95th percentile is the margin below which differences between candidates are not interpreted; report states it with every Part II table. docs/songdo_dynamic.md is the full recipe of the paper's cases.

Read 831 segment(s) across 370 recording(s)
Splits (pool median): moving 0.675% of the frame, coherence 0.422; controls: stationarity >= 0.902 and static >= 1.408%

segment            cover%  static  moving  coher.  still  texture  margin  cell
-------------------------------------------------------------------------------
D10_PM5_U1@001170    3.00    0.66    1.94    0.80   0.25     3606    4.21  high moving / coherent
D9_AM3_S2@000300     3.96    1.76    1.89    0.22   0.48     4990    2.23  high moving / mixed
D7_PM1_P1@000300     1.81    1.53    0.17    0.74   0.90     4757    1.94  low moving / coherent
D7_PM5_P2@001200     2.35    1.95    0.24    0.23   0.89     3936    1.66  low moving / mixed
D9_AM1_R1@002100     2.02    1.94    0.00       -   1.00     4068    0.52  stationary control
...
Design:
  high moving / coherent   2/2  ok  (pool: 173)
  high moving / mixed      2/2  ok  (pool: 179)
  low moving / coherent    2/2  ok  (pool: 179)
  low moving / mixed       2/2  ok  (pool: 172)
  stationary control       1/1  ok  (pool: 56)
  72 candidate(s) unclassified: too few displaced objects for coherence

Annotating images

Object boxes drive the masking axis, so annotate your frames. Part I scenes and Part II cases share one layout:

<root>/
  images/              the frames
  labels_hbb/          horizontal object boxes      -> masking: hbb
  labels_obb/          oriented object boxes        -> masking: obb, and for every mode the metric regions and IoU/OCE
  labels_confidence/   per-box conversion score
  PROVENANCE_hbb.txt   PROVENANCE_obb.txt

HBB2OBB provides two commands. hbb2obb-detect writes the horizontal boxes, by default with the Geo-trax vehicle detector for high-altitude drone imagery (car, bus, truck, motorcycle). hbb2obb then prompts SAM with each horizontal box and writes its oriented box, in the same order. The settings the benchmark uses:

hbb2obb-detect <root>/images --model geotrax --no_confidence --save_provenance
hbb2obb <root>/images --sam_models sam_l --imgsz 1024 --scale_factors 0.02 --opening_kernel_percentage -0.1 --fragment_ratio 0.1 --confidence_dir --save_provenance

• Add --device 0 (or cpu, mps) to either command.
• Checkpoints download into models/ under the working directory unless HBB2OBB_MODELS_DIR or --models_dir points elsewhere.
• Run both commands from the directory holding <root>, with relative paths: hbb2obb writes the command and its paths into the provenance files as typed.
• Where SAM finds no usable mask, hbb2obb keeps the horizontal box and scores it 0.0 in labels_confidence/.

The benchmarks read these labels and never run a model themselves (only survey calls the two commands above), so every pipeline receives identical masks.

stabilo-benchmark static configs/sample/static.yaml --scenes <scenes> --set pipeline.masking=none
stabilo-benchmark oracle <case> -c <config> --set oracle.masking=none
stabilo-benchmark dynamic <case> -c <config> --set pipeline.masking=none --set oracle.masking=none

Configuration

Experiments are YAML files (JSON also loads). Every quantity that shapes a reported number is a config key, so a result is reproducible from its manifest:

kind: static
seed: 0
trials: 100

perturbation:
  geometric: { translation_ratio: 0.10, rotation_deg: 15.0, scale_ratio: 0.05, projective: 5.0e-5 }
  photometric:
    - { name: brightness, range: [0.75, 1.25] }
    - { name: noise, max_sigma: 3.0 }
    - { name: jpeg, quality: [70, 95] }

pipeline: # the Cartesian product of these lists becomes the run grid
  preset: [orb, sift, rsift, xfeat, disk_lightglue, loftr]
  masking: [none, hbb, obb]

--set key.path=value overrides any existing key, so a one-parameter sweep needs no file per point:

for deg in 0 15 30 45 60 90; do
  stabilo-benchmark static configs/songdo/static/rotation_sweep.yaml --scenes <scenes> \
    --set perturbation.geometric.rotation_deg="[$deg,$deg]" --name "rotation_${deg}deg" -o
done

IoU and OCE are computed on the oriented boxes for every run, whatever its masking strategy. To use the horizontal boxes, set metrics.object_boxes: hbb; report then labels the headers IoU_HBB and OCE_HBB and refuses to mix the two settings. The paper's campaigns are in configs/songdo/.

stabilo_benchmark/      the package (what `pip install` gets): code and the named pipeline presets
configs/
  sample/               static.yaml, dynamic.yaml: the quick start
  songdo/               the paper's campaigns: static/*.yaml with the scene split, dynamic/*.yaml
                        with the case lists
scripts/                songdo_static.sh, songdo_dynamic.sh: run one campaign of the paper;
                        draw_exclusion.sh: a case's above-ground polygons, in LabelMe
docs/                   songdo_static.md, songdo_dynamic.md: how the paper's data were built
data/
  registry.yaml         the Zenodo record `fetch` downloads, with its pinned checksums
  sample/               committed: a subset of the paper's record, for the quick start (README.md)
  songdo/               not committed: the paper's data and results, from `fetch songdo`
reports/                not committed: the tables and figures `report` writes by default
assets/                 the images of this README

The Songdo Experiment

The paper's benchmark runs on 4K footage of a multi-drone traffic experiment over Songdo, South Korea, recorded from 140 to 150 m (see the background paper). Part I uses 29 frames of its static campaign, split by site into tuning and evaluation scenes. Part II uses nine hovers over intersections, two per cell of a 2×2 design on moving foreground × coherence plus a stationary control, chosen by survey from the recordings; each case holds its reference frame and 135 later frames in twelve octave-wide offset bins up to two minutes. Five further cases from other intersections tune Part II's pipelines.

🗺️ The nine Part II cases at their reference frames, each read as survey reads it 22 frames later: objects found displaced together with the largest group (orange), displaced otherwise (blue), found where they were, or not found.

Getting the Data

The data record, 10.5281/zenodo.23623540 (CC BY 4.0), holds Part I's scenes, Part II's evaluation and tuning cases with their ground control points, the surveys they were selected from, the results of every campaign, the CSV files the paper names, and a copy of configs/songdo/. The source videos are not released; the cases are published as their sampled frames. To recreate data/songdo/, from the repository root:

stabilo-benchmark fetch --list     # what is registered and what is present
stabilo-benchmark fetch songdo     # download, verify, extract into data/songdo/ (the default dataset)
stabilo-benchmark fetch --check    # later: re-verify what is on disk, download nothing

fetch downloads the record's five archives with its README, license and SHA256SUMS (6.2 GB), checks each file against the size and MD5 pinned in data/registry.yaml, and extracts the archives in place (8.7 GB). The archives stay in data/songdo/ so that --check can re-verify them; delete them once extracted if space is short. It prints the download size and the free space first, refuses a download that would not fit, asks before one over 1 GB (--yes skips the question), and resumes an interrupted download.

Versions. Each version of a Zenodo record has its own DOI, and the registry pins one: this release fetches v1, the version its results were produced with, and verifies every file against that version's sizes and checksums, so a later version is never picked up silently. A release that moves to a later version of the record updates the registry with it, while an earlier tag of this repository still fetches the version it was run with. fetch refuses to extract one version over another; move the old copy aside first.

Adding your data. fetch is not tied to the paper's record: it downloads whatever data/registry.yaml lists, and --registry <file> points it at another registry. If you publish a dataset in the benchmark's layout on Zenodo, scenes or prepared cases as in data/sample/, please open a pull request that adds its entry to the registry, so that others can fetch it by name; the file's header shows the fields and the checks to run.

cd data/songdo
shasum -a 256 -c --ignore-missing SHA256SUMS    # or: sha256sum -c --ignore-missing SHA256SUMS
for z in *.zip; do unzip -q "$z"; done

Reproducing the Paper

Tables and figures from the published results. report reads the stored campaigns and writes LaTeX tables, CSV files and PDF figures into reports/tables/ and reports/figures/ (--output writes them elsewhere):

D=data/songdo
for c in budget tuning detectors detectors_default detectors_hover estimators oracle_tuning; do
  stabilo-benchmark report $D/static/results/$c --kind static
done
stabilo-benchmark report $D/static/results/rotation_*deg
stabilo-benchmark report $D/survey/SURVEY.json
for c in offline_tuning realtime_tuning; do
  stabilo-benchmark report $D/dynamic_tuning/*/results/$c
done
stabilo-benchmark report $D/dynamic/*/results/offline_margin
stabilo-benchmark report $D/dynamic/*/results/offline_repeats --kind dynamic
for c in realtime_masking offline_masking; do
  stabilo-benchmark report $D/dynamic/*/results/$c --audit-case 2022-10-06_D9_AM3_S2
done

Figures that register a stored pair again (detectors_trial.pdf, *_capture_example.pdf and the graphical abstract) are drawn only when the new registration reproduces the stored estimate; otherwise report skips them and says so.

Rerunning a campaign. Each campaign of the paper is one call of a script, from the repository root. RESULTS_DIR writes your rerun beside the published results instead of over them, so the two can be compared:

RESULTS_DIR=reruns/static ./scripts/songdo_static.sh detectors          # -> reruns/static/detectors
RESULTS_DIR=reruns/dynamic ./scripts/songdo_dynamic.sh offline_masking  # -> reruns/dynamic/<case>/offline_masking
stabilo-benchmark report reruns/static/detectors --kind static --output reruns/paper

Without RESULTS_DIR the scripts write into data/songdo/, where an existing campaign is skipped unless -o (overwrite) or -r (resume) is passed. docs/songdo_static.md and docs/songdo_dynamic.md describe every campaign, the order they run in, and how the scenes and cases were built.

Citation

If you use Stabilo Benchmark or its data, please cite the paper:

@article{fonod2026benchmark,
  title   = {Consensus Capture in Reference-Based UAV Video Stabilization: A Benchmark on Dynamic Urban Scenes},
  author  = {Fonod, Robert and Geroliminis, Nikolas},
  journal = {arXiv preprint},
  year    = {2026}
}

When you use the data record, please also cite it:

@dataset{fonod2026benchmarkdata,
  title     = {Accompanying data for the paper "Consensus Capture in Reference-Based UAV Video Stabilization: A Benchmark on Dynamic Urban Scenes"},
  author    = {Fonod, Robert and Geroliminis, Nikolas},
  year      = {2026},
  version   = {v1},
  publisher = {Zenodo},
  doi       = {10.5281/zenodo.23623540}
}

The background paper introduced the stabilization approach, the imagery and the static campaign this work extends:

@article{fonod2025advanced,
  title   = {Advanced computer vision for extracting georeferenced vehicle trajectories from drone imagery},
  author  = {Fonod, Robert and Cho, Haechan and Yeo, Hwasoo and Geroliminis, Nikolas},
  journal = {Transportation Research Part C: Emerging Technologies},
  volume  = {178},
  pages   = {105205},
  year    = {2025},
  doi     = {10.1016/j.trc.2025.105205}
}

Contributing

Contributions are welcome. If you encounter an issue or have a suggestion, please open a GitHub Issue or submit a pull request. Datasets are welcome too: a Zenodo record in the benchmark's layout joins fetch through a pull request that adds it to data/registry.yaml (see Adding your data).

License

The code is distributed under the MIT License; see LICENSE. The data record and the sample are under CC BY 4.0.