Visual Verifier

Anonymization QA for images and videos

Test blur, redaction, masking, and other visual processing frame by frame. Get a deterministic PASS/FAIL, temporal evidence, and CI-ready reports.

🔒 Runs entirely on your machine. Your media is never uploaded — the package has no network dependency at all, and a test enforces it on every commit.

pip install visual-verifier
visual-verifier demo

FAIL
Frames missed: 4, 8, 12

The problem

Your anonymizer says the video passed. Frame 4 still exposes the licence plate.

A blur, redaction, or masking pipeline reports success. Three frames out of fifteen were silently skipped. A global image metric averages the failure away. Manual frame review does not scale and does not reproduce.

Visual Verifier finds the frames your privacy pipeline missed, and answers one narrow question with evidence you can attach to a build:

Did the expected visual processing occur, in every frame, and can I catch it in CI when it does not?

Anonymization is the named use case, not the limit

The engine is a general pixel-level change verifier. It measures what actually changed between a reference and a candidate — region by region, frame by frame — so anything that alters pixels can be verified the same way:

Privacy work is where the failure is most expensive, so it is what the documentation leads with. The verification contract itself makes no assumption about why the pixels changed.

What it looks like

Fifteen dashcam frames. A licence-plate blur that silently skipped three of them. Watch the verdict flip on frames 4, 8, and 12:

The same run on the command line:

$ visual-verifier video \
    --reference examples/media/video_raw.mp4 \
    --candidate examples/media/video_blur_partial.mp4 \
    --output outputs/partial_check

Visual Verifier result
======================
Status:                    FAIL
Policy:                    generic_change_every_frame
Reference:                 examples/media/video_raw.mp4
Candidate:                 examples/media/video_blur_partial.mp4

Measurements
------------
Frames checked:            15
Frames with processing:    12
Frames without processing: 3
Processing coverage:       80.0%
Accepted regions:          32
Rejected regions:          0

Temporal tracking
-----------------
Tracks:                    15
Track observations:        32
Track events:              53
Tracks with gaps:          1
Mean continuity ratio:     0.986667

Failures
--------
UNPROCESSED_FRAMES: No accepted processing was detected in frames [4, 8, 12].
Failed frames:             4, 8, 12

Evidence
--------
annotated_video:           outputs/partial_check/annotated_video.mp4
frame_report:              outputs/partial_check/frame_report.csv
html_report:               outputs/partial_check/index.html
region_report:             outputs/partial_check/region_report.csv
rejected_region_report:    outputs/partial_check/rejected_region_report.csv
summary_json:              outputs/partial_check/summary.json
track_event_report:        outputs/partial_check/track_event_report.csv
track_observation_report:  outputs/partial_check/track_observation_report.csv
track_report:              outputs/partial_check/track_report.csv

$ echo $?
2

Frames 4, 8, and 12 were never blurred. The exit code fails the build, the CSV reports say exactly where, and annotated_video.mp4 shows a reviewer the same thing in ten seconds.

Install

Requires Python 3.10 or newer.

pip install visual-verifier
visual-verifier doctor

Or with uv:

uv tool install visual-verifier      # as a standalone CLI
uv add visual-verifier               # as a project dependency

To work on Visual Verifier itself, clone and run uv sync. On Windows PowerShell a bootstrap script sets the same environment up:

Set-ExecutionPolicy -Scope Process -ExecutionPolicy Bypass -Force
Unblock-File ./scripts/bootstrap_uv.ps1
./scripts/bootstrap_uv.ps1

visual-verifier doctor reports your Python, OpenCV, and codec support, so you can confirm the environment before trusting a result.

Quick start

No clone, no fixtures to download, no media of your own required:

visual-verifier demo

That generates a short sample clip in which a licence plate is blurred on every frame except three, verifies it, writes the full evidence set to ./visual-verifier-demo/report/, and prints the exact command to reproduce it. The sample is synthesized locally, so nothing is fetched.

Then point it at your own media:

visual-verifier video \
    --reference raw.mp4 \
    --candidate anonymized.mp4 \
    --output outputs/check

Add --json for a machine-readable document, --quiet to report only through the exit code, and visual-verifier video --help for every threshold.

Use it in CI

On GitHub, the bundled action installs the tool, runs the comparison, writes a job summary, and uploads the evidence:

- name: Verify anonymization coverage
  uses: SAMtheROCKET/visual-verifier@v0.3.0
  with:
    reference: fixtures/source.mp4
    candidate: build/anonymized.mp4

The summary names the failing frames, draws a frame timeline, and links the evidence artifact, so a reviewer sees the answer without downloading anything. docs/github_action.md lists every input and output.

Everywhere else the exit code is the contract, so no wrapper is required:

- name: Verify anonymization coverage
  run: |
    visual-verifier video \
      --reference fixtures/source.mp4 \
      --candidate build/anonymized.mp4 \
      --output artifacts/verification \
      --quiet

Errors print a stable machine-greppable code, for example ERROR [MEDIA_READ_ERROR], so a failing build tells you whether the media was bad or the processing was.

Python API

from visual_verifier import TrackingConfig, verify_video

result = verify_video(
    reference="examples/media/video_raw.mp4",
    candidate="examples/media/video_blur_partial.mp4",
    output_dir="outputs/partial_check",
    tracking_config=TrackingConfig(
        association_iou_threshold=0.20,
        minimum_confirmation_hits=2,
        maximum_gap_frames=3,
        lineage_overlap_threshold=0.20,
    ),
)

print(result.status.value)          # FAIL
print(result.failed_frames)         # (4, 8, 12)
print(result.measurements["tracking"]["track_count"])

Use it as a test assertion:

def test_every_frame_is_anonymized() -> None:
    verify_video("source.mp4", "anonymized.mp4").raise_for_failure()

Disable tracking while preserving frame-level verification:

result = verify_video("reference.mp4", "candidate.mp4", enable_tracking=False)

The package is fully typed and ships py.typed. Everything you need is exported from the top level: verify_image, verify_video, DetectionConfig, TrackingConfig, VerificationResult, VerificationStatus, and the structured error hierarchy rooted at VisualVerifierError.

How it compares

Averaging the difference between two frames is the check most teams reach for first. The Anonymization Gap Benchmark measures what that costs, over 78 generated sequences carrying 13 labelled failure modes:

The global metrics are handed a held-out calibration set to pick their best threshold. Visual Verifier runs at shipped defaults. They still drop to 0% recall on compressed, noisy, and offset-blur footage, because a metric that averages the whole frame cannot separate a missing blur from a re-encode. Region-by-region measurement can.

Three families defeat plain change detection entirely: weak blur, partially covered targets, and a missed plate among several. Every method above scores 0% on all three, because all three are the same gap: something was processed, but not the thing that mattered.

Telling the tool which region had to be anonymized closes all three and takes the benchmark to 100%:

visual-verifier video --reference raw.mp4 --candidate out.mp4 \
    --targets plates.csv --min-severity 50

See Target annotation.

Verification contract

Reference media
+ Candidate processed media
+ Detection configuration
+ Optional tracking configuration
= PASS/FAIL + measurements + temporal evidence

With processing required on every frame:

• PASS: every synchronized frame contains an accepted changed region.
• FAIL: one or more frames contain no accepted changed region.

The bundled regression fixture is enforced by examples/expected/demo_expectations.json, which the test suite executes on every run so the table below cannot drift:

Verifying a specific region

Supplying reviewed targets changes the question from did anything change to did the required region change:

visual-verifier video \
    --reference raw.mp4 \
    --candidate anonymized.mp4 \
    --targets plates.csv

A frame where two of three plates were blurred passes the generic check and fails this one. Target boxes between reviewed frames are interpolated and always marked as such, so a generated box is never mistaken for one a human drew. docs/target_annotation.md documents the file format and every limit.

Evidence outputs

Video verification with tracking enabled writes:

index.html                   Self-contained report you can email to a reviewer
annotated_video.mp4          Candidate with persistent track labels (T001)
frame_report.csv             One row per synchronized frame
region_report.csv            One row per accepted changed region
rejected_region_report.csv   One row per rejected region, with reasons
track_report.csv             One row per completed track
track_observation_report.csv One row per region-to-track assignment
track_event_report.csv       Lifecycle and split/merge lineage events
summary.json                 The complete machine-readable result

Every panel below is generated from a real run by scripts/generate_readme_assets.py, with the region boxes read back from track_observation_report.csv:

The report you actually send someone

index.html opens in any browser and needs no explanation of CSV schemas. It shows the verdict, a frame timeline where every unprotected frame links to its own evidence, and a before/after wipe slider for each failure.

It references nothing external — no CDN, no web font, no linked image. Thumbnails are embedded, so it works offline, survives being attached to a ticket, and opening it cannot signal to anyone that a privacy report exists. It is also deterministic, so two runs over the same media produce identical documents and can be diffed.

The primary temporal measures are continuity ratio, fragmentation index, missing frames and longest gap, mean and minimum association IoU, center jitter, area stability, recovery count, and split/merge involvement. See docs/temporal_tracking.md and docs/output_schema.md.

Current release

Version 0.3.0 is the V5.3 milestone: target-aware verification. The package is early software classified Alpha: the API and report schemas may still change before a stable release.

• Image-to-image and synchronized video verification
• Multiple changed-region detection, filtering, and severity scoring
• Reviewed targets declaring the regions that must be anonymized, with union coverage, bounded interpolation, and explicit provenance
• Deterministic IoU-based temporal association
• Tentative, confirmed, lost, recovered, and closed track lifecycles
• Configurable short-gap tolerance and split/merge lineage evidence
• Track continuity, fragmentation, stability, and recovery metrics
• A published benchmark measuring all of it against simpler methods
• Typed Python API and a self-documenting CLI
• Linux, macOS, and Windows CI across Python 3.10–3.13

Temporal tracking remains evidence-only: it does not silently alter the active frame- or target-level verification policy.

What this is not

A tracking ID represents a persistent changed region — not a proven face, licence plate, person, or other semantic object. V5.2 assumes synchronized media and performs no motion prediction, appearance re-identification, camera-motion compensation, or automatic temporal alignment.

Visual Verifier is an engineering QA tool. It is not a regulatory certificate and not a guarantee of anonymization. Read docs/limitations.md before relying on a PASS.

Documentation

Full documentation: https://samtherocket.github.io/visual-verifier/

Development

uv sync
uv run ruff format --check .
uv run ruff check .
uv run mypy src --python-version 3.12
uv run pytest -q --cov=visual_verifier

Or run the same gate in one step:

./scripts/run_quality.sh      # Linux and macOS

./scripts/run_quality.ps1     # Windows

Before a release, run the complete release gate. It checks the frame-level and temporal regression contracts, target-aware verification, CLI health, package contents, and distribution building:

./scripts/validate_v5_2.ps1

That script runs validate_release.ps1 first and then the frozen V5.2 temporal regression contract, so one command covers both. The V5.2 gate is retained deliberately: it pins the temporal behaviour that shipped in that milestone, so later work cannot drift it unnoticed. Run ./scripts/validate_release.ps1 alone when you only need the current gate without the historical contract.

Function length, line length, docstrings, annotations, control characters in documentation, and the agreement between the Git tag, the package version, and CITATION.cff are all enforced by Ruff and the test suite rather than by review.

Repository layout

src/visual_verifier/           Active package
src/visual_verifier/tracking/  Temporal association and integrity analysis
tests/                         Unit and end-to-end regressions
examples/media/                Reproducible 15-frame fixtures
examples/expected/             Executable demo contract
examples/targets/              Reviewed target example
docs/                          Architecture and behavior documentation
benchmarks/                    Measured comparison against simpler methods
scripts/                       Bootstrap, quality, cleanup, and release gates
action.yml                     Composite GitHub Action wrapping the CLI
archive/legacy_cells/          Read-only historical prototypes

Roadmap

V5.3 target-aware verification is complete. V5.4 adds the selectable policy system, and V5.5 batch verification. See ROADMAP.md.

Licence and citation

Visual Verifier is licensed under Apache-2.0. Citation metadata is provided in CITATION.cff.