data-doctor

Diagnose vision datasets before they lie to you: duplicate images, train/val leakage, and annotation defects — in one command, with no heavy dependencies (Pillow only).

Why

Metrics are only as honest as the split behind them. This tool exists because of a real production dataset whose delivered train/val split contained 81 groups of byte-identical images spanning both sides — the reference model scored 0.75 recall on that split and substantially less on an honest one. Nothing in the training stack warned about it. data-doctor makes that check (and its cousins) a 10-second habit instead of a post-mortem.

Install

pip install data-doctor

Usage

Find corrupt files, exact duplicates, and near-duplicates in a folder:

data-doctor scan data/images --json report.json

Check a train/val split for leakage (byte-identical and perceptually near-identical frames on both sides):

data-doctor leakage --train data/train --val data/val

train: 714 images | val: 231 images

  FAIL exact leaks (byte-identical in both splits): 81
         val/frame_0117.jpg == train/frame_0116.jpg
         ...
  FAIL near leaks (pixel-verified >= 90% similar): 12
         val/frame_0201.jpg ~~ train/frame_0200.jpg (99.4% similar)
         val/shot_114.jpg ~~ train/shot_113.jpg (98.7% similar, rot90)

  40.3% of the validation set is leaked from train.
  Metrics measured on this split overstate real performance.

Structural checks on a COCO annotation file:

data-doctor coco annotations/train.json --images data/images

Checks: duplicate image ids and file names, annotations referencing missing images, unknown category references, degenerate boxes (zero width/height), degenerate polygons (< 3 points), referenced files missing on disk, and a count of zero-annotation images (hard negatives or missing labels — you decide which, the tool makes sure you see them).

Exit codes

0 when clean, 1 when any check fails — drop it straight into CI:

- run: data-doctor leakage --train data/train --val data/val

Python API

from data_doctor import scan_directory, check_leakage, check_coco

result = check_leakage(Path("data/train"), Path("data/val"))
print(result.leaked_val_fraction)

How near-duplicate detection works

Three stages, so every reported duplicate is a concrete, verified claim — not a fuzzy hash coincidence:

1. Exact — SHA-256 catches byte-identical copies.
2. Nominate — two perceptual hash families (difference hash and DCT pHash), computed over all 8 rotations/flips, propose candidate pairs. Candidates are found by 16-bit quadrant bucketing (pigeonhole: any two hashes within hamming distance 3 share an identical quadrant), so scans stay fast on large folders.
3. Verify — every candidate pair is confirmed on decoded pixels: grayscale thumbnails compared across the 8 dihedral orientations. Only pairs at or above the similarity floor are reported, each with its measured score and matching orientation (99.4% similar, rot90).

Hashes alone never convict — they only nominate. Tune recall with --near-threshold (default 3) and precision with --min-similarity (default 0.90). Rotated, flipped, re-encoded, and resized copies are all caught; the JSON report carries every verified pair with its score.

License

MIT