dset — version control for datasets

Commit, diff, and roll back millions of files the way you already do with code — and see what actually changed: samples, classes, label distributions. Not bytes.

Pure Python, zero dependencies, single file.

Install

pip install .

(from this folder — or just run python dset/cli.py ... directly)

Quick start

cd my-dataset
dset init
dset add images/ labels.csv
dset commit "June survey, initial labels"

# ... collect more data, fix labels ...

dset add .
dset commit "July imagery + relabeling pass"

dset diff v1 v2

Output:

comparing v1 → v2  (301 → 406 files)

  files       +120   −15   ~1 modified
  images      +120   −15
  labels      ~1 modified
  size        1.5 MB → 2.1 MB  (+568.2 KB)

  class       acacia  19.7% → 38.8%  (+19.1%)
  class       shrub   43.7% → 33.3%  (−10.3%)
  class       bare    36.7% → 27.9%  (−8.8%)

Go back to any version, exactly:

dset checkout v1

Commands

Refs can be a tag (v3), HEAD, or a commit-id prefix.

How it works

• Content-addressed storage. Every file is hashed (SHA-256) and stored once under .dset/objects/, like git. A new version costs only what changed; re-adding identical data costs nothing.
• Commits are manifests — JSON maps of path → hash. Checkout rebuilds the working tree from objects, restoring and deleting as needed.
• Semantic stats are computed at commit time (file counts by type, total size, class distribution parsed from CSV/TSV label files with a label/class/category/target column), so dset diff is instant even on huge datasets.
• Safe by construction. Objects are copied (never hard-linked), written atomically, and stored read-only, so editing a working file can never corrupt history. On btrfs/XFS the copy is a free copy-on-write clone.
• Uncommitted changes block dset checkout unless you pass --force.

dset check — the quality engine

dset check ./my-dataset            # scan and score
dset check ./my-dataset --fix      # also quarantine exact dups + corrupted

Scans every image (parallel, ~2,000 images/sec) and reports:

• corrupted / unreadable images
• exact duplicates (content hash) and near-duplicates (perceptual dhash + LSH)
• train/test leakage — exact and near-duplicate pairs across train/ and test/
• class imbalance (parsed from CSV/TSV label files)
• outliers: anomalous file size, odd resolutions, near-blank images
• images missing from label files, and label rows pointing to missing files

Prints a health score out of 100 and writes a browsable dset_report.html. --fix never deletes anything — it moves exact duplicates and corrupted files to _dset_quarantine/ for you to review.

Requires Pillow (installed automatically with the package).

Current limits (MVP)

• Class stats read CSV/TSV label files only (COCO/YOLO parsers are next).
• No remote yet — dset push / dset pull to a self-hosted server is the next milestone.
• No .dsetignore yet; hidden files and dot-directories are always skipped.