dset — version control for datasets

Commit, diff, and roll back millions of files the way you already do with code — and see what actually changed: samples, classes, label distributions. Not bytes.

Pure Python, zero dependencies, single file.

Install

pip install .

(from this folder — or just run python dset/cli.py ... directly)

Quick start

cd my-dataset
dset init
dset add images/ labels.csv
dset commit "June survey, initial labels"

# ... collect more data, fix labels ...

dset add .
dset commit "July imagery + relabeling pass"

dset diff v1 v2

Output:

comparing v1 → v2  (301 → 406 files)

  files       +120   −15   ~1 modified
  images      +120   −15
  labels      ~1 modified
  size        1.5 MB → 2.1 MB  (+568.2 KB)

  class       acacia  19.7% → 38.8%  (+19.1%)
  class       shrub   43.7% → 33.3%  (−10.3%)
  class       bare    36.7% → 27.9%  (−8.8%)

Go back to any version, exactly:

dset checkout v1

Commands

Refs can be a tag (v3), HEAD, or a commit-id prefix.

How it works

• Content-addressed storage. Every file is hashed (SHA-256) and stored once under .dset/objects/, like git. A new version costs only what changed; re-adding identical data costs nothing.
• Commits are manifests — JSON maps of path → hash. Checkout rebuilds the working tree from objects, restoring and deleting as needed.
• Semantic stats are computed at commit time (file counts by type, total size, class distribution parsed from CSV/TSV label files with a label/class/category/target column), so dset diff is instant even on huge datasets.
• Safe by construction. Objects are copied (never hard-linked), written atomically, and stored read-only, so editing a working file can never corrupt history. On btrfs/XFS the copy is a free copy-on-write clone.
• Uncommitted changes block dset checkout unless you pass --force.

Current limits (MVP)

• Class stats read CSV/TSV label files only (COCO/YOLO parsers are next).
• No remote yet — dset push / dset pull to a self-hosted server is the next milestone.
• No .dsetignore yet; hidden files and dot-directories are always skipped.