framepin 0.3.0

A lockfile for your video/sequence-ML datasets and experiments — reproduce any run without copying a single frame.

framepin

A lockfile for your video/sequence-ML datasets and experiments — reproduce any run without copying a single frame.

framepin pins the exact data a training run saw, links it to the run's params/metrics/commit, and tells you — when a metric moves — whether it was the code or the data. Zero dependencies, no server, no data copies. Just JSON that commits cleanly to git.

pip install framepin   # zero dependencies — pure standard library

Status: v0.1 alpha. Core (snapshot / diff / track / regress) works and is tested. APIs may shift before 1.0.

---

The problem

If you train models on video or sequence data, this has happened to you:

• A run from three weeks ago had a better val_loss. Which dataset version was it? You re-sampled frames and re-labeled twice since. Gone.
• val_loss got worse after a change. Was it your code… or did someone swap out 30% of the clips? Nobody can say quickly.
• wandb/mlflow track the run but treat the dataset as an opaque string. dvc versions the data but is heavy and lives apart from your metrics. Neither answers "is this run reproducible from these exact bytes?"

Video datasets are big, churn constantly, and get reorganized — so a directory reshuffle looks like "half my data changed" in a naive diff.

What framepin does

import framepin

with framepin.track(name="baseline", params={"lr": 3e-4}) as run:
    run.use_dataset("data/clips")      # content-hashes + pins this exact version
    model = train(...)
    run.log_metric("val_loss", 0.21)   # pinned to that version forever

Then, three weeks later:

$ framepin regress <old_run> <new_run> -m val_loss
regress 02189c0fbd14 -> ab09cc84a35a
  val_loss: 0.21 -> 0.28  (Δ +0.07)

  ⚠ DATA CHANGED between these runs — the dataset version differs.
    A metric move here cannot be attributed to code alone.

That last line is the whole point. framepin separates "the code regressed" from "the data changed under you."

Before / after

Before — the dataset is a string you hope is still true:

run.log({"dataset": "data/clips (v3?)", "val_loss": 0.21})
# 3 weeks later: what was v3? which frames? was it re-labeled? 🤷

After — the dataset is a content hash you can reproduce and diff:

with framepin.track(name="baseline") as run:
    run.use_dataset("data/clips")   # -> version c29aa729c669 (Merkle root of every byte)
    run.log_metric("val_loss", 0.21)
# later: `framepin show <run>` -> exact version; `framepin diff v1 v2` -> what changed

60-second quickstart

cd your-project
framepin init                     # creates .framepin/  (commit it to git)

framepin snapshot data/clips      # -> snapshot c29aa729c669 (N files, no copies)
# ... change / relabel / resample your data ...
framepin snapshot data/clips      # -> a new version id

framepin diff c29aa729c669 <new>  # added / removed / modified / MOVED
framepin log                       # list every pinned version, newest first
framepin gc --apply                # prune versions no run references (dry-run without --apply)

Large datasets: add --jobs 16 to snapshot/verify for parallel hashing. Scripts, CI and AI agents: add --json to snapshot/verify/log for machine-readable output.

Datasets defined by path-list files (train.txt of absolute paths)

Many teams don't train on "a directory" — they train on txt manifests: one absolute path per line (500k clips, several lists concatenated per experiment). framepin pins that whole construct — the list files and the bytes they point at:

framepin snapshot --from-list train_urban.txt train_highway.txt
# -> one version id covering: both txt files + every referenced clip
#    re-encode one clip, or edit one line  -> different version id
#    a listed path that no longer exists   -> "⚠ recorded as missing"

Repeated paths across lists are deduped. For 100k+ files, content hashes are cached (.framepin/hashcache.json) so later snapshots only re-read files whose size/mtime changed; --fast skips content reads entirely (size+mtime fingerprint) for routine checks — take a periodic full snapshot as your anchor of record.

man = framepin.snapshot_from_lists(["train_urban.txt", "train_highway.txt"])
with framepin.track(name="exp-8") as run:
    run.use_dataset(man)               # pins the exact list+content version

Track experiments from Python:

import framepin

with framepin.track(name="exp-7", params={"lr": 3e-4, "aug": "mixup"}) as run:
    run.use_dataset("data/clips")
    for epoch in range(epochs):
        ...
    run.log_metric("val_loss", best_val)
    run.log_metric("map50", 0.63)

PyTorch: nothing about your Dataset/DataLoader changes — see examples/pytorch_integration.py for the 3-line pattern in a real train loop.

Inspect lineage and compare:

framepin runs                     # every run, its metrics, and its data version
framepin show <run>               # full lineage: run -> dataset version -> files
framepin regress <a> <b> -m map50 # metric delta + "was it code or data?"

CI gate: fail the build when the data drifted

Pin a baseline version once, then let CI refuse changes that silently alter the dataset:

framepin verify data/clips --against c29aa729c669     # exit 0 = intact
# ✗ DATASET DRIFT vs pinned c29aa729c669:  +1 -0 ~2 →0   -> exit 3, build fails
framepin verify --from-list train_a.txt train_b.txt --against <version>

Better: don't hardcode the version in CI config. framepin pin writes it to a one-line pinfile you commit next to your code, and --against-file reads it back — accepting an intentional data change becomes a normal reviewed commit, not a CI-config edit:

framepin pin data/clips                                # snapshot + write framepin.pin
git add framepin.pin .framepin && git commit -m "pin dataset baseline"
# in CI:
framepin verify data/clips --against-file framepin.pin
# data changed on purpose? re-pin and commit — the diff shows up in review:
framepin pin data/clips

pin also takes --from-list, --split, --file <path> (default framepin.pin), and --version <id> to pin an already-stored version without re-hashing. Ready-made GitHub Actions workflows — including one that opens a baseline-update PR automatically when the data drifts — are in examples/ci/.

Expected churn (say, labels get re-exported every week) doesn't have to break the build — allow-list it:

framepin verify data/clips --against c29aa729c669 --allow 'labels/*'
# ✓ allowed drift vs pinned c29aa729c669:  +0 -0 ~3 →0  (all changed paths match --allow)

--allow is repeatable; globs match manifest-relative paths (* matches across /). The gate still fails if any changed path falls outside the allow-list — allowed changes are just summarized instead of listed. With --json you get "gate": "pass"|"fail" plus an "allowed" count.

Stratified datasets: per-split version ids

One dataset, several splits, one manifest — give each split its own version id so drift reports tell you which split moved:

framepin snapshot data/clips --split train='train/*' --split val='val/*'
# snapshot a208968bd4cf  (500 files, 81236 bytes)
#   split train  2519d0cc5791  (400 files, 64988 bytes)
#   split val    985005ef8b59  (100 files, 16248 bytes)

Splits are pure metadata — the top-level version id is unchanged whether or not you record them. On verify drift, the report adds splits changed: train / splits unchanged: val (and "splits_changed" under --json), so a re-labeled val set never sends you digging through the train half of the diff.

Already on W&B or MLflow? Export the lineage

framepin doesn't compete with your tracker — it feeds it. One call tags the active W&B / MLflow run with the exact pinned dataset version id(s), so the data version shows up right next to your metrics in their UI:

with framepin.track(name="baseline") as run:
    run.use_dataset("data/clips")
    framepin.integrations.to_mlflow(run)   # tags: framepin.run_id, framepin.dataset.*
    # or: framepin.integrations.to_wandb(run)

Neither library is a dependency — the imports are lazy, framepin stays at zero deps.

Why not W&B / MLflow / DVC?

	framepin	W&B / MLflow	DVC
Pins run ↔ exact dataset version	✅ first-class	⚠️ manual string/artifact	⚠️ separate from metrics
"Code vs data" regression answer	✅ regress	❌	❌
Detects moved/renamed files (not add+remove)	✅	—	partial
Copies your data	❌ never (hashes only)	⚠️ artifacts	✅ cache/remote
Server / account required	❌	⚠️ (or self-host)	❌
Runtime dependencies	0	many	several
Git-friendly plain-JSON store	✅	❌	✅ (pointers)

framepin is deliberately small. It is not a full experiment platform — it does the one thing those tools under-serve: making the dataset version a first-class, reproducible, diffable part of every run. Use it alongside W&B if you like; framepin just owns the data-lineage question.

How it works

• Snapshot walks a directory, streams a SHA-256 over each file, and records a deterministic manifest keyed by a Merkle root of (path, content-hash) pairs. Same bytes → same version id, regardless of walk order or timestamps. Your data is never moved or copied — the manifest is a few KB of JSON.
• Diff classifies changes into added / removed / modified / moved, so a reorg of identical clips reads as moves, not churn.
• Track records a run's params, metrics, git commit, and the dataset version(s) it consumed, into .framepin/runs/<id>.json.
• Regress compares two runs' metrics and their pinned dataset versions.

Everything lives under .framepin/ as plain, sorted JSON. Commit it.

Install from source

git clone https://github.com/boogy-ro/framepin
cd framepin
pip install -e .          # or just add the repo to PYTHONPATH — no deps
python -m unittest discover -s tests

Requires Python ≥ 3.9. No third-party packages.

See the whole workflow in ~2 seconds (builds a throwaway dataset, simulates a re-label + re-encode, then shows the code-vs-data verdict):

python3 examples/quickstart_demo.py

Roadmap

• remote manifest registry for teams
• richer CI gate (verify --allow, pinfiles and baseline auto-update PRs ship today)

Feedback and issues very welcome — the niche (video/sequence ML data lineage) is exactly where this should earn its keep or die. Tell me where it falls short.

Using this on a team? Comment on the team-features issue (CI gate / shared registry / audit reports) — it decides what gets built next.

FAQ

How do I version an ML dataset without copying it? framepin snapshot data/clips content-hashes every file into an immutable version id (a Merkle root). Your data never moves; the snapshot is a few KB of JSON you commit to git.

How do I know which dataset version an old training run used? If the run was wrapped in framepin.track(...) with run.use_dataset(...), framepin show <run> prints its exact pinned dataset version and file list.

My metric regressed — was it my code or my data? framepin regress <old_run> <new_run> -m val_loss compares the two runs' metrics and their pinned dataset versions, and prints either ⚠ DATA CHANGED or ✓ same dataset version so you know where to look.

My dataset is a txt file of absolute paths (500k clips), not a directory. framepin snapshot --from-list train_a.txt train_b.txt pins the list files and every file they reference, deduped across lists. Dead paths are recorded as missing. Use --fast for size+mtime fingerprints on huge datasets.

Does framepin replace W&B, MLflow, or DVC? No. It runs alongside them and owns one thing they under-serve: the dataset-version ↔ run link and the code-vs-data question. No server, no account, zero dependencies.

How do I detect that files were renamed/reorganized rather than changed? framepin diff v1 v2 pairs identical-content files across paths and reports them as MOVED instead of added+removed.

License

MIT — see LICENSE.