tether

Version control (fingerprints, pins, & forks) for heterogeneous datasets.

A dataset rarely lives in one place: an Icechunk repository, a directory of raw files, a Postgres database, an Iceberg table, and the code that produced them. tether tracks all of them from one git or jj repository. Each commit records every object's exact state -- and, where the system allows it, pins that state with a native ref so it stays readable, and forks a writable branch off it when you want to change things without touching main.

Think of it as DVC for branchable systems: like DVC it commits small manifests to your repository, but where DVC only fingerprints files, tether also pins and forks live systems.

Status: pre-release (0.1.0 betas). See the changelog and ROADMAP.md for what is tested against real services and what is still experimental. Vibe coded with Claude Fable 5.1. USE AT YOUR OWN RISK.

Install

pip install "tether-vcs[cli]"                 # core + CLI
pip install "tether-vcs[cli,icechunk,lance]"  # add the backends you use
pip install "tether-vcs[all]"                 # everything

The distribution is tether-vcs; the package you import and the command you run are both tether. Python 3.11 or newer; git and/or jj on PATH. Extras: objectstore (S3/GCS/Azure for file), icechunk, neon, iceberg, delta, lance, lakefs, ducklake, dolt, postgres, all.

Quickstart

tether lives inside a git or jj repository. Nothing below needs a cloud account: point it at an Icechunk repository and a directory you have on disk.

jj git init my-dataset && cd my-dataset      # or: git init my-dataset
tether init
tether add zarr/imaging --kind icechunk ../data/imaging.icechunk
tether add raw/plate1   --kind file     ../data/raw/plate1/

tether status                       # new / clean / modified per object
tether commit -m "Baseline"         # pin the Icechunk snapshot; record the directory; jj/git commit

  pinned zarr/imaging -> tether.4e9503fc.73e341e966c83d38
  recorded raw/plate1 (not recoverable)
committed b55326ca55a0

That commit id names the exact state of every object, forever: tether open zarr/imaging --rev b55326ca55a0 opens the pinned snapshot read-only, and TETHER_REV=b55326ca55a0 python report.py makes every repo.open() in a script do the same.

To change data without touching main, work on a bookmark:

tether new -b relabel               # one branch per system, created on first write
python relabel.py                   # writes through repo.open("zarr/imaging")
tether commit -m "Relabel plate1"   # pin the branch head; move the bookmark
tether promote                      # fast-forward main in every system, then move the main bookmark
tether verify --all-history         # every pin any commit ever named still resolves

from tether import Repo

repo = Repo.find(".")
h = repo.open("zarr/imaging")  # writable IcechunkHandle on this bookmark's branch
ro = repo.open("zarr/imaging", rev="main")  # read-only at main's pin

Every command that writes to a store takes --dry-run (and --plan FILE / --from-plan FILE) so the writes can be reviewed first. The Getting Started guide runs this walkthrough with full output; the user guide takes it from there.

How it works

• A dataset is a git/jj repository with one small manifest per object. History, branching, workspaces, and sharing are the VCS's.
• Commits hold pins. Each dataset commit records, per object, an exact state and -- where the system allows -- a pin: a native, GC-proof ref (an Icechunk tag, a protected Neon branch, a git tag) that holds that state. tether open KEY --rev C reads it back.
• Bookmarks hold branches. The trunk bookmark (main) stands for every object's upstream branch; working on it writes there. Any other bookmark stands for one branch per system, tether.ws.<dataset>.<bookmark>, forked from the pins where it started and created on the first write.
• commit, promote, pull are the three verbs: pin the bookmark's branch heads and move the bookmark; fast-forward (or merge) each system's upstream branch to the bookmark's and move main; read what upstream has now and commit it.
• tether is never in the data path. open returns the system's native handle -- an Icechunk session, a Postgres URL, a DeltaTable -- and steps aside.

What the VCS cannot see -- what tether did to the stores -- is in tether's own operation log (tether ops, tether undo, tether repair).

What each backend can do

Every backend is fingerprinted; the rest depends on what the system offers.

Recover: can a committed state be opened again later? ✅ held by a pin; 🟡 re-openable by version id for as long as the system keeps that version; ❌ tether can only tell you it changed. Pin: a durable native ref created at commit time. Fork: a writable branch off a pin for tether new. Promote / Merge: fast-forward, or three-way merge, the upstream branch to the fork. Diff: tether diff --content describes what changed inside the object. History: tether log lists the object's own snapshots. Experimental backends have run against a fake of the service, not the service itself. The full matrix -- state fields, flags, per-backend caveats -- is in the backends guide.

jj or tether?

A tether command exists where an operation has two halves -- one in the VCS, one in the stores -- that must happen together: commit, new, pull, promote, restore, abandon, drop, forget-workspace, and undo / repair for what tether itself did. Everything that only touches files and history -- describe, squash, rebase, push -- is the VCS's, and tether notices what it needs to: a bookmark deleted, renamed, or moved by hand shows up in status with what to do. The concepts guide has the table.

Non-goals

• Sitting in the data path: open hands back the system's native handle and steps aside.
• Running between commands: no daemon, no watcher; states are compared when a command asks.
• Reimplementing history, branching, or sharing: that is the VCS's job. tether ops / undo / repair cover only what tether did to the stores.
• Cross-system transactions or a query layer.

Documentation

https://evanlyall.com/tether/ -- getting started, concepts, worked examples, CLI guide, backends, and the generated API and CLI reference.

Why this exists

Every existing tool versions a single layer:

Nothing provides unified version control across files + Icechunk + Postgres + Iceberg with pinning and forking. So tether borrows jj's working-copy and operation-log ideas (new REV, stale-working-copy detection, undo), DVC's manifests-in-VCS layout, and Yggdrasil's observe-then-record shape: a workspace that watches independent systems and records their snapshots, rather than a store that holds the data.

Development

uv sync --all-extras
uv run ruff format --check . && uv run ruff check .
uv run ty check
uv run pytest

The publish / import tests start an ephemeral PostgreSQL cluster through pytest-postgresql; they need pg_ctl on PATH or a Homebrew / Debian install (brew install postgresql@16; GitHub's Ubuntu runners ship it) and skip otherwise. The use-cases story (tests/test_use_cases.py) regenerates the guide's command output with TETHER_UPDATE_DOCS=1.

License

Apache-2.0. See LICENSE.