tether

Version control (fingerprints, pins, & forks) for heterogeneous datasets.

Status: alpha. See the changelog.

State: vibe coded with Claude Fable 5.1. USE AT YOUR OWN RISK.

Think of tether as DVC for branchable systems: like DVC it commits small manifests into your git/jj repo, but where DVC only fingerprints files, tether also pins and forks live systems. Every backend is fingerprinted; what else tether can do with each depends on what the system offers:

Recover: can a committed state be opened again later? ✅ held by a pin; 🟡 reopenable by version id for as long as the system keeps that version (object versioning, Delta/DuckLake snapshots) -- tether records it but has no ref of its own to hold it; ❌ tether can only tell you it changed. Pin: a durable native ref (tag, protected branch) tether creates at commit time. Fork: a writable branch off a pin for tether new. Promote: fast-forward the base branch to the fork; Merge: a native three-way merge when it is not a fast-forward. Diff: tether diff --content describes what changed inside the object. History: tether log lists the object's own snapshots/versions/commits. The full matrix -- state fields, flags, per-backend caveats -- is in the backends guide.

Documentation: https://evanlyall.com/tether/ -- getting started, concepts, pinning, branching and writing, reclaiming storage, use cases, CLI, configuration, backends, registries and SQL, writing a backend, caveats and performance, plus the generated API and CLI reference.

The model

A dataset is a git/jj repository with one small manifest per object, and its bookmarks and the stores' branches are one shape:

• Commits hold references. Each dataset commit records, per object, an exact state and -- where the system allows it -- a pin: a native, GC-proof ref (an Icechunk tag, a protected Neon branch, a git tag) that holds that state. A commit is a complete, reproducible position for every object at once; tether open KEY --rev C reads it.
• Bookmarks hold branches. The trunk bookmark (main) stands for every object's upstream branch: working on it writes there. Any other bookmark stands for one branch per system, named after it (tether.ws.<dataset>.feature), forked from the pins of the commit it started at -- decided by tether new -b feature, created on the first write. A working copy on no bookmark is read-only.

commit pins the heads of your bookmark's branches and moves the bookmark to the new commit, the way a git branch follows its commits. promote lands a bookmark on the trunk: each system's upstream branch fast-forwards (or merges) to the bookmark's branch, then main moves to the bookmark's commit. pull is the fetch: on main it reads every object's upstream branch (and, for systems without branches, the object itself) and commits what moved.

Everything history-shaped (commits, bookmarks, workspaces, sharing, undo of the manifests) is the VCS's. What the VCS cannot see -- what tether did to the stores -- is in tether's own operation log (tether ops, tether undo).

Why this exists (prior art)

Every existing tool versions a single layer:

Nothing provides unified version control across files + Icechunk + Postgres + Iceberg with pinning and forking. So tether borrows jj's working-copy and operation-log ideas (new REV, stale-working-copy detection, undo), DVC's manifests-in-VCS layout, and Yggdrasil's observe-then-record shape (a workspace that watches independent systems and records their snapshots, rather than a store that holds the data).

Install

pip install tether-vcs[cli]                 # core + CLI
pip install tether-vcs[cli,icechunk,neon]   # add backends you need
pip install tether-vcs[all]                 # everything

Extras: cli, objectstore (S3/GCS/Azure for file; s3/gcs/azure are aliases), icechunk, neon, iceberg, delta, lance, lakefs, ducklake, dolt, postgres (tether publish / import against Postgres), all. git/jj must be on PATH.

Quickstart

tether lives inside a git or jj repository and commits small manifests there; that repository's history, branches, and workspaces are the dataset's too. Nothing is contacted until you ask for a state.

jj git init my-dataset && cd my-dataset      # or: git init my-dataset
tether init
tether add zarr/imaging --kind icechunk s3://bucket/imaging.icechunk
tether add db/metrics   --kind neon --project-id prj-123 --database neondb --role runner
tether add raw/plate1   --kind file s3://bucket/raw/plate1/

tether status                       # clean / modified / drifted per object (local; --snapshot re-fingerprints)
tether commit -m "Baseline"         # on main: pin each object's upstream branch, write the manifests, jj/git commit
tether new -b relabel               # a bookmark: one branch per system, named after it, forked on first write
tether open db/metrics              # postgresql://... on the relabel branch
tether commit -m "Relabel plate1"   # pins the branches' heads and moves the bookmark
tether promote                      # land it: each system's main fast-forwards (or merges), then main moves to the commit
tether verify --all-history         # every pin any commit ever named still resolves

from tether import Repo

repo = Repo.find(".")
h = repo.open("zarr/imaging")  # writable IcechunkHandle on this bookmark's branch
ro = repo.open(
    "zarr/imaging", rev="main"
)  # read-only at main's pin; TETHER_REV=<rev> makes this the default

Every command that writes to a store takes --dry-run (and --plan FILE / --from-plan FILE) so the writes can be reviewed first. The getting started guide walks through this with output; the CLI guide has every command.

jj or tether?

A tether command exists where an operation has two halves -- one in the VCS, one in the stores -- that must happen together: commit (pin, commit, move the bookmark), new (create or join a bookmark, then its branches), pull, promote, restore, abandon, forget-workspace, and undo / repair for what tether itself did. Everything that only touches files and history -- describe, squash, rebase, push, jj undo of a non-tether operation -- is the VCS's, and tether notices what it needs to: a bookmark deleted, renamed, or moved by hand shows up in status with what to do, as does a vanished dataset commit. The concepts guide has the table.

Non-goals

• Sitting in the data path: open hands back the system's native handle and steps aside.
• Running between commands: no daemon, no watcher; states are compared when a command asks.
• Reimplementing history, branching, or sharing: that is the VCS's job. tether ops / undo / repair cover only what tether did to the stores, which the VCS cannot see.
• Cross-system transactions or a query layer.

Development

uv sync --all-extras
uv run ruff format --check . && uv run ruff check .
uv run ty check
uv run pytest

The publish / import tests start an ephemeral PostgreSQL cluster through pytest-postgresql; they need pg_ctl on PATH or a Homebrew / Debian install (brew install postgresql@16; GitHub's Ubuntu runners ship it) and skip otherwise.

License

Apache-2.0. See LICENSE.