tether

jj-style version control for heterogeneous datasets.

Status: alpha. See the changelog.

State: vibe coded with Claude Fable 5.1. USE AT YOUR OWN RISK.

tether ties heterogeneous data objects -- files and object-store prefixes (S3, GCS, Azure), Icechunk repositories, Neon Postgres databases, Apache Iceberg tables, Delta Lake tables, Lance datasets, lakeFS repositories, DuckLake catalogs, Dolt databases, and git/jj repositories -- to the commits of an existing git or jj repository.

A tether commit records each object's state and, for systems that support it, creates a durable native reference (an Icechunk tag, a Neon child branch, a git tag, an Iceberg tag, a Lance tag, a lakeFS tag, a Dolt tag) so the exact state can be recovered and branched from later. tether new forks fresh writable branches off any committed state. The VCS supplies history, undo, bookmarks, workspaces, and sharing; tether only implements what is novel: cross-system fan-out, drift detection, and the pin/fork lifecycle.

Think of it as DVC for branchable systems: like DVC it commits small manifests into your git/jj repo, but where DVC only fingerprints files, tether also pins and forks live systems.

Documentation: https://evanlyall.com/tether/ -- user guide (getting started, concepts, pinning, branching and writing, CLI, configuration, backends, writing a backend) plus the generated API and CLI reference. The guide sources live in user_guide/.

Why this exists (prior art)

Every existing tool versions a single layer:

Nothing provides unified version control across files + Icechunk + Postgres + Iceberg with pinning and forking. So tether borrows jj's working-copy model (fingerprint, snapshot on every command, stale-working-copy detection), DVC's manifests-in-VCS layout, and Yggdrasil's observe-then-record shape (a workspace that watches independent systems and records their snapshots, rather than a store that holds the data).

Install

pip install tether-vcs[cli]                 # core + CLI
pip install tether-vcs[cli,icechunk,neon]   # add backends you need
pip install tether-vcs[all]                 # everything

The distribution is tether-vcs (PyPI reserves the bare name); the package you import and the command you run are both tether.

Extras: cli, objectstore (S3/GCS/Azure for file; s3/gcs/azure are aliases), icechunk, neon, iceberg, delta, lance, lakefs, ducklake, dolt, postgres (tether publish / import against Postgres), all. git/jj must be on PATH.

Quickstart

tether has no store of its own. It lives inside a git or jj repository and commits small manifests there; that repository's history, branches, bookmarks, workspaces, undo, and remotes are tether's too. Start in one:

jj git init my-dataset && cd my-dataset      # or: git init my-dataset
tether init                                  # writes tether.toml + .tether/ into the working copy

Then register objects. Nothing is contacted yet; each add writes one .tether/objects/<key>.toml:

tether add zarr/imaging --kind icechunk s3://bucket/imaging.zarr.icechunk --write fork
tether add db/rosebud   --kind neon --project-id prj-123 --database neondb --role runner
tether add raw/plate1   --kind file s3://bucket/raw/plate1/     # Observed unless versioned
tether add raw/manifest --kind file gs://bucket/manifest.csv --file versioned   # Addressable
tether add features     --kind lance s3://bucket/features.lance
tether add events       --kind delta s3://bucket/events        # Addressable (retention-bound)
tether add lake         --kind lakefs --repository analytics --branch main --prefix raw/
tether add warehouse    --kind ducklake ducklake:postgres:dbname=lake --table events
tether add ledger       --kind dolt --host dolt.internal --database ledger --branch main
tether add code         --kind git ../analysis-code --remote origin   # another repo, as an object

tether commit is a VCS commit: it pins each object natively, writes the states into the manifests, and runs jj commit / git commit on them. Every REV below is a jj revset or git revision of the enclosing repository.

tether status                       # fan-out: modified / unpinned / drifted per object
tether commit -m "Baseline imaging + metrics"   # pins, writes manifests, jj/git commit
jj log                              # the dataset's history *is* the repo's history
tether new main                     # jj new main / git checkout main, then fork writable branches off its pins
tether open db/rosebud              # -> postgresql://...tether.ws.ab12cd34...
tether open zarr/imaging -r main    # read-only handle at main's pinned tag
tether diff main @ --content        # what changed inside each object between two revisions, natively
tether log zarr/imaging             # an object's *native* history (snapshots); ids feed `add --at`
tether add old/imaging --kind icechunk s3://bucket/imaging.zarr.icechunk --pick   # start from an older snapshot
tether verify --all-history --deep  # walks every commit of the repo
tether commit -m "..." --dry-run    # every store-writing command plans first; --plan/--from-plan save + apply
tether add scratch/feat s3://bucket/feat.lance --kind lance --pin record   # no tag per commit; fork from the recorded state

History surgery is the VCS's job, and tether follows it. Abandon or squash a dataset commit with jj abandon / jj squash (or git rebase -i) and the pins only that commit named become unreferenced; gc releases them:

jj squash --from <first-try>::<last-try> --into <result>   # drop intermediate dataset commits
tether gc                           # dry run: pins no commit references, orphaned listings (never branches)
tether gc --no-dry-run
tether gc --prune-workspaces --keep-workspace <id> --no-dry-run   # dead workspaces' tether.ws.* branches whose head is pinned; --force-prune for the rest

Multiple people (or agents) work in jj workspaces / git worktrees of the same repository; each gets its own tether.ws.<workspace-id>.* branches in every system, and pushing the repository publishes the dataset history. The tether.toml and .tether/ paths are the only things tether adds to the repo; .tether/workspace.toml is per-checkout and ignored.

Python:

from tether import Repo

repo = Repo.find(".")
repo.status()
repo.commit("baseline")
repo.new("main")
handle = repo.open("zarr/imaging")  # writable IcechunkHandle
session = handle.session  # native icechunk session
ro = repo.open("zarr/imaging", rev="main")  # read-only at the pinned tag

Set TETHER_REV=<vcs rev> and repo.open(key) defaults to a read-only handle at that commit's pinned state -- convenient for downstream, reproducible reads.

Concept map (jj / DVC -> tether)

Repository layout

<dataset-root>/
  tether.toml                 # committed: snapshot.auto (CLI snapshots by default), verify.on_status,
                              #   new.auto_fork (commit re-forks working refs), default policies,
                              #   backend options, credential *references*
  .tether/
    .gitignore                # ignores workspace.toml
    objects/<key>.toml        # committed, one per object: kind, locator, policy, state, pin
    listings/<hash>.jsonl     # committed, content-addressed per-file listings for directory/prefix
                              #   states (so `diff --content` can compare them); pruned by gc
    workspace.toml            # untracked: workspace_id, base manifest hash, working refs, last snapshot

pin_id = blake2b(kind, locator identity, state)[:12]; the native ref is tether.<pin_id>. Identical state yields the same pin, so re-committing an unchanged object is a no-op and identical states dedupe to one pin.

Capability tiers

Backends declare capabilities; commands degrade explicitly by tier.

• Observed (FINGERPRINT): drift detection only. commit records state and marks it recoverable = false. E.g. a local file by mtime/size.
• Addressable (+ ADDRESSABLE): an immutable version id can be read later; open -r <rev> works, nothing to create or GC. E.g. an S3 object in a versioned bucket, a Delta table version.
• Pinnable (+ PIN): can create/delete a durable, GC-proof ref; commit pins, verify/gc apply. E.g. an Icechunk tag.
• Forkable (+ FORK): a writable branch can be created off a pin; new forks, open returns a writable handle. E.g. Icechunk, Neon, Iceberg, Lance, lakeFS, Dolt, git/jj.

Orthogonal flags: CHEAP_FINGERPRINT, RETENTION_BOUND, NEEDS_QUIESCENCE, ATOMIC_REF, DIFF (the backend can describe what changed between two recorded states; see Content diffs), HISTORY (the backend can list its native history for tether log, and accepts a detached base via the locator's at field: tether add --at <id> / --pick).

Compatibility matrix

Command requirements: status/snapshot/verify need FINGERPRINT; commit pins PIN objects (records ADDRESSABLE, records + warns for Observed, or fails with --strict); new forks FORK objects; gc applies to PIN objects; diff --content applies to DIFF objects.

Content diffs

tether diff A B compares manifests: which objects were added, removed, or changed (with pin ids). --content additionally asks each changed object's backend for its native diff, using metadata only (no data is read), run concurrently across objects:

Directory and prefix states are digests, so the file backend hands the engine a per-file listing at commit time; it is stored content-addressed under .tether/listings/ and committed with the manifests (the same trick as DVC's .dir files). Diffs of commits made before listings existed fall back to a count/size summary. gc removes listings no manifest references. Entries are capped at 2000 per object; counts are exact.

Registries and SQL

Data registries keep pointers and metadata in SQL; tether keeps its facts as TOML in the VCS. Three commands bridge them without moving the source of truth (the same shape as git-history, Quilt's package tables, and Kart's SQL working copy; see the guide):

tether export tether.sqlite                 # commits, objects, object_states, refs, ... (also parquet/csv/jsonl)
tether publish --to postgresql://.../rb     # same tables upserted into a Postgres schema; incremental by commit
tether import postgresql://.../rb --query "SELECT 'zarr/' || plate_name AS key, 'icechunk' AS kind, object_uri AS uri FROM data_object" --dry-run

objects.uri joins a catalog's object_uri; objects.commit_id joins a tether_rev provenance column; object_states is the deduplicated set of versions. import plans add / update / remove (--sync) on the manifests from rows with tether's canonical columns -- the mapping is SQL on the registry side -- and never records or pins state; tether commit does. Python: Repo.export() returns an ExportBundle (to_sqlite, to_dir, to_arrow, to_postgres); Repo.import_objects(rows) and plan_import / apply_import mirror the CLI.

Writing a backend

Implement the ObjectBackend protocol (tether.backends.base): declare kind and capabilities, then identity, fingerprint, pin, unpin, list_pins, verify, fork, delete_working_ref, and open. A backend whose tier varies per object (like file) can override effective_capabilities(locator, policy). To support diff --content, add DIFF and implement diff(locator, a, b, *, listings) returning an ObjectDiff; if your state is a digest, also implement listing(locator, state) and the engine will store and hand back the text. Register it with register_backend.

Validate it against the importable conformance suite:

from tether.testing import run_conformance


class MyHarness:
    capabilities = ...  # optional; defaults to backend.capabilities
    backend = MyBackend()

    def new_object(self): ...  # create a fresh system, return a locator
    def mutate(self, locator, working_ref): ...  # change current state


run_conformance(MyHarness())  # runs only the tier-appropriate checks

Caveats

• No cross-system atomicity. A commit records each object's state within a capture window; point-consistency requires quiesced writers (the Neon check helps; tether commit refuses while writers are active unless --force).
• Pin-then-commit ordering. Pins are created before the VCS commit; if the commit never lands, pins leak until gc (which scans VCS history + the current workspace). Re-run tether verify after merging manifests across branches.
• Data history costs storage. Every pin holds bytes in its system until the commit that names it is dropped (jj abandon/squash, git rebase -i) and gc releases it. --pin record skips the native ref on any backend and forks from the recorded state instead, at the price of retention-bound recoverability. commit/new/gc all take --dry-run (and --plan FILE / --from-plan FILE) so store writes can be reviewed before they happen.
• Neon lineage. Pins are child branches, so a working branch can't be deleted while its pins exist, gc only drops leaf pins, and fork-per-new deepens the tree. Neon cannot merge/promote a child into main; use --write track on main for production. protected needs a paid plan; branch quotas vary.
• Neon fingerprint noise/cost. LSN moves without user writes (next_xid is the real change signal); snapshot may wake a suspended compute. Use --no-snapshot to skip.
• Files on unversioned storage are Observed: detectably but not recoverably drifted. Bucket/container versioning promotes single objects to Addressable; prefixes stay Observed (tether does not record per-object version ids). Object-store credentials come from the environment; [backends.file] storage_options in tether.toml passes non-secret options (region, endpoint, account name) to obstore.
• Iceberg catalog variance. S3 Tables disables maintenance when user refs exist; use --pin record there (forks then come from the snapshot id, valid until snapshot expiry).
• Delta has no refs. Versions are recorded, not pinned; VACUUM (7-day default) and log retention (30 days) bound how long open -r works. A recreated table (new metadata id) verifies as drifted, not re-addressed.
• Lance versions are branch-scoped. State is (branch, version); an untouched fork reports its parent's address so no-op commits stay no-ops. Lance refuses to delete a branch a tag references, so a pinned working branch outlives its workspace until gc drops the tag (a re-fork picks name.2).
• lakeFS staging. Uncommitted lakeFS changes are not in any commit; a dirty branch is reported in status and refused at commit.
• DuckLake attachments. A DuckDB-file metadata catalog can be attached only once per process, so tether attaches per operation and a DuckLakeHandle owns its attachment: close() it (or use it as a context manager) before the next status, and don't hold your own writer connection open in the same process. Postgres/SQLite/MySQL catalogs have no such limit. Snapshots are catalog-wide; table only scopes the handle. Put CREATE SECRET statements in [backends.ducklake] init_sql rather than credentials in the locator.
• Dolt working sets. Uncommitted changes on a branch are not in any commit; a dirty branch is reported and refused at commit. The password comes from $DOLT_PASSWORD (configurable via [backends.dolt] password_env), never the manifest. Requires a Dolt with the dirty column on dolt_branches for dirty detection (older servers report clean).
• Secrets are never stored in manifests -- only credential references (env var / secret names).

Performance notes

tether is an orchestrator; its cost is the network round-trips it fans out, not Python.

• Fan-out uses threads, not asyncio. Every backend SDK tether wraps (obstore, psycopg, pyiceberg, deltalake, lance, lakefs, duckdb, pymysql, git) is synchronous, the per-object Python work is microseconds, and a thread pool gives the same wall-clock concurrency without forcing an async protocol on backend authors. snapshot, verify, and the fork step of new all run concurrently.
• History walks are batched. gc and verify --all-history stream every manifest at every commit through one git cat-file --batch process (jj repos are git-backed, so the same plumbing applies), caching trees and blobs by object id and verifying each distinct (system, state, pin) once. On a repo with 200 commits and 5 objects this is ~20-40 ms end to end versus 6-12 s with one process per read.
• Fingerprints are metadata-only. Local trees use os.scandir (~4 us/file); a remote object is one HEAD; a prefix is one paged LIST.
• Content diffs are metadata-only too. Each backend reads manifests, logs, or catalog tables -- never data -- so diff --content costs about one fingerprint per changed object (two manifest reads for Icechunk/Lance, one log walk for Delta, one catalog query for DuckLake, one prolly-tree diff for Dolt). Two stored file listings diff in pure Python at ~50 ms per 100k paths.

Non-goals

• Being in the data path (tether hands back native handles and steps aside).
• Reimplementing history/undo/branching/sharing (that is the VCS's job).
• Cross-system transactions or a query layer.

Development

uv sync --all-extras
uv run ruff format --check . && uv run ruff check .
uv run ty check
uv run pytest

The publish / import tests start an ephemeral PostgreSQL cluster through pytest-postgresql; they need pg_ctl on PATH or a Homebrew / Debian install (brew install postgresql@16; GitHub's Ubuntu runners ship it) and skip otherwise.

License

Apache-2.0. See LICENSE.