tether

jj-style version control for heterogeneous datasets.

Status: alpha. See the changelog.

State: vibe coded with Claude Fable 5.1. USE AT YOUR OWN RISK.

tether ties heterogeneous data objects -- files and object-store prefixes (S3, GCS, Azure), Icechunk repositories, Neon Postgres databases, Apache Iceberg tables, Delta Lake tables, Lance datasets, lakeFS repositories, DuckLake catalogs, Dolt databases, and git/jj repositories -- to the commits of an existing git or jj repository.

A tether commit records each object's state and, for systems that support it, creates a durable native reference (an Icechunk tag, a Neon child branch, a git tag, an Iceberg tag, a Lance tag, a lakeFS tag, a Dolt tag) so the exact state can be recovered and branched from later. tether new forks fresh writable branches off any committed state. The VCS supplies history, undo, bookmarks, workspaces, and sharing; tether only implements what is novel: cross-system fan-out, drift detection, and the pin/fork lifecycle.

Think of it as DVC for branchable systems: like DVC it commits small manifests into your git/jj repo, but where DVC only fingerprints files, tether also pins and forks live systems.

Documentation: https://evanlyall.com/tether/ -- user guide (getting started, concepts, pinning, branching and writing, CLI, configuration, backends, writing a backend) plus the generated API and CLI reference. The guide sources live in user_guide/.

Why this exists (prior art)

Every existing tool versions a single layer:

Nothing provides unified version control across files + Icechunk + Postgres + Iceberg with pinning and forking. So tether borrows jj's working-copy model (fingerprint, snapshot on every command, stale-working-copy detection), DVC's manifests-in-VCS layout, and Yggdrasil's observe-then-record shape (a workspace that watches independent systems and records their snapshots, rather than a store that holds the data).

Install

pip install tether-vcs[cli]                 # core + CLI
pip install tether-vcs[cli,icechunk,neon]   # add backends you need
pip install tether-vcs[all]                 # everything

The distribution is tether-vcs (PyPI reserves the bare name); the package you import and the command you run are both tether.

Extras: cli, objectstore (S3/GCS/Azure for file; s3/gcs/azure are aliases), icechunk, neon, iceberg, delta, lance, lakefs, ducklake, dolt, all. git/jj must be on PATH.

Quickstart

tether init
tether add zarr/imaging --kind icechunk s3://bucket/imaging.zarr.icechunk --write fork
tether add db/rosebud   --kind neon --project-id prj-123 --database neondb --role runner
tether add raw/plate1   --kind file s3://bucket/raw/plate1/     # Observed unless versioned
tether add raw/manifest --kind file gs://bucket/manifest.csv --file versioned   # Addressable
tether add features     --kind lance s3://bucket/features.lance
tether add events       --kind delta s3://bucket/events        # Addressable (retention-bound)
tether add lake         --kind lakefs --repository analytics --branch main --prefix raw/
tether add warehouse    --kind ducklake ducklake:postgres:dbname=lake --table events
tether add ledger       --kind dolt --host dolt.internal --database ledger --branch main
tether add code         --kind git . --remote origin

tether status                       # fan-out: modified / unpinned / drifted per object
tether commit -m "Baseline imaging + metrics"   # pins, writes manifests, jj/git commit
tether new main                     # fork fresh writable branches off main's pins
tether open db/rosebud              # -> postgresql://...tether.ws.ab12cd34...
tether open zarr/imaging -r main    # read-only handle at main's pinned tag
tether diff main @ --content         # what changed inside each object, natively
tether log zarr/imaging              # native snapshot history; ids feed `add --at`
tether add old/imaging --kind icechunk s3://bucket/imaging.zarr.icechunk --pick   # start from an older snapshot
tether verify --all-history --deep
tether gc --dry-run

Python:

from tether import Repo

repo = Repo.find(".")
repo.status()
repo.commit("baseline")
repo.new("main")
handle = repo.open("zarr/imaging")  # writable IcechunkHandle
session = handle.session  # native icechunk session
ro = repo.open("zarr/imaging", rev="main")  # read-only at the pinned tag

Set TETHER_REV=<vcs rev> and repo.open(key) defaults to a read-only handle at that commit's pinned state -- convenient for downstream, reproducible reads.

Concept map (jj / DVC -> tether)

Repository layout

<dataset-root>/
  tether.toml                 # committed: snapshot.auto (CLI snapshots by default), verify.on_status,
                              #   new.auto_fork (commit re-forks working refs), default policies,
                              #   backend options, credential *references*
  .tether/
    .gitignore                # ignores workspace.toml
    objects/<key>.toml        # committed, one per object: kind, locator, policy, state, pin
    listings/<hash>.jsonl     # committed, content-addressed per-file listings for directory/prefix
                              #   states (so `diff --content` can compare them); pruned by gc
    workspace.toml            # untracked: workspace_id, base manifest hash, working refs, last snapshot

pin_id = blake2b(kind, locator identity, state)[:12]; the native ref is tether.<pin_id>. Identical state yields the same pin, so re-committing an unchanged object is a no-op and identical states dedupe to one pin.

Capability tiers

Backends declare capabilities; commands degrade explicitly by tier.

• Observed (FINGERPRINT): drift detection only. commit records state and marks it recoverable = false. E.g. a local file by mtime/size.
• Addressable (+ ADDRESSABLE): an immutable version id can be read later; open -r <rev> works, nothing to create or GC. E.g. an S3 object in a versioned bucket, a Delta table version.
• Pinnable (+ PIN): can create/delete a durable, GC-proof ref; commit pins, verify/gc apply. E.g. an Icechunk tag.
• Forkable (+ FORK): a writable branch can be created off a pin; new forks, open returns a writable handle. E.g. Icechunk, Neon, Iceberg, Lance, lakeFS, Dolt, git/jj.

Orthogonal flags: CHEAP_FINGERPRINT, RETENTION_BOUND, NEEDS_QUIESCENCE, ATOMIC_REF, DIFF (the backend can describe what changed between two recorded states; see Content diffs), HISTORY (the backend can list its native history for tether log, and accepts a detached base via the locator's at field: tether add --at <id> / --pick).

Compatibility matrix

Command requirements: status/snapshot/verify need FINGERPRINT; commit pins PIN objects (records ADDRESSABLE, records + warns for Observed, or fails with --strict); new forks FORK objects; gc applies to PIN objects; diff --content applies to DIFF objects.

Content diffs

tether diff A B compares manifests: which objects were added, removed, or changed (with pin ids). --content additionally asks each changed object's backend for its native diff, using metadata only (no data is read), run concurrently across objects:

Directory and prefix states are digests, so the file backend hands the engine a per-file listing at commit time; it is stored content-addressed under .tether/listings/ and committed with the manifests (the same trick as DVC's .dir files). Diffs of commits made before listings existed fall back to a count/size summary. gc removes listings no manifest references. Entries are capped at 2000 per object; counts are exact.

Writing a backend

Implement the ObjectBackend protocol (tether.backends.base): declare kind and capabilities, then identity, fingerprint, pin, unpin, list_pins, verify, fork, delete_working_ref, and open. A backend whose tier varies per object (like file) can override effective_capabilities(locator, policy). To support diff --content, add DIFF and implement diff(locator, a, b, *, listings) returning an ObjectDiff; if your state is a digest, also implement listing(locator, state) and the engine will store and hand back the text. Register it with register_backend.

Validate it against the importable conformance suite:

from tether.testing import run_conformance


class MyHarness:
    capabilities = ...  # optional; defaults to backend.capabilities
    backend = MyBackend()

    def new_object(self): ...  # create a fresh system, return a locator
    def mutate(self, locator, working_ref): ...  # change current state


run_conformance(MyHarness())  # runs only the tier-appropriate checks

Caveats

• No cross-system atomicity. A commit records each object's state within a capture window; point-consistency requires quiesced writers (the Neon check helps; tether commit refuses while writers are active unless --force).
• Pin-then-commit ordering. Pins are created before the VCS commit; if the commit never lands, pins leak until gc (which scans VCS history + the current workspace). Re-run tether verify after merging manifests across branches.
• Neon lineage. Pins are child branches, so a working branch can't be deleted while its pins exist, gc only drops leaf pins, and fork-per-new deepens the tree. Neon cannot merge/promote a child into main; use --write track on main for production. protected needs a paid plan; branch quotas vary.
• Neon fingerprint noise/cost. LSN moves without user writes (next_xid is the real change signal); snapshot may wake a suspended compute. Use --no-snapshot to skip.
• Files on unversioned storage are Observed: detectably but not recoverably drifted. Bucket/container versioning promotes single objects to Addressable; prefixes stay Observed (tether does not record per-object version ids). Object-store credentials come from the environment; [backends.file] storage_options in tether.toml passes non-secret options (region, endpoint, account name) to obstore.
• Iceberg catalog variance. S3 Tables disables maintenance when user refs exist; use --pin record there.
• Delta has no refs. Versions are recorded, not pinned; VACUUM (7-day default) and log retention (30 days) bound how long open -r works. A recreated table (new metadata id) verifies as drifted, not re-addressed.
• Lance versions are branch-scoped. State is (branch, version); an untouched fork reports its parent's address so no-op commits stay no-ops. Lance refuses to delete a branch a tag references, so a pinned working branch outlives its workspace until gc drops the tag (a re-fork picks name.2).
• lakeFS staging. Uncommitted lakeFS changes are not in any commit; a dirty branch is reported in status and refused at commit.
• DuckLake attachments. A DuckDB-file metadata catalog can be attached only once per process, so tether attaches per operation and a DuckLakeHandle owns its attachment: close() it (or use it as a context manager) before the next status, and don't hold your own writer connection open in the same process. Postgres/SQLite/MySQL catalogs have no such limit. Snapshots are catalog-wide; table only scopes the handle. Put CREATE SECRET statements in [backends.ducklake] init_sql rather than credentials in the locator.
• Dolt working sets. Uncommitted changes on a branch are not in any commit; a dirty branch is reported and refused at commit. The password comes from $DOLT_PASSWORD (configurable via [backends.dolt] password_env), never the manifest. Requires a Dolt with the dirty column on dolt_branches for dirty detection (older servers report clean).
• Secrets are never stored in manifests -- only credential references (env var / secret names).

Performance notes

tether is an orchestrator; its cost is the network round-trips it fans out, not Python.

• Fan-out uses threads, not asyncio. Every backend SDK tether wraps (obstore, psycopg, pyiceberg, deltalake, lance, lakefs, duckdb, pymysql, git) is synchronous, the per-object Python work is microseconds, and a thread pool gives the same wall-clock concurrency without forcing an async protocol on backend authors. snapshot, verify, and the fork step of new all run concurrently.
• History walks are batched. gc and verify --all-history stream every manifest at every commit through one git cat-file --batch process (jj repos are git-backed, so the same plumbing applies), caching trees and blobs by object id and verifying each distinct (system, state, pin) once. On a repo with 200 commits and 5 objects this is ~20-40 ms end to end versus 6-12 s with one process per read.
• Fingerprints are metadata-only. Local trees use os.scandir (~4 us/file); a remote object is one HEAD; a prefix is one paged LIST.
• Content diffs are metadata-only too. Each backend reads manifests, logs, or catalog tables -- never data -- so diff --content costs about one fingerprint per changed object (two manifest reads for Icechunk/Lance, one log walk for Delta, one catalog query for DuckLake, one prolly-tree diff for Dolt). Two stored file listings diff in pure Python at ~50 ms per 100k paths.

Non-goals

• Being in the data path (tether hands back native handles and steps aside).
• Reimplementing history/undo/branching/sharing (that is the VCS's job).
• Cross-system transactions or a query layer.

Development

uv sync --all-extras
uv run ruff format --check . && uv run ruff check .
uv run ty check
uv run pytest

License

Apache-2.0. See LICENSE.