casket

A simple content-addressed store for large files, kept out of git as plain files and linked into projects with symlinks.

Goals

• No service to run: stores and remotes are plain directories, local or reached over ssh.
• No location tracking: remotes are asked what they have when needed, nothing records where objects live.
• Clear control over storage: objects are plain read-only files, links show the key they point to, the config is a text file.

Model

A central store is the source of truth. Its tree of symlinks is the catalog, organized by hand. Projects consume it: they hold symlinks to assets and fetch the objects behind them into their own .casket.

casket init                                  # create .casket here
casket remote add central /mnt/nas/assets/.casket
casket put rock.png draft.png                # store files, replacing them by links
casket publish rock.png textures/            # add to central as textures/rock.png
casket publish assets/ lib/                  # add every link under assets to central's lib
casket -c central list -l textures/          # browse central's catalog, with sizes and keys
casket -c central mv textures/ lib/textures  # reorganize central's catalog
casket add textures/rock.png assets/         # link a central asset here, fetching it
casket add textures/ assets/textures         # link every asset under it; again to get new ones
casket fetch                                 # get the objects of the links under .
casket get rock.png                          # replace a link by a writable copy
casket drop assets/                          # free local copies central has, keep the links
casket gc                                    # remove unreferenced objects central has
casket verify                                # check every object against its key

The commands pair up, each moving one thing in one direction:

Commands that move things between two stores (fetch, drop, add, publish, gc) take the remote with -r. Every other command works on one store: the one owning its target, or the one given with -c/--cas, before or after the command. -c takes a path, an ssh url or the name of a remote, and runs the command in that store's root, so paths are relative to it; over ssh the command runs on the server.

publish only sends the objects central lacks, and only along with a link in its tree: nothing reaches central without a place in the catalog.

In a git project, commit the links and ignore the store: add /.casket to .gitignore, without a trailing slash so that it also matches a .casket symlink to a shared cache.

The central store does not know who links to it. Cleaning it (gc --force on central) may break the history of projects that linked what was removed; that is a deliberate decision of whoever runs it.

Remotes

A remote is another store's .casket directory (a store is always a directory named .casket, which init and remote add enforce), given as:

• a path: /mnt/nas/assets/.casket, or relative to the project root;
• over ssh: nas:/srv/assets/.casket, me@nas:assets/.casket (relative to the home directory), ssh://me@nas:2222/srv/assets/.casket.

An ssh remote uses the system ssh, so keys, agents and ~/.ssh/config work as for git. On the server it needs python3 (3.9 or later) and nothing installed: casket sends its own code over the connection and runs it there, so a store over ssh follows exactly the rules of a local one. Each command uses one connection; fetched objects are checked locally, uploads are checked on the server before they land.

Change where a remote is with casket remote set-url NAME URL, forget one with casket remote remove NAME.

Remotes added with --global (-g) go in ~/.config/casket/config (under $XDG_CONFIG_HOME) and are known to every store, and to -c NAME anywhere:

casket remote add -g nas nas:/srv/assets/.casket
casket -c nas list

A store's own remote of the same name wins over a global one. remote list marks the global remotes; set-url and remove take -g too. A global remote's path must be absolute: there is no project root to be relative to.

Create a store on a server the same way as a local one: casket --cas nas:/srv/assets/.casket init.

Set CASKET_SSH to change the ssh command, e.g. CASKET_SSH="ssh -i ~/.ssh/assets_key".

Compatibility contract

Projects commit the symlinks to git, so what a symlink contains must stay valid forever. Everything else may change.

1. Key format. Keys are SHA256E-s{size}--{sha256}{extension}, e.g. SHA256E-s123456--abcdef….png. New key formats may be added, but every key format ever written must keep being recognized.
2. Link targets. A link is a relative symlink to .casket/objects/{ab}/{cd}/{KEY}/{KEY}, where abcd are the first 4 hex characters of the MD5 of the key string. The .casket and objects names and this layout never change.
3. Scope. A directory holding an entry named .casket is the root of a scope: its tree, minus .casket itself, ignored directories (.git) and the trees of nested scopes. Any entry named .casket makes a scope, whatever it is: a store, a symlink to a shared cache (missing or not), or something that is not a store at all. The scope of a path is the nearest one at or above it, never one further up.
4. Link recognition. A symlink belongs to a scope's store if it lies in the scope and its target's final name parses as a key.
5. Store marker. A store's config holds [casket] format = 1. casket refuses to work in a scope whose .casket is not a store of a format it handles: a directory without the marker, a dangling symlink, a future format. It never falls back to a store further up, which would claim and rewrite links that are not its own. A future layout gets a new format number, which older versions refuse.

.casket may itself be a directory or a symlink to a shared cache: links go through the .casket name either way.

Shared cache

Projects can share one store by making their .casket a symlink to it:

casket --cas ~/.cache/casket/.casket init
ln -s ~/.cache/casket/.casket .casket

Each project still scopes its own tree, but the objects and the config, remotes included, are shared. gc only sees the links of the project it runs in, so in a shared cache it removes objects other projects use: gc makes them fetch again, gc --force can lose their local-only objects.

License

casket is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version. See LICENSE.