fdl

Versions of a DuckLake catalog on object storage, kept the way Git keeps commits.

FDL is a storage layout and a tool for it. Each version is a frozen catalog with a record of its parent, author and message; refs such as refs/heads/main say which version readers get. Data files are shared between versions and never rewritten. All it needs is a store that can read, write and write on condition: a folder, S3, R2 or anything S3-compatible. There is no server.

<root>/
  fdl.json
  HEAD                              ref: refs/heads/main
  refs/heads/main                   019a3c4e-…
  versions/019a3c4e-…/
    version.json                    parents, author, message, sha256
    ducklake.duckdb                 the frozen catalog
  data/main/towns/ducklake-….parquet

The documentation and the specification are at fdl.flo8s.com.

Quick start

pip install 'fdl-cli[s3]'

fdl init s3://my-bucket/towns/
fdl checkout s3://my-bucket/towns/ work/lake.sqlite

The package is fdl-cli on PyPI. It installs the fdl command and the fdl Python package (import fdl).

Write to the working catalog with DuckDB, attached the way checkout prints:

ATTACH 'ducklake:work/lake.sqlite' AS lake (
  DATA_PATH 's3://my-bucket/towns/data/', OVERRIDE_DATA_PATH true,
  META_TYPE 'sqlite', DATA_INLINING_ROW_LIMIT 0, META_JOURNAL_MODE 'WAL', BUSY_TIMEOUT 5000
);
CREATE TABLE lake.main.towns AS SELECT * FROM 'towns.csv';

Rows have to go to Parquet, not into the catalog (DATA_INLINING_ROW_LIMIT 0): a version is read over HTTP, and inlined rows would not be in it.

Then make a version and move main to it:

fdl commit work/lake.sqlite -m "First towns" --author alice
fdl log s3://my-bucket/towns/

Anyone who can read the bucket can attach it:

fdl url s3://my-bucket/towns/

ATTACH 'ducklake:s3://my-bucket/towns/versions/019a3c4e-…/ducklake.duckdb' AS towns (
  DATA_PATH 's3://my-bucket/towns/data/', OVERRIDE_DATA_PATH true, READ_ONLY
);

Credentials and the endpoint come from boto3's usual places (AWS_* environment variables, AWS_ENDPOINT_URL for R2 and other S3-compatible stores). A folder works as a repository too: pass its path instead of an s3:// URL.

Commands

A push that is not built on what main holds now is refused: check out again and redo the work.

Python

from fdl import repo
from fdl.stores import open_store

store = open_store("s3://my-bucket/towns/")
for version in repo.log(store):
    print(version.seq, version.id, version.message)

fdl.repo has the operations without a terminal (init, checkout, commit, push, log, expire, prune), for tools and services that build on FDL.

Earlier releases

Releases up to 0.11 were a different tool for managing a single frozen catalog, published as frozen-ducklake. Their code is at the legacy branch.

License

MIT