dbt-lakecompute

The lake compute (dbt Compute) adapter plugin for dbt-core.

Lake compute is a remote query service whose workers run DuckDB over Iceberg tables and accept DuckDB-dialect SQL.

The adapter is under active development. dbt debug, dbt compile, dbt show, dbt run (tables, incremental and ephemeral models), dbt seed and dbt test work.

Supported and unsupported features

Identifiers are never quoted, matching the service's own SQL, and are matched case-insensitively: a relation created as MixedCase is found by ref('mixedcase') and still renders as MixedCase.

Profile

method defaults to token, which reads token — a service bearer token. For a Fivetran personal access token (a dct_ PAT) use method: fivetran and fivetran_credential instead: a PAT is exchanged for a session rather than sent as a bearer token, so method: token with a PAT fails with 401 invalid token.

Always supply auth material through an environment variable, never inline.

my_project:
  target: dev
  outputs:
    dev:
      type: lakecompute
      database: my_database
      schema: my_schema

      method: fivetran
      fivetran_credential: "{{ env_var('DBT_COMPUTE_AUTH_TOKEN') }}"

      # optional; falls back to $DBT_COMPUTE_BASE_URL, then the public endpoint
      base_url: https://api.lake-compute.fivetran.com/

With a bearer token, the last two auth lines become:

      method: token   # the default, so it can be omitted
      token: "{{ env_var('DBT_COMPUTE_AUTH_TOKEN') }}"

Set DBT_COMPUTE_AUTH_TOKEN to whichever credential you are using. The ADBC driver is fetched automatically; see Driver below. Run dbt debug to check the connection.

Driver

The adapter needs a closed-source ADBC shared library, which is not published to PyPI. It downloads it from dbt's public CDN the first time a connection is opened and caches it at {cache_dir}/com.getdbt/adbc/{arch}-{os}/, so a machine that already has it downloads nothing. Later runs only stat the file.

The driver version is pinned, and every platform artifact is verified against a pinned SHA-256. A platform with no pinned checksum is refused rather than installed unverified. Both live in install.py.

Set driver_path in the profile to bypass the download entirely — for a locally built driver, or a platform with no prebuilt artifact.

Connection settings

Development

cd dbt-lakecompute
hatch run setup
hatch run code-quality
hatch run unit-tests
hatch run integration-tests

Unit tests need no endpoint. Never commit test.env.

The functional suite runs against a live endpoint. Copy test.env.example to test.env. It needs DBT_COMPUTE_BASE_URL, DBT_COMPUTE_AUTH_TOKEN and DBT_COMPUTE_DATABASE; the tests skip without them. DBT_COMPUTE_AUTH_METHOD selects the auth method (fivetran by default; set token for a plain bearer token). Set DBT_COMPUTE_SCHEMA as well to build everything in one named scratch schema rather than a generated name per test file — useful against a shared catalog, and it makes leftovers easy to find if a run is interrupted. Pinning the schema means test classes cannot run in parallel, so pass -n0 with it. Every statement is a full round trip at 3–15 seconds, so a run of the whole functional suite takes tens of minutes.