dbt-hardcharger 2.0.0

Build your dbt project on Hardcharger: remote DuckDB 2.0, over Warp SQL.

🔌 What is it?

The dbt adapter for Hardcharger™, GizmoData™'s DuckDB 2.0 server. dbt connects through the Hardcharger ADBC driver (adbc-driver-hardcharger), which speaks Warp SQL over HTTPS: Arrow results, bulk ingest, TLS by default, and every Hardcharger sign-in.

• 🧱 Every materialization: table, view, incremental (append, delete+insert, merge, microbatch), ephemeral, snapshots (check and timestamp), and external files written by the server (Parquet, CSV, JSON, partitioned, to local disk or object storage).
• 🐍 Python models, run on the dbt machine with a local DuckDB and shipped to the server by bulk ingest; record batch streams are streamed, and session.remote_sql() pushes a query down to the server.
• 🌱 Seeds read by DuckDB's CSV reader (real types and NULLs) and bulk-ingested.
• 🔐 Every sign-in: password, access token, client certificate (mutual TLS) and the server's browser sign-in.
• 🦆 DuckLake catalogs, constraints (enforced), persist_docs, dbt docs generate, dbt show, unit tests.

It is the Hardcharger port of dbt-gizmosql, and passes the same dbt adapter test suite (Testing).

📦 Installation

Python 3.11 or newer. From PyPI, once 2.0.0 is published:

pip install dbt-hardcharger

From the repositories (you need Go and a C compiler for the driver's library):

git clone https://github.com/gizmodata/hardcharger-adbc.git
git clone https://github.com/gizmodata/dbt-hardcharger.git
make -C hardcharger-adbc/go lib
python3 -m venv .venv
.venv/bin/pip install --editable hardcharger-adbc/python ./dbt-hardcharger

⚡ Quick start

A Hardcharger server with a certificate (self-signed here), using the hardcharger-server binary from a Hardcharger release:

openssl req -x509 -newkey ec -pkeyopt ec_paramgen_curve:prime256v1 -nodes -days 30 \
  -subj /CN=localhost -addext subjectAltName=DNS:localhost -keyout server.key -out server.pem
HARDCHARGER_ADMIN_PASSWORD='change-me' hardcharger-server \
  --tls-cert server.pem --tls-key server.key --database shop.duckdb

A dbt project in the same directory. dbt_project.yml:

name: shop
version: "1.0.0"
profile: shop
models:
  shop:
    +materialized: table

profiles.yml (dbt reads it from the project directory, or from ~/.dbt/). TLS is on by default, and tls_root_certs trusts the server's self-signed certificate:

shop:
  target: dev
  outputs:
    dev:
      type: hardcharger
      host: localhost
      port: 31337
      user: hardcharger
      password: "{{ env_var('HARDCHARGER_PASSWORD') }}"
      tls_root_certs: server.pem
      database: shop
      schema: main
      threads: 4

Two seeds, seeds/customers.csv and seeds/orders.csv:

id,name,signed_up
1,Ada,2026-01-05
2,Grace,2026-02-11
3,Linus,

id,customer_id,amount,ordered_at
1,1,120.50,2026-03-01
2,1,35.00,2026-03-04
3,2,410.25,2026-03-09
4,3,15.75,2026-04-10

A model, models/customer_orders.sql, with tests in models/schema.yml:

select
    c.id as customer_id,
    c.name,
    count(o.id) as orders,
    sum(o.amount) as total_amount
from {{ ref('customers') }} c
left join {{ ref('orders') }} o on o.customer_id = c.id
group by all
order by customer_id

models:
  - name: customer_orders
    columns:
      - name: customer_id
        data_tests: [unique, not_null]

And a Python model, models/big_spenders.py:

def model(dbt, session):
    dbt.config(materialized="table")
    return dbt.ref("customer_orders").filter(expr="total_amount > 100").order("customer_id")

Load the seeds, build the models, run the tests, and look at the result:

export HARDCHARGER_PASSWORD='change-me'
dbt seed
dbt run
dbt test
dbt show --select big_spenders

| customer_id | name  | orders | total_amount |
| ----------- | ----- | ------ | ------------ |
|           1 | Ada   |      2 |       155.50 |
|           2 | Grace |      1 |       410.25 |

🔐 Profiles and sign-in

An access token from somewhere else, for example Hardcharger's token endpoint (here with the quick start's server):

export HARDCHARGER_TOKEN=$(curl -s --cacert server.pem https://localhost:31337/warp/v1/auth/token \
  -H 'Content-Type: application/json' \
  -d '{"grant_type": "password", "username": "hardcharger", "password": "change-me"}' | jq -r .access_token)

shop:
  target: dev
  outputs:
    dev:
      type: hardcharger
      host: localhost
      auth_type: token
      token: "{{ env_var('HARDCHARGER_TOKEN') }}"
      tls_root_certs: server.pem

A client certificate signs in without a password on a server that verifies client certificates and lets them sign in (here analyst@example.com, the certificate's e-mail address, is an administrator; see Hardcharger's README, "Client certificates as identity"):

HARDCHARGER_ADMIN_PASSWORD='change-me' hardcharger-server \
  --tls-cert server.pem --tls-key server.key --mtls-ca-cert ca.pem \
  --mtls-certificate-sign-in --mtls-admin-identities analyst@example.com

shop:
  target: dev
  outputs:
    dev:
      type: hardcharger
      host: localhost
      auth_type: client_certificate
      tls_root_certs: ca.pem
      tls_client_cert: analyst.pem
      tls_client_key: analyst-key.pem

Browser sign-in through the server's identity provider (a server started with --oauth-client-id), on the server's own port: dbt opens the sign-in page once per run, and every connection of the run shares that sign-in.

shop:
  target: dev
  outputs:
    dev:
      type: hardcharger
      host: localhost
      auth_type: external
      tls_root_certs: ca.pem

🐍 Python models

Python models run on the dbt machine, in a local DuckDB, so the whole DuckDB Python API is there; the result goes to Hardcharger by ADBC bulk ingest. dbt.ref() and dbt.source() fetch their relation from the server as Arrow and hand it over as a DuckDB relation (with pandas-style column access). A model may return a DuckDB relation, a pandas DataFrame, a pyarrow.Table, or Arrow record batches:

• a pyarrow.RecordBatch or a pyarrow.RecordBatchReader;
• a list or a generator of pyarrow.RecordBatches;
• any object with the Arrow PyCapsule stream protocol (__arrow_c_stream__, a polars DataFrame for instance).

Readers, generators and stream objects are streamed to the server batch by batch, so the whole result never has to fit in the dbt machine's memory (models/numbers.py):

import pyarrow as pa


def model(dbt, session):
    dbt.config(materialized="table")
    schema = pa.schema(fields=[("id", pa.int64()), ("label", pa.string())])

    def batches():
        for start in (0, 1000, 2000):
            ids = list(range(start, start + 1000))
            yield pa.record_batch(data={"id": ids, "label": [f"n{i}" for i in ids]}, schema=schema)

    return pa.RecordBatchReader.from_batches(schema=schema, batches=batches())

dbt wants model() itself to return once, so the yields go in an inner function. A stream with no rows still creates its (empty, typed) table; an empty list or generator has no schema, so return an empty RecordBatchReader or Table with an explicit schema instead.

Streaming from another database. A returned reader is read after model() returns, so its source must still be open then. A reader from an ADBC cursor.fetch_record_batch() closes with its cursor, so don't return it from inside a with block (the model fails with "returned a record batch stream that was already closed"): yield it instead. dbt-hardcharger streams the reader, then resumes the generator, and the with block closes the connection. Any ADBC driver works; here the source is another Hardcharger server (models/copied_orders.py):

import os

from adbc_driver_hardcharger import dbapi as hardcharger


def extract(query):
    with (
        hardcharger.connect(
            uri=os.environ["SOURCE_URI"],
            username=os.environ["SOURCE_USERNAME"],
            password=os.environ["SOURCE_PASSWORD"],
            tls_root_certs=os.environ["SOURCE_TLS_ROOT_CERTS"],
        ) as conn,
        conn.cursor() as cursor,
    ):
        cursor.execute(operation=query)
        yield cursor.fetch_record_batch()  # `yield`, not `return`


def model(dbt, session):
    dbt.config(materialized="table")
    return extract(query="select * from shop.main.orders")

This works for incremental models too, and an empty result still creates the (empty) table. To collect the whole result first instead, return cursor.fetch_arrow_table().

Pushing a query down with session.remote_sql(). dbt.ref("big_table").filter(...) brings the whole table to the dbt machine before filtering it. session.remote_sql(query) runs the query on the server, and only its result crosses the network (models/large_orders.py):

def model(dbt, session):
    dbt.config(materialized="table")
    orders = f"{dbt.this.database}.{dbt.this.schema}.orders"
    # Runs on the Hardcharger server; only the matching rows come back.
    return session.remote_sql(query=f"select * from {orders} where amount > 100")

It returns a local DuckDB relation (chain .filter(), .project(), .df(), ...). The rest of session is the local DuckDB connection (session.sql(), session.register(), ...).

🌱 Seeds

Seeds are read by DuckDB's CSV reader on the dbt machine and bulk-ingested: empty fields become NULL (not the text 'null'), dates are DATE, integers BIGINT, and so on. column_types and delimiter work as in other adapters, and it is much faster than dbt's default batches of INSERTs.

📤 External files written by the server

materialized='external' writes a model to Parquet, CSV or JSON files with DuckDB's COPY, and creates a view over them for downstream models to ref(). The config is dbt-duckdb's. The COPY runs on the Hardcharger server, not on the dbt machine: paths are the server's, and so are the credentials for object storage (s3://, gs://, azure://, ...). That's usually what you want: the server has the CPUs, the network and the cloud role, and the result never travels to the dbt machine and back.

models/orders_export.sql writes orders_export.parquet in external_root (by default the server's working directory); models/orders_by_month.sql writes one zstd Parquet directory per month:

{{ config(materialized='external') }}
select * from {{ ref('orders') }}

{{ config(
    materialized='external',
    options={'partition_by': 'month', 'compression': 'zstd'}
) }}
select *, strftime(ordered_at, '%Y-%m') as month from {{ ref('orders') }}

A single-file COPY doesn't create directories, so the directory of a location (or external_root) must exist on the server; object storage URIs have no directories to create. External models are always rebuilt in full (no incremental strategies), and dbt-duckdb's client-side plugin and glue_register options are refused with an error.

🦆 DuckLake catalogs

Set database to a DuckLake catalog attached to the server and dbt builds there. dbt-hardcharger finds DuckLake catalogs in duckdb_databases() and works around what DuckLake doesn't support: it replaces relations without DROP ... CASCADE, creates only NOT NULL contract constraints (the others, and the indexes config, are skipped with a warning; warn_unsupported: false silences one), so a shared schema.yml still builds.

DuckLake holds its metadata transaction open, with its Postgres session idle, for the whole of a write, and a streamed Python model result keeps it open while the source produces its batches. A metadata server whose idle_in_transaction_session_timeout is shorter kills the write at commit (Failed to commit: Failed to execute query "ROLLBACK": ; the adapter adds a hint). Raise the timeout for the catalog's role (ALTER ROLE <catalog_role> SET idle_in_transaction_session_timeout = '2h'), or return cursor.fetch_arrow_table() instead of streaming from a slow source.

🧩 Also

• Constraints: check, not_null, unique, primary_key and foreign_key, all enforced.
• Documentation: persist_docs (COMMENT ON relations and columns) and dbt docs generate.
• Macros: DuckDB versions of dateadd, last_day, listagg and split_part.
• Grants: dbt's grants config isn't applied yet (dbt-hardcharger warns); run Hardcharger GRANT statements in a post-hook meanwhile.
• Transactions: every statement commits on its own (autocommit), as in dbt-gizmosql.

🔁 Coming from dbt-gizmosql

The same features and macros; the profile changes:

🧪 Testing

The suite is dbt's adapter test suite (dbt-tests-adapter), dbt-gizmosql's own tests, sign-in tests and this README: its marked examples run as written against real servers (tests/functional/test_readme.py), and CI installs the package with the Installation commands. The tests start hardcharger-server processes on free loopback ports from DBT_HARDCHARGER_TEST_SERVER_BINARY; without it they fail, never skip. The S3 test needs an S3 gateway at DBT_HARDCHARGER_TEST_S3_ENDPOINT (default 127.0.0.1:9000).

On a Mac (or anywhere with Docker), scripts/test-in-docker.sh runs it all in a Linux container, with the server from ghcr.io/gizmodata/hardcharger:main and the driver built from the hardcharger-adbc commit pinned in .github/hardcharger-adbc.ref:

DBT_HARDCHARGER_TEST_ADBC_SRC=~/git/hardcharger-adbc scripts/test-in-docker.sh -n 6

The driver stays pinned: to move to a newer hardcharger-adbc, change the commit in .github/hardcharger-adbc.ref on its own, and merge it only with a green CI run.

Lint, format and type checks: ruff check ., ruff format --check . and mypy.

📦 Releasing

Semantic versioning and CHANGELOG.md. A vX.Y.Z tag runs the whole CI on the tagged commit (release.yml calls ci.yml), then builds the wheel and sdist and creates a GitHub Release with the tag's CHANGELOG section. PyPI publishing comes when dbt-hardcharger and adbc-driver-hardcharger go public.

📄 License

Apache License 2.0; see LICENSE.

_{Built by GizmoData™}

_{Hardcharger™ and GizmoData™ are trademarks of GizmoData LLC. dbt is a trademark of dbt Labs, Inc.}