Skip to main content

The duckdb adapter plugin for dbt (data build tool)

Project description

dbt-duckdb

Installation

This project is hosted on PyPI, so you should be able to install it and the necessary dependencies via:

pip3 install dbt-duckdb

The latest supported version targets dbt-core 1.2.x and duckdb version 0.5.0 (but should also work with 0.4.0 and 0.3.2!)

Configuring your profile

DuckDB is an embedded database, similar to SQLite, but designed for OLAP-style analytics instead of OLTP. The only configuration parameter that is required in your profile (in addition to type: duckdb) is the path field, which should refer to a path on your local filesystem where you would like the DuckDB database file (and it's associated write-ahead log) to be written. You can also specify the schema parameter if you would like to use a schema besides the default (which is called main).

Note that dbt-duckdb currently only works in single-threaded mode, just like dbt-sqlite; if you try to run dbt-duckdb with multiple threads configured in your profile, the adapter will raise an exception. We are looking forward to fixing this limitation in the near future.

There is also a database field defined in the DuckDBCredentials class for consistency with the parent Credentials class, but it defaults to main and setting it to be something else will likely cause strange things to happen that I cannot fully predict, so, ya know, don't do that.

As of version 1.2.0, dbt-duckdb also allows you to configure your S3 settings in your credentials, including s3_region and either s3_session_token or s3_access_key_id and s3_secret_access_key, so that you can use dbt-duckdb to read and transform data stored in S3 files. You can also specify an arbitrary number of DuckDB extensions to load as part of your dbt-duckdb project using the extensions: [] credentials field.

Developer Workflow

If you need to add features to dbt-duckdb itself, use this workflow to get started:

$ git clone https://github.com/jwills/dbt-duckdb.git
$ cd dbt-duckdb
$ "hack on stuff for awhile"
$ pip3 install .
$ pytest tests/

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

dbt-duckdb-1.2.0.tar.gz (13.4 kB view details)

Uploaded Source

Built Distribution

dbt_duckdb-1.2.0-py3-none-any.whl (15.7 kB view details)

Uploaded Python 3

File details

Details for the file dbt-duckdb-1.2.0.tar.gz.

File metadata

  • Download URL: dbt-duckdb-1.2.0.tar.gz
  • Upload date:
  • Size: 13.4 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.0 CPython/3.9.12

File hashes

Hashes for dbt-duckdb-1.2.0.tar.gz
Algorithm Hash digest
SHA256 1fb5e42f20e749c75c654227a61b13440daf09714200b1d4774512b7d7e56184
MD5 7905646524d6e74244bd014135c169f1
BLAKE2b-256 984e9a1e00324779911a106f9abd37fe7dd30814d10622ccfff752984f1f6d33

See more details on using hashes here.

File details

Details for the file dbt_duckdb-1.2.0-py3-none-any.whl.

File metadata

  • Download URL: dbt_duckdb-1.2.0-py3-none-any.whl
  • Upload date:
  • Size: 15.7 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.0 CPython/3.9.12

File hashes

Hashes for dbt_duckdb-1.2.0-py3-none-any.whl
Algorithm Hash digest
SHA256 8326358832347f6fb43d22a2b700bb49d576c27c6dcf9864fe21f4188f73c3e8
MD5 688399945104a237aee07b9fb2843de2
BLAKE2b-256 2bf40a092ad9bb50f90355a6987c186ab5694fc5b6783bae87a8e10a7262aa0b

See more details on using hashes here.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Pingdom Monitoring Sentry Error logging StatusPage Status page