The duckdb adapter plugin for dbt (data build tool)
Project description
dbt-duckdb
Installation
This project is hosted on PyPI, so you should be able to install it and the necessary dependencies via:
pip3 install dbt-duckdb
The latest supported version targets dbt-core
1.2.x and duckdb
version 0.5.0 (but should also work with 0.4.0 and 0.3.2!)
Configuring your profile
DuckDB is an embedded database, similar to SQLite, but designed for OLAP-style analytics instead of OLTP. The only
configuration parameter that is required in your profile (in addition to type: duckdb
) is the path
field, which should refer to
a path on your local filesystem where you would like the DuckDB database file (and it's associated write-ahead log) to be written.
You can also specify the schema
parameter if you would like to use a schema besides the default (which is called main
).
Prior to version 1.2.1, dbt-duckdb was limited to a single dbt execution thread; in version 1.2.1 this constraint was lifted and now dbt-duckdb can run with as many threads as you can throw at it.
There is also a database
field defined in the DuckDBCredentials
class for consistency with the parent Credentials
class,
but it defaults to main
and setting it to be something else will likely cause strange things to happen that I cannot fully predict,
so, ya know, don't do that.
As of version 1.2.0, dbt-duckdb also allows you to configure your S3 settings in your credentials, including s3_region
and
either s3_session_token
or s3_access_key_id
and s3_secret_access_key
, so that you can use dbt-duckdb to read and transform
data stored in S3 files. You can also specify an arbitrary number of DuckDB extensions to
load as part of your dbt-duckdb project using the extensions: []
credentials field.
Developer Workflow
If you need to add features to dbt-duckdb itself, use this workflow to get started:
$ git clone https://github.com/jwills/dbt-duckdb.git
$ cd dbt-duckdb
$ "hack on stuff for awhile"
$ pip3 install .
$ pytest tests/
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for dbt_duckdb-1.2.1-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 69a1cd8b8bfffd5169859244779e683c95bfc459b66779296c2a0220d03aecba |
|
MD5 | b8e368d883213e72225bd59c7cce0091 |
|
BLAKE2b-256 | 6c54d117d58671fe2a776ec13a58d2f2052c16a020ef5f31ee03751529beed34 |