Combine duckdb-dbt and Kedro Dataset to easily read Kedro Dataset configs (yaml), enabling conversion of Kedro projects to dbt.
Project description
Combine duckdb-dbt and Kedro Datasets to enable:
- extension of dbt to ingest wide array of data, and;
- conversion of Kedro projects to dbt by easily reading your Kedro data catalog configs (yaml files)
Demo
You can add your existing Kedro definitions to your dbt sources like so:
pip install dbt_duckdb_kedro_datasets
version: 2
sources:
- name: my_source # can call this anything
schema: main
meta:
plugin: dbt_duckdb_kedro_datasets # this library
tables:
- name: my_table # can call this anything
description: "A dbt_duckdb_kedro_datasets test"
meta:
type: pandas.CSVDataset
filepath: ./data/1_raw/bikes.csv # file to ingest
load_args:
sep: ','
Now we can access this CSV in dbt
select *
from {{ source('my_source', 'my_table') }}
For a more complete example look at this
Functionality
This gives you access to read/write Excel Sheets, Parquet, Json, DeltaTable, Pickle and many more!
note: I've only tested this with CSV data so far so please let me know if you run into any issues. Particularly non-tabular data (i.e. picture bit values etc.) will probably not be compatible (since dbt expects dataframe like objects returned).
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for dbt_duckdb_kedro_datasets-0.1.2.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | 83186b64b9fb7f7dba47fa04c465736416bd2a8abbd518a8bc1bbf8566498379 |
|
MD5 | 98e80fc7b22c386a78a2f5991f68862b |
|
BLAKE2b-256 | 46c90453eca6776d043f2cbf070901b59f7b59756134340a2efface0fcec718d |
Hashes for dbt_duckdb_kedro_datasets-0.1.2-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 57cafd3dd67ce7feaf171769929081609e51a2f7032abd3ee328e72aa0bfa343 |
|
MD5 | 3c246070c1a7e6746dc11f26bb96bd22 |
|
BLAKE2b-256 | 5134ee3d60322e79dba145ed2a6d13f941a159d368f03639c5147ccbcd2fdfff |