The trino adapter plugin for dbt (data build tool)
Project description
dbt-trino
Introduction
One frequently asked question in the context of using dbt
tool is:
Can I connect my dbt project to two databases?
(see the answered question on the dbt website).
tldr; dbt
stands for transformation as in T
within ELT
pipelines, it doesn't move data from source to a warehouse.
The creators of the dbt
tool have added however support for handling such scenarios via
dbt-presto plugin.
This repository represents a fork of the dbt-presto with slight adaptations to make it work with Trino SQL compute engine.
Compatibility
This dbt plugin has been tested against trino
version 362
.
Installation
This dbt adapter can be installed via pip:
$ pip install dbt-trino
Configuring your profile
A dbt profile can be configured to run against Trino using the following configuration:
Option | Description | Required? | Example |
---|---|---|---|
method | The Trino authentication method to use | Optional (default is none ) |
none or kerberos |
user | Username for authentication | Required | commander |
password | Password for authentication | Optional (required if method is ldap or kerberos ) |
none or abc123 |
http_headers | HTTP Headers to send alongside requests to Trino, specified as a yaml dictionary of (header, value) pairs. | Optional | X-Trino-Client-Info: dbt-trino |
http_scheme | The HTTP scheme to use for requests to Trino | Optional (default is http , or https for method: kerberos and method: ldap ) |
https or http |
session_properties | Sets Trino session properties used in the connection | Optional | query_max_run_time: 5d |
database | Specify the database to build models into | Required | analytics |
schema | Specify the schema to build models into. Note: it is not recommended to use upper or mixed case schema names | Required | public |
host | The hostname to connect to | Required | 127.0.0.1 |
port | The port to connect to the host on | Required | 8080 |
threads | How many threads dbt should use | Optional (default is 1 ) |
8 |
Example profiles.yml entry:
my-trino-db:
target: dev
outputs:
dev:
type: trino
user: commander
host: 127.0.0.1
port: 8080
database: analytics
schema: public
threads: 8
http_scheme: http
session_properties:
query_max_run_time: 5d
exchange_compression: True
For reference on which session properties can be set on the the dbt profile do execute
SHOW SESSION;
on your Trino instance.
Usage Notes
Supported Functionality
Due to the nature of Trino, not all core dbt
functionality is supported.
The following features of dbt are not implemented on Trino:
- Archival
- Incremental models
Also, note that upper or mixed case schema names will cause catalog queries to fail. Please only use lower case schema names with this adapter.
Required configuration
dbt fundamentally works by dropping and creating tables and views in databases. As such, the following Trino configs must be set for dbt to work properly on Trino:
hive.metastore-cache-ttl=0s
hive.metastore-refresh-interval = 5s
hive.allow-drop-table=true
hive.allow-rename-table=true
Use table properties to configure connector specifics
Trino connectors use table properties to configure connector specifics.
Check the Trino connector documentation for more information.
{{
config(
materialized='table',
properties={
"format": "'PARQUET'",
"partitioning": "ARRAY['bucket(id, 2)']",
}
)
}}
Running tests
Build dbt container locally:
./docker/dbt/build.sh
Run a Trino server locally:
./docker/init.bash
Run tests against Trino:
./docker/run_tests.bash
Run the locally-built docker image (from docker/dbt/build.sh):
export DBT_PROJECT_DIR=$HOME/... # wherever the dbt project you want to run is
docker run -it --mount "type=bind,source=$HOME/.dbt/,target=/root/.dbt" --mount="type=bind,source=$DBT_PROJECT_DIR,target=/usr/app" --network dbt-net dbt-trino /bin/bash
Running integration tests
Install the libraries required for development in order to be able to run the dbt tests:
pip install -r dev_requirements.txt
Run from the base directory of the project the command:
tox
or
pytest test/integration/trino.dbtspec
Code of Conduct
Everyone interacting in the dbt project's codebases, issue trackers, chat rooms, and mailing lists is expected to follow the PyPA Code of Conduct.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.