The Starrocks adapter plugin for dbt
Project description
dbt-starrocks
This project is under development.
The dbt-starrocks
package contains all the code enabling dbt to work with StarRocks.
This is an experimental plugin:
- We have not tested it extensively
- Requirements at least StarRocks version 2.5.0+
- version 3.1.x is recommended
- Previous versions will no longer support
Installation
This plugin can be installed via pip:
$ pip install dbt-starrocks
Supported features
Notice
- When StarRocks Version < 2.5,
Create table as
can only set engine='OLAP' and table_type='DUPLICATE' - When StarRocks Version >= 2.5,
Create table as
support table_type='PRIMARY' - When StarRocks Version < 3.1 distributed_by is must
Profile Configuration
Example entry for profiles.yml:
starrocks:
target: dev
outputs:
dev:
type: starrocks
host: localhost
port: 9030
schema: analytics
username: your_starrocks_username
password: your_starrocks_password
Option | Description | Required? | Example |
---|---|---|---|
type | The specific adapter to use | Required | starrocks |
host | The hostname to connect to | Required | 192.168.100.28 |
port | The port to use | Required | 9030 |
schema | Specify the schema (database) to build models into | Required | analytics |
username | The username to use to connect to the server | Required | dbt_admin |
password | The password to use for authenticating to the server | Required | correct-horse-battery-staple |
version | Let Plugin try to go to a compatible starrocks version | Optional | 3.1.0 |
Example
dbt seed properties(yml):
Complete configuration:
config:
engine: 'OLAP'
keys: ['id', 'name', 'some_date']
table_type: 'PRIMARY' //PRIMARY or DUPLICATE or UNIQUE
distributed_by: ['id']
buckets: 3 //default 10
partition_by: ['some_date']
partition_by_init: ["PARTITION p1 VALUES [('1971-01-01 00:00:00'), ('1991-01-01 00:00:00')),PARTITION p1972 VALUES [('1991-01-01 00:00:00'), ('1999-01-01 00:00:00'))"]
properties: [{"replication_num":"1", "in_memory": "true"}]
dbt run config(table/incremental):
Minimum configuration:
{{ config(materialized=var("materialized_var", "table"), distributed_by=['id'])}}
{{ config(materialized='incremental', distributed_by=['id']) }}
Complete configuration:
{{ config(materialized='table', engine='OLAP', buckets=32, distributed_by=['id'], properties={"in_memory": "true"}) }}
{{ config(materialized='incremental', engine='OLAP', buckets=32, distributed_by=['id'], properties={"in_memory": "true"}) }}
Test Adapter
consult the project
Contributing
Welcome to contribute for dbt-starrocks. See Contributing Guide for more information.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
dbt-starrocks-1.1.1.tar.gz
(15.0 kB
view hashes)
Built Distribution
Close
Hashes for dbt_starrocks-1.1.1-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 659b934f28661eac30b7d029aab6f9419d6769e9390e65b204efaf21b1de2792 |
|
MD5 | 672bb61d0cd119a7062f9dbf35aabae6 |
|
BLAKE2b-256 | 65f40811eb4717636624d9d2e2a0f36bd23469ea638a3e8765649e8c591816dd |