Skip to main content

The TiDB adapter plugin for dbt

Project description

TiDB logo dbt logo

dbt-tidb

PyPI PyPI - Python Version PyPI - Downloads Integration Test

The dbt-tidb package contains all of the code enabling dbt to work with TiDB.

This repository is partly based on dbt-mysql. Thanks to them for their excellent work.

Table of Contents

Installation

Compile by source code.

$ git clone https://github.com/pingcap/dbt-tidb.git
$ cd dbt-tidb
$ pip install .

Also, you can get it from pypi.

$ pip install dbt-tidb

Supported features

TiDB 4.X TiDB 5.0 ~ 5.2 TiDB >= 5.3 Feature
Table materialization
View materialization
Incremental materialization
Ephemeral materialization
Seeds
Sources
Custom data tests
Docs generate
Snapshots
Connection retry
Grant

Note:

  • TiDB 4.0 ~ 5.0 does not support CTE, you should avoid using WITH in your SQL code.
  • TiDB 4.0 ~ 5.2 does not support creating a temporary table or view.
  • TiDB 4.X does not support using SQL func in CREATE VIEW, avoid it in your SQL code. You can find more detail here.

Supported functions

cross-db macros are moved from dbt-utils into dbt-core, so you can use the following functions directly, see dbt-util on how to use them.

  • bool_or
  • cast_bool_to_text
  • dateadd
  • datediff
  • date_trunc
  • hash
  • safe_cast
  • split_part
  • last_day
  • cast_bool_to_text
  • concat
  • escape_single_quotes
  • except
  • intersect
  • length
  • position
  • replace
  • right
  • listagg (not support yet)

pay attention that datediff is a little different from dbt-util that it will round down rather than round up.

Profile Configuration

TiDB targets should be set up using the following configuration in your profiles.yml file.

Example entry for profiles.yml:

your_profile_name:
  target: dev
  outputs:
    dev:
      type: tidb
      server: 127.0.0.1
      port: 4000
      schema: database_name
      username: tidb_username
      password: tidb_password
      retries: 2
Option Description Required? Example
type The specific adapter to use Required tidb
server The server (hostname) to connect to Required yourorg.tidb.com
port The port to use Required 4000
schema Specify the schema (database) to build models into Required analytics
username The username to use to connect to the server Required dbt_admin
password The password to use for authenticating to the server Required correct-horse-battery-staple
retries The retry times for connection to TiDB (1 in default) Optional 2

Database User Privileges

Your database user would be able to have some abilities to read or write, such as SELECT, CREATE, and so on. You can find some help here with TiDB privileges management.

Required Privilege
SELECT
CREATE
CREATE TEMPORARY TABLE
CREATE VIEW
INSERT
DROP
SHOW DATABASE
SHOW VIEW
SUPER

Running Tests

See tests/README.md for details on running the integration tests.

Example

Click here to see a simple example about using dbt with dbt-tidb.

Contributing

Welcome to contribute for dbt-tidb. See Contributing Guide for more information.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

dbt-tidb-1.3.1.tar.gz (22.6 kB view hashes)

Uploaded source

Built Distribution

dbt_tidb-1.3.1-py3-none-any.whl (26.7 kB view hashes)

Uploaded py3

Supported by

AWS AWS Cloud computing Datadog Datadog Monitoring Facebook / Instagram Facebook / Instagram PSF Sponsor Fastly Fastly CDN Google Google Object Storage and Download Analytics Huawei Huawei PSF Sponsor Microsoft Microsoft PSF Sponsor NVIDIA NVIDIA PSF Sponsor Pingdom Pingdom Monitoring Salesforce Salesforce PSF Sponsor Sentry Sentry Error logging StatusPage StatusPage Status page