dbt-dlc 1.0.0

A dbt adapter for Tencent Cloud DLC (Data Lake Compute), built on dbt-spark

dbt-dlc

A dbt adapter for Tencent Cloud DLC (Data Lake Compute) — a serverless Spark SQL engine.

dbt-dlc extends dbt-spark, replacing the connection layer with dlc-sql-python (a DBAPI 2.0 Thrift client) while inheriting the full Spark SQL dialect and macros.

---

Installation

Requires Python 3.9+ and dbt 1.8+.

pip install dbt-dlc

Verify:

dbt --version
# Should include: dlc: 0.1.0

---

Quick Start

1. Create a profiles.yml (~/.dbt/profiles.yml)

my_dlc_project:
  target: dev
  outputs:
    dev:
      type: dlc
      host: dlc-thrift.example.com           # DLC Thrift endpoint
      port: 10009                             # default: 10009
      engine_name: my-engine                  # DLC engine name
      resource_group_name: default-rg-xxxx    # resource group
      catalog: DataLakeCatalog                # catalog name (default: DataLakeCatalog)
      schema: my_database                     # DLC database name
      auth_mode: AccessKey                    # "AccessKey" or "RoleBasedSSO"
      transport_mode: binary                  # "binary" | "http" | "https"
      http_path: /cliservice                  # HTTP path (default)
      secret_id: AKIDxxxxxxxxxxxxxxxx         # Tencent Cloud SecretId
      secret_key: xxxxxxxxxxxxxxxx            # Tencent Cloud SecretKey
      socket_timeout_ms: 300000               # 5 min for engine cold start
      threads: 4

2. Create dbt_project.yml

name: my_dlc_project
version: '1.0'
config-version: 2
profile: my_dlc_project

model-paths: ["models"]

models:
  my_dlc_project:
    materialized: table
    +file_format: iceberg               # recommended for DLC

3. Write your first model (models/my_first_model.sql)

select
    1 as id,
    'hello' as name,
    current_timestamp() as created_at

4. Run

dbt debug           # verify connection
dbt run             # build models
dbt test            # run data quality tests

---

Authentication

AccessKey (Development)

auth_mode: AccessKey
secret_id: AKIDxxxxxxxxxx
secret_key: xxxxxxxxxxxxxx

RoleBasedSSO — OAuth M2M (CI/CD)

auth_mode: RoleBasedSSO
auth_flow: m2m
role_arn: qcs::cam::uin/xxxxxxxxxx:roleName/role-name
identity_provider_name: dlc_connect
oauth_token_url: https://login.microsoftonline.com/{tenant}/oauth2/v2.0/token
oauth_client_id: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
oauth_client_secret: xxxxxxxxxxxxxx

RoleBasedSSO — OAuth U2M (Local Dev)

auth_mode: RoleBasedSSO
auth_flow: u2m
role_arn: qcs::cam::uin/xxxxxxxxxx:roleName/role-name
identity_provider_name: dlc_connect
oauth_token_url: https://login.microsoftonline.com/{tenant}/oauth2/v2.0/token
oauth_authorize_url: https://login.microsoftonline.com/{tenant}/oauth2/v2.0/authorize
oauth_client_id: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
oauth_client_secret: xxxxxxxxxxxxxx
oauth_persistence_path: ~/.dlc/token_cache.json

---

Model Configuration

Table

models:
  my_dlc_project:
    materialized: table
    +file_format: iceberg

View

models:
  my_dlc_project:
    staging:
      materialized: view

Incremental (Merge / Append / Insert Overwrite)

{{
  config(
    materialized='incremental',
    file_format='iceberg',
    unique_key='event_id',
    strategy='merge'
  )
}}

select event_id, event_name, event_date
from source_events
{% if is_incremental() %}
  where event_date > (select max(event_date) from {{ this }})
{% endif %}

Supported strategies:

Strategy	Description
merge	MERGE INTO — upsert based on unique_key
append	INSERT INTO — append-only
insert_overwrite	INSERT OVERWRITE — replace partition or entire table

Table Properties

{{
  config(
    materialized='table',
    file_format='iceberg',
    partition_by=['event_date'],
    clustered_by=['user_id'],
    buckets=16,
    location_root='cosn://my-bucket/path',
    tblproperties={'write.format.default': 'parquet'},
    persist_docs={'relation': true, 'columns': true},
    comment="My table description"
  )
}}

Seed Configuration

seeds:
  my_dlc_project:
    +quote_columns: false
    +file_format: iceberg

---

Three-Level Namespace

DLC uses catalog.database.table naming. dbt-dlc maps these as follows:

dbt Term	DLC Layer	Profile Field
database	Catalog	catalog (default: DataLakeCatalog)
schema	Database	schema
identifier	Table/View	model name / alias

---

Iceberg Procedures (Convenience Macros)

Available as post-hook macros for managing Iceberg tables:

Expire Old Snapshots

{{ config(
    post_hook="
      {{ dlc__expire_snapshots(this, retain_last=100) }}
    "
) }}

Compaction / Re-sort

{{ config(
    post_hook="
      {{ dlc__rewrite_data_files(this, strategy='sort', sort_order='id DESC NULLS LAST') }}
    "
) }}

Remove Orphan Files

{{ dlc__remove_orphan_files(this, older_than=\"TIMESTAMP '2026-06-01'\") }}

Rollback to a Snapshot

{{ dlc__rollback_to_snapshot(this, snapshot_id=123456789) }}

---

Known Limitations

Item	Detail
Python models	Not yet supported
ALTER TABLE RENAME	DLC does not support ALTER TABLE RENAME; dbt-dlc uses CREATE OR REPLACE as a workaround
USE CATALOG	DLC does not support USE CATALOG; catalog is set via session property spark.sql.defaultCatalog
clustered_by / buckets	DLC may not support these options; use sort_order + Iceberg compaction instead
Transactions	Not supported (DLC/Spark has no transactions)

---

Testing

# Unit tests (no DLC connection required)
pytest tests/ -v

# Integration tests (requires DLC connection — see tests/integration/.env.example)
cd tests/integration
cp .env.example .env        # fill in your credentials
./run_tests.sh full dev     # full lifecycle: debug → seed → run → run → test → docs

---

Project Structure

dbt-dlc/
├── dbt/
│   ├── adapters/dlc/
│   │   ├── __init__.py         # Plugin registration
│   │   ├── connections.py      # DLCCredentials + DLCConnectionManager
│   │   ├── impl.py             # DLCAdapter
│   │   └── relation.py         # DLCRelation (3-level namespace)
│   └── include/dlc/
│       ├── dbt_project.yml
│       └── macros/
│           └── adapters.sql    # Spark SQL macros + DLC overrides
├── tests/
│   ├── test_connections.py
│   ├── test_relation.py
│   ├── test_macros.py
│   └── integration/
└── docs/plans/