dlc-sql-python 1.0.0

A Python DBAPI 2.0 client for Tencent Cloud DLC (Data Lake Compute) via HiveServer2 Thrift

dlc-sql-python

A Python DBAPI 2.0 (PEP 249) client for Tencent Cloud DLC (Data Lake Compute), connecting to its HiveServer2-compatible Thrift endpoint (TCLIService).

It is the connection layer used by dbt-dlc (a dbt adapter that extends dbt-spark), and is the Python counterpart of the TypeScript dlc-sql-nodejs reference implementation.

Features

• DBAPI 2.0 compliant — apilevel='2.0', paramstyle='pyformat', full connect / cursor / execute / fetchone / fetchmany / fetchall / description / rowcount API and PEP 249 exception hierarchy.
• Three transport modes
  • binary (default) — TCP socket + custom SASL PLAIN handshake + framed TBinaryProtocol.
  • http — HTTP POST with X-DLC-* auth headers.
  • https — HTTPS POST with X-DLC-* auth headers (production / public endpoints).
• Two authentication modes
  • AccessKey — Tencent Cloud SecretId / SecretKey.
  • RoleBasedSSO — role-based SSO with a pre-obtained access token (Manual), plus an OAuthManager helper for M2M (client_credentials) and U2M (authorization_code + PKCE, browser login) token acquisition.
• Async execution + polling — ExecuteStatement(runAsync=true) → poll GetOperationStatus → GetResultSetMetadata → FetchResults.
• Columnar result parsing — TColumn union (bool/byte/i16/i32/i64/double/ string/binary) + nulls bitmap → Python values.

Installation

pip install dlc-sql-python

From source (development):

cd dlc-sql-python
pip install -e ".[dev]"
pytest

Quick start

import dlc_sql

conn = dlc_sql.connect(
    host="dlc-thrift.example.com",
    engine_name="my-spark-engine",
    resource_group_name="my-resource-group",
    catalog="hive",
    database="default",
    auth_mode="AccessKey",
    transport_mode="https",      # or "binary" / "http"
    port=443,
    secret_id="AKIDxxxx",        # via env var in production
    secret_key="xxxx",
)
cursor = conn.cursor()
cursor.execute("SELECT id, name FROM users WHERE id = %(id)s", {"id": 1})
print(cursor.description)
print(cursor.fetchall())
cursor.close()
conn.close()

Connection parameters

Parameter	Required	Default	Description
host	yes	—	DLC Thrift endpoint host
engine_name	yes	—	Spark engine name
resource_group_name	yes	—	Resource group name
catalog	yes	—	Catalog (e.g. hive)
database	yes	—	Database (schema is accepted as an alias)
auth_mode	yes	—	AccessKey or RoleBasedSSO
transport_mode	no	binary	binary / http / https
port	no	10009 (binary/http), 443 (https)	Thrift port
http_path	no	/cliservice	HTTP path (http/https)
socket_timeout_ms	no	60000	Transport timeout
secret_id / secret_key	AccessKey	—	Tencent Cloud AK/SK
role_arn / identity_provider_name / access_token	RoleBasedSSO	—	SSO role + token (Manual)
query_timeout_seconds	no	0 (unlimited)	Server-side query timeout
poll_interval_ms	no	500	Status polling interval
max_rows	no	10000	Rows per FetchResults batch
start_session_props	no	{}	Extra OpenSession configuration
use_arrow_result / arrow_batch_size	no	False / 10000	Kyuubi Arrow result format (enhancement)

Security: never hard-code credentials. Pass secret_id / secret_key / access_token via environment variables. HTTPS mode verifies server certificates with the system CA store.

Authentication

AccessKey (binary)

The SASL username encodes engine&rg&AKSK[&catalog[&database]] (AKSK is the wire tag for the AccessKey auth mode) and the password encodes secretId&secretKey; the SDK assembles these automatically.

RoleBasedSSO

Pass a pre-obtained access_token, or resolve one with OAuthManager:

from dlc_sql.auth import OAuthManager, RoleBasedSSOAuth

# M2M (client_credentials) — CI/CD
mgr = OAuthManager(
    token_url="https://login.microsoftonline.com/<tenant>/oauth2/v2.0/token",
    client_id="<client_id>",
    client_secret="<client_secret>",
    scopes=["api://dlc/.default"],
)
token = mgr.get_access_token()

# U2M (authorization_code + PKCE, browser) — local dev
mgr = OAuthManager(
    token_url="https://login.microsoftonline.com/<tenant>/oauth2/v2.0/token",
    authorize_url="https://login.microsoftonline.com/<tenant>/oauth2/v2.0/authorize",
    client_id="<client_id>",
    scopes=["api://dlc/.default", "offline_access"],
    callback_port=8030,
    persistence_path="~/.dlc/token_cache.json",
)
token = mgr.get_access_token()

Then connect with auth_mode="RoleBasedSSO", role_arn=..., identity_provider_name=..., access_token=token.

Architecture

dlc_sql.connect(...)
   └── Connection ── open_session ──┐
         └── Cursor ── execute / fetch
               │
   ┌───────────┴───────────────┐
   │  ThriftTransport          │  binary | http | https
   │   └ send_request(bytes)   │
   └───────────┬───────────────┘
               │
   thrift_api/codec.create_client(transport)
        → TCLIService.Client over TBinaryProtocol
               │
   dlc_sql/thrift_api/TCLIService/   (vendored from PyHive)

The codec bridges our custom ThriftTransport (which carries raw TBinaryProtocol request/response bytes — handling the custom SASL PLAIN handshake and framing, or HTTP POST) onto a Thrift TTransport so the vendored TCLIService.Client drives the RPCs directly. This avoids the thrift_sasl dependency (DLC's SASL is a custom variant) while keeping the full Thrift message envelope correct.

Module layout

dlc_sql/
├── __init__.py          # DBAPI 2.0 entry point (apilevel, connect, exceptions)
├── connection.py        # Connection + connect()
├── cursor.py            # Cursor (execute / poll / fetch)
├── session.py           # OpenSession/CloseSession + config assembly
├── exceptions.py        # PEP 249 + DLC-specific exceptions
├── types.py             # options, type map, columnar result + schema parsing
├── auth/
│   ├── base.py          # AuthProvider ABC
│   ├── access_key.py    # AccessKeyAuth (AK/SK)
│   ├── role_sso.py      # RoleBasedSSOAuth (Manual token)
│   ├── oauth.py         # OAuthManager (M2M + U2M/PKCE)
│   └── __init__.py      # factory + assemble_sasl_credentials
├── transport/
│   ├── base.py          # ThriftTransport ABC
│   ├── binary.py        # BinaryTransport (SASL + framed)
│   ├── http.py          # HttpTransport (HTTP/HTTPS + X-DLC-* headers)
│   ├── factory.py       # create_transport()
│   └── __init__.py
└── thrift_api/
    ├── codec.py         # TTransport adapter → TCLIService.Client
    └── TCLIService/     # vendored from PyHive (HiveServer2 IDL)

DBAPI 2.0 conformance

• apilevel = "2.0", paramstyle = "pyformat", threadsafety = 1.
• Connection: cursor(), commit() / rollback() (no-ops, DLC has no transactions), close(), context-manager support.
• Cursor: execute(), executemany(), fetchone(), fetchmany(), fetchall(), cancel(), close(), description, rowcount (-1, DLC does not report affected rows), arraysize, context-manager support.
• Exceptions: Error, Warning, InterfaceError, DatabaseError, InternalError, OperationalError, ProgrammingError, IntegrityError, DataError, NotSupportedError (+ DlcSqlConnectionError, DlcSqlAuthError, DlcSqlOperationError subclasses).

Relationship to dbt-dlc

dbt-dlc extends dbt-spark, reusing all Spark SQL macros and overriding only the connection layer to use dlc-sql-python. See the companion spec in docs/specs/ for the dbt-dlc design.

License

Apache License 2.0.