dscribe-dq

Run dScribe data quality rules against your Databricks or MSSQL databases and write the results back to dScribe.

For library internals, architecture, and contributing, see DEVELOPMENT.md.

Prerequisites

• A dScribe account with at least one asset that has data quality rules defined in its ODCS spec
• Your dScribe API key (Settings → API keys in the dScribe UI)
• The asset UUID you want to validate
• Access to the database the rules target (Databricks or MSSQL)

Installation

pip install dscribe-dq

How it works

Initialize DScribeDQ once with your credentials, then call run_validation with a list of post-processors. Post-processors are small pipeline steps that run in order after validation — writing results back to dScribe, uploading failed-row CSVs, generating reports, etc.

DScribeDQ(credentials) → dq.run_validation(connector_config, postprocessors=[...])

The only post-processor built into the SDK is write_back_to_dscribe, which posts pass/fail results to dScribe so the asset's quality status updates in the UI.

Quickstart

1. Find your asset ID and API key

In the dScribe UI, open the asset you want to validate. The asset ID is the UUID in the URL:

https://app.dscribe.cloud/catalog/assets/337eaa9e-47ed-4b37-a124-050d4932a520
                                                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Your API key is under Settings → API keys.

2. Run validation and write back to dScribe

from dscribe_dq import DScribeDQ

dq = DScribeDQ(
    api_key="<your-api-key>",
    base_url="https://app.dscribe.cloud/catalog/api",
    asset_id="337eaa9e-47ed-4b37-a124-050d4932a520",
)

ctx = dq.run_validation(
    source_configs={
        # key must match the server id in the ODCS servers block
        "09bcc0f9-9d21-460d-9cb9-942b00e360bf": {
            "type": "databricks",
            "host": "adb-858283489583940.0.azuredatabricks.net",
            "client_id": "<client-id>",
            "client_secret": "<client-secret>",
            "tenant_id": "<tenant-id>",
            "http_path": "/sql/1.0/warehouses/abc123def456",
            "catalog": "hive_metastore",   # optional
            "schema": "default",           # optional
        }
    },
    postprocessors=[dq.write_back_to_dscribe()],
)

Each rule in dScribe gets a lastCheckStatus (passed or failed), lastCheckTimestamp, and failure metrics added to its customProperties after the pipeline runs.

Databricks notebook

%pip install dscribe-dq

from dscribe_dq import DScribeDQ

dq = DScribeDQ(
    api_key=dbutils.secrets.get(scope="dscribe-dq", key="DSCRIBE_API_KEY"),
    base_url="https://app.dscribe.cloud/catalog/api",
    asset_id="<your-asset-uuid>",
)

ctx = dq.run_validation(
    source_configs={
        "<server-id>": {
            "type": "databricks",
            "host": spark.conf.get("spark.databricks.workspaceUrl"),
            "client_id": dbutils.secrets.get(scope="dscribe-dq", key="CLIENT_ID"),
            "client_secret": dbutils.secrets.get(scope="dscribe-dq", key="CLIENT_SECRET"),
            "tenant_id": dbutils.secrets.get(scope="dscribe-dq", key="TENANT_ID"),
            "http_path": "/sql/1.0/warehouses/<warehouse-id>",
        }
    },
    postprocessors=[dq.write_back_to_dscribe()],
)

The http_path can be found in the Databricks UI under SQL Warehouses → your warehouse → Connection details.

Connecting to MSSQL

SQL Server authentication:

source_configs={
    "<server-id>": {
        "type": "sqlserver",
        "host": "your-server.database.windows.net",
        "database": "your-db",
        "schema": "SalesLT",
        "authentication": "SQL Server",
        "username": "your-user",
        "password": "your-password",
    }
}

Entra ID (service principal) authentication:

source_configs={
    "<server-id>": {
        "type": "sqlserver",
        "host": "your-server.database.windows.net",
        "database": "your-db",
        "schema": "SalesLT",
        "authentication": "Entra ID",
        "tenant_id": "<tenant-id>",
        "client_id": "<client-id>",
        "client_secret": "<client-secret>",
    }
}

Multiple sources in one call

If your asset has rules targeting both Databricks and MSSQL, pass both in source_configs. Rules are automatically grouped by source and run independently:

source_configs={
    "<mssql-server-id>": {"type": "sqlserver", ...},
    "<databricks-server-id>": {"type": "databricks", ...},
}

CI/CD usage (env vars)

For automated pipelines, set env vars and call the module-level run_validation directly:

DSCRIBE_API_KEY=...
DSCRIBE_ASSET_ID=...
DSCRIBE_BASE_URL=...
DATABRICKS_HOST=...
DATABRICKS_CLIENT_ID=...
DATABRICKS_CLIENT_SECRET=...
DATABRICKS_TENANT_ID=...
DATABRICKS_HTTP_PATH=...

from dscribe_dq import run_validation

ctx = run_validation()  # reads all config from env vars

Options

DScribeDQ.__init__

DScribeDQ.run_validation

Supported ODCS metrics

Environment variable reference