dscribe-dq

Run dScribe data quality rules against your Databricks or MSSQL databases and write the results back to dScribe.

For library internals, architecture, and contributing, see DEVELOPMENT.md.

Prerequisites

• A dScribe account with at least one asset that has data quality rules defined in its ODCS spec
• Your dScribe API key (Settings → API keys in the dScribe UI)
• The asset UUID you want to validate
• Access to the database the rules target (Databricks or MSSQL)

Installation

pip install dscribe-dq

How it works

run_validation fetches your rules from dScribe, runs them with Great Expectations, and returns a DQContext. You then pipe that context through one or more post-processors — small steps that each do one thing (write back to dScribe, upload CSVs, generate reports). This keeps validation and reporting cleanly separated.

run_validation() → DQContext → [step1, step2, ...] → done

The only post-processor included in the SDK is write_back_to_dscribe, which posts pass/fail results back to dScribe so the asset's quality status updates in the UI. Additional post-processors (blob uploads, HTML reports) are available in the separate postprocessors package used by the dScribe runner.

Quickstart

1. Find your asset ID and API key

In the dScribe UI, open the asset you want to validate. The asset ID is the UUID in the URL:

https://app.dscribe.cloud/catalog/assets/337eaa9e-47ed-4b37-a124-050d4932a520
                                                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Your API key is under Settings → API keys.

2. Run validation and write back to dScribe

from dscribe_dq import run_validation, DScribeClient, write_back_to_dscribe

ctx = run_validation(
    dscribe_key="<your-api-key>",
    asset_id="337eaa9e-47ed-4b37-a124-050d4932a520",
    source_configs={
        # key must match the server id in the ODCS servers block
        "09bcc0f9-9d21-460d-9cb9-942b00e360bf": {
            "type": "databricks",
            "host": "adb-858283489583940.0.azuredatabricks.net",
            "client_id": "<client-id>",
            "client_secret": "<client-secret>",
            "tenant_id": "<tenant-id>",
            "http_path": "/sql/1.0/warehouses/abc123def456",
            "catalog": "hive_metastore",   # optional
            "schema": "default",           # optional
        }
    },
)

client = DScribeClient(api_key="<your-api-key>", base_url="<your-base-url>")

pipeline = [write_back_to_dscribe(client)]

for step in pipeline:
    step(ctx)

Each rule in dScribe gets a lastCheckStatus (passed or failed), lastCheckTimestamp, and failure metrics added to its customProperties after the pipeline runs.

Databricks notebook

%pip install dscribe-dq

from dscribe_dq import run_validation, DScribeClient, write_back_to_dscribe

DSCRIBE_API_KEY = dbutils.secrets.get(scope="dscribe-dq", key="DSCRIBE_API_KEY")
DSCRIBE_BASE_URL = "<your-base-url>"
ASSET_ID = "<your-asset-uuid>"

ctx = run_validation(
    dscribe_key=DSCRIBE_API_KEY,
    base_url=DSCRIBE_BASE_URL,
    asset_id=ASSET_ID,
    source_configs={
        "<server-id>": {
            "type": "databricks",
            "host": spark.conf.get("spark.databricks.workspaceUrl"),
            "client_id": dbutils.secrets.get(scope="dscribe-dq", key="CLIENT_ID"),
            "client_secret": dbutils.secrets.get(scope="dscribe-dq", key="CLIENT_SECRET"),
            "tenant_id": dbutils.secrets.get(scope="dscribe-dq", key="TENANT_ID"),
            "http_path": "/sql/1.0/warehouses/<warehouse-id>",
        }
    },
)

client = DScribeClient(api_key=DSCRIBE_API_KEY, base_url=DSCRIBE_BASE_URL)

pipeline = [write_back_to_dscribe(client)]

for step in pipeline:
    step(ctx)

The http_path can be found in the Databricks UI under SQL Warehouses → your warehouse → Connection details.

Connecting to MSSQL

SQL Server authentication:

source_configs={
    "<server-id>": {
        "type": "sqlserver",
        "host": "your-server.database.windows.net",
        "database": "your-db",
        "schema": "SalesLT",
        "authentication": "SQL Server",
        "username": "your-user",
        "password": "your-password",
    }
}

Entra ID (service principal) authentication:

source_configs={
    "<server-id>": {
        "type": "sqlserver",
        "host": "your-server.database.windows.net",
        "database": "your-db",
        "schema": "SalesLT",
        "authentication": "Entra ID",
        "tenant_id": "<tenant-id>",
        "client_id": "<client-id>",
        "client_secret": "<client-secret>",
    }
}

Multiple sources in one call

If your asset has rules targeting both Databricks and MSSQL, pass both in source_configs. Rules are automatically grouped by source and run independently:

source_configs={
    "<mssql-server-id>": { "type": "sqlserver", ... },
    "<databricks-server-id>": { "type": "databricks", ... },
}

Options

Supported ODCS metrics

Environment variable reference