dscribe-dq

Run dScribe data quality rules against your Databricks or MSSQL databases and write the results back to dScribe — all in one function call.

For library internals, architecture, and contributing, see DEVELOPMENT.md.

Prerequisites

• A dScribe account with at least one asset that has data quality rules defined in its ODCS spec
• Your dScribe API key (Settings → API keys in the dScribe UI)
• The asset UUID you want to validate
• Access to the database the rules target (Databricks or MSSQL)

Installation

pip install dscribe-dq

Walkthrough

1. Find your asset ID and API key

In the dScribe UI, open the asset you want to validate. The asset ID is the UUID in the URL:

https://app.dscribe.cloud/catalog/assets/337eaa9e-47ed-4b37-a124-050d4932a520
                                                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                                                  this is your asset_id

Your API key is under Settings → API keys.

2. Define your data quality rules in dScribe

Rules are defined in your asset's ODCS spec under schema[].quality (dataset-level) or schema[].properties[].quality (column-level). Each rule must have a sourceId in its customProperties that matches a server entry in the servers block.

The library maps ODCS metrics to Great Expectations checks automatically. Supported metrics:

3. Connect to Databricks

The library authenticates using an Azure AD service principal (OAuth M2M). You need:

• The Databricks workspace hostname
• An Azure AD application with client ID and secret
• The tenant ID of your Azure AD directory
• The HTTP path of your SQL warehouse

Set environment variables (recommended for CI/CD and notebooks):

DATABRICKS_HOST=adb-858283489583940.0.azuredatabricks.net
DATABRICKS_CLIENT_ID=f0d8e3e9-b910-402f-bc21-9bbefc5432ef
DATABRICKS_CLIENT_SECRET=<your-secret>
DATABRICKS_TENANT_ID=307d700a-c1eb-4e24-bb12-56f114c2d470
DATABRICKS_HTTP_PATH=/sql/1.0/warehouses/abc123def456
DATABRICKS_CATALOG=hive_metastore        # optional
DATABRICKS_SCHEMA=default                # optional
DSCRIBE_API_KEY=<your-api-key>
DSCRIBE_ASSET_ID=337eaa9e-47ed-4b37-a124-050d4932a520

Then run with no arguments:

from dscribe_dq import run_validation

results = run_validation()

Or pass credentials in code:

from dscribe_dq import run_validation

results = run_validation(
    dscribe_key="<your-api-key>",
    asset_id="337eaa9e-47ed-4b37-a124-050d4932a520",
    source_configs={
        # key must match the server id in the ODCS servers block
        "09bcc0f9-9d21-460d-9cb9-942b00e360bf": {
            "host": "adb-858283489583940.0.azuredatabricks.net",
            "client_id": "f0d8e3e9-b910-402f-bc21-9bbefc5432ef",
            "client_secret": "<your-secret>",
            "tenant_id": "307d700a-c1eb-4e24-bb12-56f114c2d470",
            "http_path": "/sql/1.0/warehouses/abc123def456",
            "catalog": "hive_metastore",
            "schema": "default",
        }
    },
)

The http_path can be found in the Databricks UI under SQL Warehouses → your warehouse → Connection details.

4. Connect to MSSQL

Two authentication modes are supported: SQL Server (username + password) and Entra ID (service principal).

SQL Server authentication:

results = run_validation(
    dscribe_key="<your-api-key>",
    asset_id="337eaa9e-47ed-4b37-a124-050d4932a520",
    source_configs={
        "4d0c53c5-c383-4fb0-95ea-fb7421dac8c0": {
            "host": "your-server.database.windows.net",
            "database": "your-db",
            "schema": "SalesLT",
            "table": "Product",
            "authentication": "SQL Server",
            "username": "your-user",
            "password": "your-password",
        }
    },
)

Entra ID (service principal) authentication:

results = run_validation(
    dscribe_key="<your-api-key>",
    asset_id="337eaa9e-47ed-4b37-a124-050d4932a520",
    source_configs={
        "4d0c53c5-c383-4fb0-95ea-fb7421dac8c0": {
            "host": "your-server.database.windows.net",
            "database": "your-db",
            "schema": "SalesLT",
            "table": "Product",
            "authentication": "Entra ID",
            "tenant_id": "<tenant-id>",
            "client_id": "<client-id>",
            "client_secret": "<client-secret>",
        }
    },
)

5. Run against multiple sources in one call

If your asset has rules targeting both Databricks and MSSQL, pass both in source_configs:

results = run_validation(
    dscribe_key="<your-api-key>",
    asset_id="337eaa9e-47ed-4b37-a124-050d4932a520",
    source_configs={
        "4d0c53c5-c383-4fb0-95ea-fb7421dac8c0": {
            # MSSQL config ...
        },
        "09bcc0f9-9d21-460d-9cb9-942b00e360bf": {
            # Databricks config ...
        },
    },
)

The library groups rules by source, runs each connector independently, and merges all results.

6. Understand the results

run_validation returns a list of dicts, one per rule:

[
    {
        "rule_id": "b9426493-3eb6-4ad5-872a-41a0533adac1",
        "expectation": "expect_column_values_to_not_be_null",
        "column": "ListPrice",
        "success": True,
        "result": { ... }   # full Great Expectations result dict
    },
    ...
]

Results are also written back to dScribe automatically so the asset's quality status updates in the UI. Each rule gets a lastCheckStatus (passed or failed) and lastCheckTimestamp added to its customProperties.

7. Optional: dry run and logging

Dry run — validate locally without posting results back to dScribe:

results = run_validation(..., dry_run=True)

Log level — reduce output to results only:

results = run_validation(..., log_level="RESULT")

Log levels in order: DEBUG → INFO → RESULT → WARNING → ERROR. Use RESULT for scheduled jobs to see only pass/fail lines and the summary table.

Save failed rows to CSV — collect the actual failing rows for each failed rule:

results = run_validation(
    ...,
    failed_rows_mode="csv",
    failed_rows_dir="./reports/failed",
)

Files are written to <failed_rows_dir>/<asset_id>/<dataset>_failed_rows_<timestamp>.csv.

Environment variable reference