dscribe-dq 0.0.4

Automatically generated by Nx.

dscribe-dq

Run dScribe data quality rules against your Databricks or MSSQL databases and write the results back to dScribe — all in one function call.

For library internals, architecture, and contributing, see DEVELOPMENT.md.

Prerequisites

• A dScribe account with at least one asset that has data quality rules defined in its ODCS spec
• Your dScribe API key (Settings → API keys in the dScribe UI)
• The asset UUID you want to validate
• Access to the database the rules target (Databricks or MSSQL)

Installation

pip install dscribe-dq

Walkthrough

1. Find your asset ID and API key

In the dScribe UI, open the asset you want to validate. The asset ID is the UUID in the URL:

https://app.dscribe.cloud/catalog/assets/337eaa9e-47ed-4b37-a124-050d4932a520
                                                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                                                  this is your asset_id

Your API key is under Settings → API keys.

2. Define your data quality rules in dScribe

Rules are defined in your asset's ODCS spec under schema[].quality (dataset-level) or schema[].properties[].quality (column-level). Each rule must have a sourceId in its customProperties that matches a server entry in the servers block.

The library maps ODCS metrics to Great Expectations checks automatically. Supported metrics:

ODCS metric	What it checks
rowCount	Row count within expected bounds
nullValues	No NULL values in a column
missingValues	No missing/empty values in a column
duplicateValues	All values in a column (or column set) are unique
invalidValues	Values match an allowed list or regex pattern

3. Connect to Databricks

The library authenticates using an Azure AD service principal (OAuth M2M). You need:

• The Databricks workspace hostname
• An Azure AD application with client ID and secret
• The tenant ID of your Azure AD directory
• The HTTP path of your SQL warehouse

Set environment variables (recommended for CI/CD and notebooks):

DATABRICKS_HOST=adb-858283489583940.0.azuredatabricks.net
DATABRICKS_CLIENT_ID=f0d8e3e9-b910-402f-bc21-9bbefc5432ef
DATABRICKS_CLIENT_SECRET=<your-secret>
DATABRICKS_TENANT_ID=307d700a-c1eb-4e24-bb12-56f114c2d470
DATABRICKS_HTTP_PATH=/sql/1.0/warehouses/abc123def456
DATABRICKS_CATALOG=hive_metastore        # optional
DATABRICKS_SCHEMA=default                # optional
DSCRIBE_API_KEY=<your-api-key>
DSCRIBE_ASSET_ID=337eaa9e-47ed-4b37-a124-050d4932a520

Then run with no arguments:

from dscribe_dq import run_validation

results = run_validation()

Or pass credentials in code:

from dscribe_dq import run_validation

results = run_validation(
    dscribe_key="<your-api-key>",
    asset_id="337eaa9e-47ed-4b37-a124-050d4932a520",
    source_configs={
        # key must match the server id in the ODCS servers block
        "09bcc0f9-9d21-460d-9cb9-942b00e360bf": {
            "host": "adb-858283489583940.0.azuredatabricks.net",
            "client_id": "f0d8e3e9-b910-402f-bc21-9bbefc5432ef",
            "client_secret": "<your-secret>",
            "tenant_id": "307d700a-c1eb-4e24-bb12-56f114c2d470",
            "http_path": "/sql/1.0/warehouses/abc123def456",
            "catalog": "hive_metastore",
            "schema": "default",
        }
    },
)

The http_path can be found in the Databricks UI under SQL Warehouses → your warehouse → Connection details.

4. Connect to MSSQL

Two authentication modes are supported: SQL Server (username + password) and Entra ID (service principal).

SQL Server authentication:

results = run_validation(
    dscribe_key="<your-api-key>",
    asset_id="337eaa9e-47ed-4b37-a124-050d4932a520",
    source_configs={
        "4d0c53c5-c383-4fb0-95ea-fb7421dac8c0": {
            "host": "your-server.database.windows.net",
            "database": "your-db",
            "schema": "SalesLT",
            "table": "Product",
            "authentication": "SQL Server",
            "username": "your-user",
            "password": "your-password",
        }
    },
)

Entra ID (service principal) authentication:

results = run_validation(
    dscribe_key="<your-api-key>",
    asset_id="337eaa9e-47ed-4b37-a124-050d4932a520",
    source_configs={
        "4d0c53c5-c383-4fb0-95ea-fb7421dac8c0": {
            "host": "your-server.database.windows.net",
            "database": "your-db",
            "schema": "SalesLT",
            "table": "Product",
            "authentication": "Entra ID",
            "tenant_id": "<tenant-id>",
            "client_id": "<client-id>",
            "client_secret": "<client-secret>",
        }
    },
)

5. Run against multiple sources in one call

If your asset has rules targeting both Databricks and MSSQL, pass both in source_configs:

results = run_validation(
    dscribe_key="<your-api-key>",
    asset_id="337eaa9e-47ed-4b37-a124-050d4932a520",
    source_configs={
        "4d0c53c5-c383-4fb0-95ea-fb7421dac8c0": {
            # MSSQL config ...
        },
        "09bcc0f9-9d21-460d-9cb9-942b00e360bf": {
            # Databricks config ...
        },
    },
)

The library groups rules by source, runs each connector independently, and merges all results.

6. Understand the results

run_validation returns a list of dicts, one per rule:

[
    {
        "rule_id": "b9426493-3eb6-4ad5-872a-41a0533adac1",
        "expectation": "expect_column_values_to_not_be_null",
        "column": "ListPrice",
        "success": True,
        "result": { ... }   # full Great Expectations result dict
    },
    ...
]

Results are also written back to dScribe automatically so the asset's quality status updates in the UI. Each rule gets a lastCheckStatus (passed or failed) and lastCheckTimestamp added to its customProperties.

7. Optional: dry run and logging

Dry run — validate locally without posting results back to dScribe:

results = run_validation(..., dry_run=True)

Log level — reduce output to results only:

results = run_validation(..., log_level="RESULT")

Log levels in order: DEBUG → INFO → RESULT → WARNING → ERROR. Use RESULT for scheduled jobs to see only pass/fail lines and the summary table.

Save failed rows to CSV — collect the actual failing rows for each failed rule:

results = run_validation(
    ...,
    failed_rows_mode="csv",
    failed_rows_dir="./reports/failed",
)

Files are written to <failed_rows_dir>/<asset_id>/<dataset>_failed_rows_<timestamp>.csv.

Environment variable reference

Variable	Description	Default
DSCRIBE_API_KEY	dScribe API key	—
DSCRIBE_ASSET_ID	Asset UUID to validate	—
DSCRIBE_BASE_URL	dScribe API base URL	https://app.dscribe.cloud/catalog/api
DQ_DRY_RUN	Skip write-back when true	false
DQ_FAILED_ROWS_MODE	none or csv	none
DQ_FAILED_ROWS_DIR	Output directory for failed-rows CSVs	./reports/dscribe-dq/failed_rows
DATABRICKS_HOST	Databricks workspace hostname	—
DATABRICKS_CLIENT_ID	Azure AD service principal client ID	—
DATABRICKS_CLIENT_SECRET	Azure AD service principal client secret	—
DATABRICKS_TENANT_ID	Azure AD tenant ID	—
DATABRICKS_HTTP_PATH	SQL warehouse HTTP path	—
DATABRICKS_WAREHOUSE_ID	SQL warehouse ID (alternative to HTTP path)	—
DATABRICKS_CATALOG	Default Unity Catalog catalog name	—
DATABRICKS_SCHEMA	Default schema name	—
MSSQL_HOST	MSSQL server hostname	—
MSSQL_DATABASE	MSSQL database name	—
MSSQL_USER	SQL Server username	—
MSSQL_PASSWORD	SQL Server password	—
MSSQL_AUTH	SQL Server or Entra ID	SQL Server
MSSQL_TENANT_ID	Azure tenant ID (Entra ID auth only)	—
MSSQL_CLIENT_ID	Azure client ID (Entra ID auth only)	—
MSSQL_CLIENT_SECRET	Azure client secret (Entra ID auth only)	—