dscribe-dq

Run dScribe data quality rules against your Databricks or MSSQL databases and write the results back to dScribe.

For library internals, architecture, and contributing, see DEVELOPMENT.md.

Prerequisites

• A dScribe account with at least one asset that has data quality rules defined in its ODCS spec
• Your dScribe API key (Settings → API keys in the dScribe UI)
• The asset UUID you want to validate
• Access to the database the rules target (Databricks or MSSQL)

Installation

pip install dscribe-dq

How it works

Initialize DScribeDQ once with your credentials, then call run_validation with a list of post-processors. Post-processors are small pipeline steps that run in order after validation — writing results back to dScribe, uploading failed-row CSVs, generating reports, etc.

DScribeDQ(credentials) → dq.run_validation(connector_config, postprocessors=[...])

The only post-processor built into the SDK is write_back_to_dscribe, which posts pass/fail results to dScribe so the asset's quality status updates in the UI.

Quickstart

1. Find your asset ID and API key

In the dScribe UI, open the asset you want to validate. The asset ID is the UUID in the URL:

https://app.dscribe.cloud/catalog/assets/337eaa9e-47ed-4b37-a124-050d4932a520
                                                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Your API key is under Settings → API keys.

2. Run validation and write back to dScribe

from dscribe_dq import DScribeDQ

dq = DScribeDQ(
    api_key="<your-api-key>",
    base_url="https://app.dscribe.cloud/catalog/api",
    asset_id="337eaa9e-47ed-4b37-a124-050d4932a520",
)

ctx = dq.run_validation(
    source_configs={
        # key must match the server id in the ODCS servers block
        "09bcc0f9-9d21-460d-9cb9-942b00e360bf": {
            "type": "databricks",
            "host": "adb-858283489583940.0.azuredatabricks.net",
            "client_id": "<client-id>",
            "client_secret": "<client-secret>",
            "tenant_id": "<tenant-id>",
            "http_path": "/sql/1.0/warehouses/abc123def456",
            "catalog": "hive_metastore",   # optional
            "schema": "default",           # optional
        }
    },
    postprocessors=[dq.write_back_to_dscribe()],
)

Each rule in dScribe gets a lastCheckStatus (passed or failed), lastCheckTimestamp, and failure metrics added to its customProperties after the pipeline runs.

Databricks notebook

%pip install dscribe-dq

from dscribe_dq import DScribeDQ

dq = DScribeDQ(
    api_key=dbutils.secrets.get(scope="dscribe-dq", key="DSCRIBE_API_KEY"),
    base_url="https://app.dscribe.cloud/catalog/api",
    asset_id="<your-asset-uuid>",
)

ctx = dq.run_validation(
    source_configs={
        "<server-id>": {
            "type": "databricks",
            "host": spark.conf.get("spark.databricks.workspaceUrl"),
            "client_id": dbutils.secrets.get(scope="dscribe-dq", key="CLIENT_ID"),
            "client_secret": dbutils.secrets.get(scope="dscribe-dq", key="CLIENT_SECRET"),
            "tenant_id": dbutils.secrets.get(scope="dscribe-dq", key="TENANT_ID"),
            "http_path": "/sql/1.0/warehouses/<warehouse-id>",
        }
    },
    postprocessors=[dq.write_back_to_dscribe()],
)

The http_path can be found in the Databricks UI under SQL Warehouses → your warehouse → Connection details.

Connecting to MSSQL

SQL Server authentication:

source_configs={
    "<server-id>": {
        "type": "sqlserver",
        "host": "your-server.database.windows.net",
        "database": "your-db",
        "schema": "SalesLT",
        "authentication": "SQL Server",
        "username": "your-user",
        "password": "your-password",
    }
}

Entra ID (service principal) authentication:

source_configs={
    "<server-id>": {
        "type": "sqlserver",
        "host": "your-server.database.windows.net",
        "database": "your-db",
        "schema": "SalesLT",
        "authentication": "Entra ID",
        "tenant_id": "<tenant-id>",
        "client_id": "<client-id>",
        "client_secret": "<client-secret>",
    }
}

Multiple sources in one call

If your asset has rules targeting both Databricks and MSSQL, pass both in source_configs. Rules are automatically grouped by source and run independently:

source_configs={
    "<mssql-server-id>": {"type": "sqlserver", ...},
    "<databricks-server-id>": {"type": "databricks", ...},
}

CI/CD usage (env vars)

For automated pipelines, set env vars and call the module-level run_validation directly:

DSCRIBE_API_KEY=...
DSCRIBE_ASSET_ID=...
DSCRIBE_BASE_URL=...
DATABRICKS_HOST=...
DATABRICKS_CLIENT_ID=...
DATABRICKS_CLIENT_SECRET=...
DATABRICKS_TENANT_ID=...
DATABRICKS_HTTP_PATH=...

from dscribe_dq import run_validation

ctx = run_validation()  # reads all config from env vars

Post-processors

Post-processors provide a way to perform additional processing after validation has completed. They receive the validation context (ctx) and can inspect or modify the results before the run finishes.

Pass them to run_validation in the order you want them to run:

ctx = dq.run_validation(
    source_configs={...},
    postprocessors=[
        dq.write_back_to_dscribe(),
    ],
)

Built-in: write_back_to_dscribe

write_back_to_dscribe is the built-in post-processor. It sends each rule's validation result back to dScribe so the latest status is visible in the dScribe UI.

postprocessors=[
    dq.write_back_to_dscribe(),
]

For each rule, it sends:

• lastCheckStatus
• lastCheckTimestamp
• lastCheckDetails — a structured object with success_percent, unexpected_percent, unexpected_count, and element_count, generated automatically from ctx.results. You do not need to set these yourself.

lastCheckDetails can also carry two optional links, but only if an earlier post-processor sets them first:

If they are not set, write_back_to_dscribe still writes the validation results normally — add a custom post-processor before it that uploads the relevant files and sets these values, as shown in Custom post-processors.

The ctx object

Every post-processor receives the same DQContext.

The most useful fields are:

Each entry in ctx.results looks like this:

{
    "rule_id": "675dbbcd-ee5c-45bc-85b7-34514beaea73",
    "expectation": "unexpected_rows_expectation",
    "column": "NATIONALITY_ID",
    "success": False,
    "result": {...},
    "failed_rows": [
        {"ID": "...", "NATIONALITY_ID": None},
        ...
    ],
}

For most custom post-processors, these fields are enough:

• rule_id — ID of the dScribe rule
• column — column checked by the rule, or "table" for table-level rules
• success — whether the rule passed
• failed_rows — collected rows that failed the rule

result contains the raw Great Expectations validation result and is available when you need more detailed validation information.

When profiling is enabled with enable_profiling=True, profiling data is available through ctx.profile:

{
    "__table__": {
        "row_count": 2672186
    },
    "NATIONALITY_ID": {
        "type": "categorical",
        "null_count": 12,
        "distinct_count": 4,
        "value_counts": [...]
    },
    "AGE": {
        "type": "numeric",
        "null_count": 0,
        "distinct_count": 90,
        "mean": 41.2,
        "min": 0,
        "max": 120,
        "quantiles": [0, 22, 41, 60, 120]
    },
}

Custom post-processors

A custom post-processor is simply a function that accepts ctx.

For example, to log the failing rows:

def log_failed_rows(ctx):
    for res in ctx.results:
        if not res["success"]:
            for row in res["failed_rows"]:
                print(row)

This is also how you add your own reporting or storage. For example, a post-processor that uploads failing rows to your storage provider and sets res["csv_url"], so write_back_to_dscribe picks it up as failed_rows_url:

def upload_failed_rows(ctx):
    for res in ctx.results:
        if not res["success"] and res["failed_rows"]:
            res["csv_url"] = my_storage.upload_csv(res["rule_id"], res["failed_rows"])

Add both to postprocessors, before write_back_to_dscribe:

ctx = dq.run_validation(
    source_configs={...},
    postprocessors=[
        log_failed_rows,
        upload_failed_rows,
        dq.write_back_to_dscribe(),
    ],
)

Post-processors run in the order they are listed, so write_back_to_dscribe must come last to see csv_url (or ctx.report_url) once an earlier post-processor sets it.

Options

DScribeDQ.__init__

DScribeDQ.run_validation

Known limitations

• max_failed_rows does not apply to all rule types. It is respected for all built-in library rules (nullValues, missingValues, duplicateValues — including compound/multi-column rules — and invalidValues when defined using allowed or excluded values) and for SQL rules. It is not respected for regex-based invalidValues rules (expect_column_values_to_match_regex and expect_column_values_to_not_match_regex) or for expectations defined using a custom Great Expectations YAML block — for those, the number of returned failing rows is controlled by Great Expectations and capped at 200 regardless of max_failed_rows. This limit comes from Great Expectations' internal MAX_RESULT_RECORDS setting and cannot be configured by dscribe-dq.

Supported ODCS metrics

Environment variable reference