prizm-adls-scanner

Python package that runs Prizm Exception, Lookup, Comparison, Materialize, and Behavioral jobs inside a Databricks Spark cluster.

Location: prizm-cli/prizm-adls-scanner (under Server root)

What it is:

• ✅ A library imported inside a Databricks notebook or job task — it does not read files or connect to storage itself; it operates on an already-active spark session
• ✅ Runtime deps are just requests and sqlglot — pyspark is intentionally not a dependency (Databricks already provides Spark on the cluster; installing PyPI pyspark there would conflict with it)
• ✅ pyspark is imported lazily inside each runner module, so pip install/import prizm_adls_scanner succeeds even outside Databricks (useful for local dev of non-Spark code paths)

What it is not:

• ❌ Not a CLI — no console script, no argparse/click entry point
• ❌ Not a file scanner/connector — it has no ADLS/S3 I/O logic; it assumes data is already loaded into Spark by the notebook/job that calls it
• ❌ Not the metrics-collection scanner — that's a different, unrelated concern

Compiled Modules

databricks/comparison_driver.py, databricks/lookup_driver.py, and databricks/exception_runner.py — the comparison/reconciliation, lookup-matching, and exception-detection engines — ship as compiled Cython extensions (.so), not plain Python. Everything else (databricks/__init__.py, _runtime.py, exception_metadata_runner.py, materialize_runner.py, side_snapshot_runner.py, the thin *_runner.py wrappers) stays plain Python.

Consequences:

• The published wheels are manylinux only, one per Python version (cp39–cp313) — built via cibuildwheel, see [tool.cibuildwheel] in pyproject.toml. They will not install on non-Linux platforms or Python versions outside that range. There is no sdist (an sdist is source-by-definition, which would defeat the point).
• Building it yourself (make build, or an editable pip install -e .) requires a C compiler and Cython — both pulled in automatically as build-time dependencies, but you need a working cc/gcc on your machine. An editable install does not exercise the compiled build — it just points back at the source directory, so the protected modules stay fully readable in-place during local development. Protection only applies to what actually gets published.

Install

Databricks notebook

%pip install prizm-adls-scanner
dbutils.library.restartPython()

Cluster / job library (recommended for production)

Attach the published wheel as a cluster or job library so tasks don't need %pip on every run.

From a built wheel

make build
pip install dist/prizm_adls_scanner-*.whl

Note: pyspark is not a package dependency. Databricks already provides Spark. Do not install PyPI pyspark on the cluster.

Databricks notebook usage

Use a thin notebook; all job logic lives in the package:

%pip install prizm-adls-scanner
dbutils.library.restartPython()

import json
from prizm_adls_scanner.databricks import run_metric_notebook_job

WIDGET_NAMES = [
    "job_config",
    "storage_account",
    "access_key",
    "client_id",
    "client_secret",
    "tenant_id",
    "uc_catalog",
    "uc_schema",
    "exception_url",
    "run_id",
    "file_batch",
    "job_type",
    "connector",
    "container",
    "asset_id",
    "source_id",
]

for name in WIDGET_NAMES:
    try:
        dbutils.widgets.text(name, "")
    except Exception:
        pass

widgets = {}
for name in WIDGET_NAMES:
    try:
        widgets[name] = dbutils.widgets.get(name)
    except Exception:
        widgets[name] = ""

result = run_metric_notebook_job(
    spark=spark,
    dbutils=dbutils,
    widgets=widgets,
)
dbutils.notebook.exit(json.dumps(result, default=str))

A reference notebook source also ships in the repo as prizm_adls_scanner/databricks/notebook/prizm_iceberg_notebook.py.

API

from prizm_adls_scanner.databricks import (
    run_metric_notebook_job,   # routes by widgets["job_type"] to the right runner
    run_exception_job,
    run_exception_notebook_job,
    run_lookup_job,
    run_comparison_job,
    run_materialize_job,
)
# also available directly off the top-level package (lazy import):
from prizm_adls_scanner import run_exception_job

Every runner returns a dict with at least status (success / failed). On failure, includes error / errors.

run_metric_notebook_job routes by widgets["job_type"]:

• EXCEPTION / EXCEPTION_METADATA → exception runner
• LOOKUP → lookup runner
• COMPARISON → comparison runner
• BEHAVIORAL → side-snapshot runner
• MATERIALIZE_ICEBERG / DELETE (or is_delete_action, or no job_type) → materialize runner (ADLS → Unity Catalog Delta UniForm)

Development

Development Setup

pip install -e ".[dev,test]"

Running Tests

Most of the test suite runs without pyspark installed (package-surface/import checks); a couple of tests are skipped automatically if pyspark isn't present.

make test

Building Wheels

make build

For offline/local environments where build dependencies are already installed:

make build-local

Clean Build Artifacts

make clean

Package Layout

prizm_adls_scanner/
  __init__.py                       # lazy top-level re-exports
  databricks/
    __init__.py                     # run_exception_job, run_lookup_job, run_comparison_job,
                                     # run_materialize_job, run_metric_notebook_job
    _runtime.py                     # shared runtime helpers
    exception_runner.py             # EXCEPTION / EXCEPTION_METADATA runtime
    exception_notebook.py           # reference notebook source
    exception_metadata_runner.py    # exception metadata runtime
    lookup_driver.py / lookup_runner.py         # LOOKUP runtime
    comparison_driver.py / comparison_runner.py # COMPARISON runtime
    materialize_runner.py           # MATERIALIZE_ICEBERG / DELETE runtime
    side_snapshot_runner.py         # BEHAVIORAL runtime
    catalog/                        # ADLS_TECHNICAL_CATALOG runtime
    notebook/prizm_iceberg_notebook.py  # reference Iceberg notebook source

License

MIT License

Support

For issues and questions, please visit: https://github.com/DQLabs-Inc/prizm-cli/issues