prizm-adls-scanner

Prizm ADLS Gen2 data quality scanner for collecting operational, profile, structural, and business metrics from files in Azure Data Lake Storage, and sending the results to Prizm.

Location: prizm-cli/prizm-adls-scanner (under Server root)

Key Design Principles:

• ✅ Scans ADLS Gen2 only — no S3 support (use a different connector for S3 sources)
• ✅ Works with or without an active Spark session — falls back to Azure SDK-only reads when no Spark session is available
• ✅ Safe to use in CI/CD, cron, GitHub Actions, Jenkins, etc.
• ✅ Explicit flags, no magic, deterministic behavior

Not included here: the Databricks notebook path (databricks/notebook_runner.py, notebooks/prizm_scanner.py) that runs on a Spark cluster notebook to build the Iceberg table and collect metrics — that's a separate deployment and lives outside this CLI package.

Installation

From Wheel File

pip install dist/prizm_adls_scanner-*.whl

From Source (Development)

pip install -e .

For development, you can install the package in editable mode. See the Development section for more details.

Quick Start

1. Set Environment Variables (Optional)

You can pass credentials as flags, or export them and reference via your shell:

export DQLABS_ACCESS_TOKEN=prizm_xxx

2. Scan a File and Push Metrics to Prizm

prizm-adls-scanner \
  --access-token "$DQLABS_ACCESS_TOKEN" \
  --mcp-host https://mcp.dqlabs.ai \
  --asset-id "my-asset" \
  --source-id "my-source" \
  --storage-account mystorageaccount \
  --container mycontainer \
  --file-path path/to/file.csv

Command

prizm-adls-scanner

Read a file from ADLS Gen2, compute operational/profile/structural (and optionally business/duplicate) metrics, and push the results to Prizm.

Required Flags:

• --access-token: Prizm/DQLabs access token (or --api-key, deprecated)
• --asset-id: Prizm asset ID
• --source-id: Prizm source ID
• --storage-account: ADLS storage account name
• --container: ADLS container name
• --file-path: Path to the file within the container

Optional Flags:

• --mcp-host: MCP host URL (default: https://mcp.dqlabs.ai)
• --access-key: Storage account access key (alternative: Azure AD service principal via DQLABS_CLIENT_ID/DQLABS_CLIENT_SECRET/DQLABS_TENANT_ID environment variables)
• --max-columns: Maximum columns to profile (default: 300)
• --no-duplicates: Disable duplicate-row detection
• --webhook-url: Custom webhook URL for results
• --output: json (default) or pretty
• --quiet: Suppress progress output

Example:

prizm-adls-scanner \
  --access-token "$DQLABS_ACCESS_TOKEN" \
  --asset-id "sales-daily-csv" \
  --source-id "adls-prod" \
  --storage-account mystorageaccount \
  --container raw \
  --file-path sales/2026/08/sales.csv \
  --max-columns 200 \
  --output pretty

Metrics Collected:

• Operational: row count, column count, schema, file size, freshness
• Profile: completeness, uniqueness, character, space per column
• Structural: per-column technical statistics
• Duplicate detection (full-row or partition-based, unless --no-duplicates)
• Business/exception/lookup/behavioral/comparison metrics, when configured server-side for the asset

CI/CD Integration

GitHub Actions

- name: Push ADLS scan metrics to Prizm
  env:
    DQLABS_ACCESS_TOKEN: ${{ secrets.DQLABS_ACCESS_TOKEN }}
  run: |
    prizm-adls-scanner \
      --access-token "$DQLABS_ACCESS_TOKEN" \
      --asset-id "my-asset" \
      --source-id "my-source" \
      --storage-account mystorageaccount \
      --container mycontainer \
      --file-path path/to/file.csv

Jenkins

stage('Scan ADLS file with Prizm') {
    steps {
        sh '''
            prizm-adls-scanner \
              --access-token "${DQLABS_ACCESS_TOKEN}" \
              --asset-id "my-asset" \
              --source-id "my-source" \
              --storage-account mystorageaccount \
              --container mycontainer \
              --file-path path/to/file.csv
        '''
    }
}

Development

Development Setup

pip install -e ".[dev,test]"

Running Tests

make test

Building Wheels

make build

For offline/local environments where build dependencies are already installed:

make build-local

This outputs a wheel to dist/.

Clean Build Artifacts

make clean

This removes build/ and dist/ directories, egg-info, and Python cache files.

Security

• Token-based authentication via --access-token or environment variable
• Tokens never logged or printed
• TLS enforced for all API calls

License

MIT License

Support

For issues and questions, please visit: https://github.com/DQLabs-Inc/prizm-cli/issues