dqlabs-prizm 1.0.0

DQLabs Prizm - Data Quality Metrics Collection for Databricks and Cloud Storage

DQLabs Prizm - Data Quality Metrics Library

A comprehensive Python library for collecting operational, profile, and structural metrics from cloud storage (ADLS/S3) using Databricks Spark.

---

📋 Table of Contents

• Features
• Installation
• Quick Start
• Configuration
• Usage Examples
• Metrics Collected
• API Reference
• Error Handling
• Development
• License

---

✨ Features

• ✅ Operational Metrics: Row count, column count, schema, file size, freshness
• ✅ Profile Metrics: Completeness, Uniqueness, Character, Space per column
• ✅ Structural Metrics: Per-column technical statistics
• ✅ Duplicate Detection: Full-row or partition-based duplicate counting
• ✅ Multiple Storage: ADLS Gen2 and S3 support
• ✅ Databricks Native: Optimized for Databricks Spark (Serverless compatible)
• ✅ API Integration: Sends metrics to DQLabs webhook endpoint
• ✅ Webhook Support: Custom webhook URL for results
• ✅ Job Type Support: OPERATIONAL, PROFILE, STRUCTURAL, FULL, COMPUTE_METRIC

---

📦 Installation

Option 1: Install from PyPI (Recommended)

pip install dqlabs-prizm

Option 2: Install from Source

git clone https://github.com/dqlabs/prizm-python.git
cd prizm-python
pip install -e .

Option 3: Build locally

./build_package.sh
pip install dist/dqlabs_prizm-1.0.0-py3-none-any.whl

Databricks Notebook Installation

%pip install dqlabs-prizm
dbutils.library.restartPython()

Basic Usage

from dqlabs_prizm import PrizmScanner, PrizmConfig
from dqlabs_prizm.config.settings import JobType

# Create configuration
config = PrizmConfig(
    access_token="your_access_token",
    mcp_host="https://your-mcp-host.com",
    api_version="v1",
    asset_id="your_asset_id",
    source_id="your_source_id",
    qualified_name="your_file.csv",
    run_id="run_001",
    job_type=JobType.FULL,
    connector="adls",
    storage_account="your_storage_account",
    container="your_container",
    file_path="path/to/file.csv",
    access_key="your_access_key",
    extra_headers={"ngrok-skip-browser-warning": "true"}  # For ngrok tunnels
)

# Initialize scanner with Spark session
scanner = PrizmScanner(config, spark=spark, validate_token=True)

# Run the scan
result = scanner.scan()

# Access results
print(f"Row Count: {result['row_count']:,}")
print(f"Column Count: {result['column_count']}")
print(f"Duplicate Count: {result['duplicate_count']}")

🤝 Support

For support, please contact DQLabs support or open an issue on GitHub.

📝 Changelog

Version 1.0.0

• Initial release
• Support for ADLS and S3
• Operational, Profile, Structural metrics
• Databricks Spark integration
• COMPUTE_METRIC job type
• Duplicate detection
• Webhook support
• Extra headers support
• Serverless compatible

Built with ❤️ by DQLabs