dql-core 0.5.2

Framework-agnostic validation engine for Data Quality Language (DQL)

dql-core

Framework-agnostic validation engine for Data Quality Language (DQL).

Documentation | PyPI | GitHub

dql-core provides the abstract validation, cleaner, and executor framework that can be implemented for any Python framework (Django, Flask, FastAPI, SQLAlchemy, Pandas, etc.). It handles the business logic of DQL validation without being tied to any specific data access layer.

Installation

pip install dql-core

Quick Start

Creating a Custom Executor

from dql_core import ValidationExecutor
from dql_parser import parse_dql

class MyExecutor(ValidationExecutor):
    """Custom executor for your framework."""

    def get_records(self, model_name: str):
        # Return records from your data source
        return my_database.query(model_name).all()

    def filter_records(self, records, condition):
        # Apply filtering logic
        return [r for r in records if self.evaluate_condition(r, condition)]

    def count_records(self, records):
        return len(list(records))

    def get_field_value(self, record, field_name: str):
        # Get field value from your record type
        return getattr(record, field_name)

# Parse DQL and execute validation
dql_text = """
from Customer
expect column("email") to_not_be_null
expect column("age") to_be_between(18, 100)
"""

ast = parse_dql(dql_text)
executor = MyExecutor()
result = executor.execute(ast)

print(f"Validation passed: {result.overall_passed}")
print(f"Expectations: {result.total_expectations}")
print(f"Failed: {result.failed_expectations}")

Features

Abstract Validation Framework

• 6 Built-in Validators: to_be_null, to_not_be_null, to_match_pattern, to_be_between, to_be_in, to_be_unique
• Custom Validators: Extend Validator base class
• Validator Registry: Register validators for operators
• Framework-Agnostic: Works with any data source (Django ORM, SQLAlchemy, Pandas, raw dicts)

Abstract Executor

• Template Method Pattern: Implement 4 abstract methods, get full validation logic
• Multi-Model Support: Validate multiple models in one DQL file
• Rich Results: Detailed validation results with failure info
• Severity Levels: Support for critical, warning, info

Cleaner Framework (Stories 2.4-2.8)

Cleaners automatically remediate data quality issues when expectations fail.

• 8 Built-in Cleaners: String normalization, phone/date formatting, NULL handling
• Custom Cleaners: Build your own with @cleaner decorator
• Cleaner Chains: Execute multiple cleaners sequentially
• Transaction Safety: Automatic rollback on failure
• Audit Logging: Track all modifications
• Dry-Run Mode: Preview changes before applying

Quick Example:

from dql_core import normalize_email, CleanerChain, SafeCleanerExecutor

# Single cleaner
record = {'email': '  [email protected]  '}
cleaner = normalize_email('email')
result = cleaner(record, {})
print(record['email'])  # '[email protected]'

# Cleaner chain
chain = (CleanerChain()
    .add('trim_whitespace', 'email')
    .add('lowercase', 'email'))
result = chain.execute(record, {})

# Transaction safety
from dql_core import DictTransactionManager
manager = DictTransactionManager()
executor = SafeCleanerExecutor(manager)
result = executor.execute_cleaners(cleaners, record, {})
# Automatic rollback if any cleaner fails

Documentation:

• Tutorial - 5-minute quickstart
• Cleaner Catalog - All 8 built-in cleaners
• Custom Cleaners Guide - Build your own
• Best Practices - Performance and security
• Troubleshooting - Common issues
• Examples - Runnable code samples

External API Adapters

• Adapter Pattern: Create adapters for external APIs
• Rate Limiting: Built-in rate limiter
• Retry Logic: Exponential backoff retry utilities
• Factory Pattern: APIAdapterFactory for creating adapters

Built-in Validators

to_be_null / to_not_be_null

expect column("optional_field") to_be_null
expect column("required_field") to_not_be_null

to_match_pattern

expect column("email") to_match_pattern("^[\\w\\.-]+@[\\w\\.-]+\\.\\w+$")
expect column("phone") to_match_pattern("^\\d{3}-\\d{3}-\\d{4}$")

to_be_between

expect column("age") to_be_between(18, 120)
expect column("price") to_be_between(0.0, 9999.99)

to_be_in

expect column("status") to_be_in("active", "inactive", "pending")
expect column("category") to_be_in("A", "B", "C")

to_be_unique

expect column("email") to_be_unique
expect column("username") to_be_unique

Development

# Install dev dependencies
pip install -e ".[dev]"

# Run tests
pytest

# Format code
black dql_core tests

License

MIT License - see LICENSE file for details.

Documentation

Full documentation: https://yourusername.github.io/dql-core/

• Concepts - Core architecture
• Validator Guide - Create custom validators
• Cleaner Guide - Write cleaner functions
• Executor Guide - Implement framework executors
• API Reference - Complete API

Related Packages

• dql-parser - Pure Python DQL parser (docs)
• django-dqm - Django integration (docs)

Package Selection

Not sure which package to use? See the Package Selection Guide