dq-engine 0.5.0

Framework de validação e limpeza de dados com PySpark

dq-engine

Framework PySpark para validacao e tratamento de qualidade de dados.

O que faz

• Resolve rule_set por nome de coluna
• Compila regras declarativas em expressoes Spark
• Gera score e status por coluna
• Aplica tratamento em lote
• Executa pipeline antes/depois com comparativo

Requisitos

• Python 3.10+
• PySpark 3.4+

Instalacao

pip install -e .

Uso rapido

from pyspark.sql import SparkSession
from dq_engine import DataQualityEngine
from dq_engine.config.conventions import CONVENTIONS

spark = SparkSession.builder.getOrCreate()
engine = DataQualityEngine(spark=spark, conventions=CONVENTIONS)

df = spark.table("lakehouse.clientes")
result = engine.validate_table(df=df, table_name="CLIENTES", run_id="2026-07-02")

result.output_df.orderBy("column_name").show(truncate=False)

Pipeline completo

df_tratado, result_before, result_after = engine.run_dq_pipeline(
    df=df,
    table_name="CLIENTES",
    output_dir="/mnt/dq_output",  # opcional
)

API principal

• DataQualityEngine.validate_table
• DataQualityEngine.apply_treatment
• DataQualityEngine.run_dq_pipeline
• DataQualityEngine.get_column_treatment_contexts
• DataQualityResult.table_summary
• DataQualityResult.failed_columns

Configuracao

Arquivo central de convencoes:

• src/dq_engine/config/conventions.py

Documentacao

• docs/architecture.md
• docs/rules.md