vd-dlt 0.1.0

Core DLT ingestion framework for VibeData pipelines

vd-dlt

Core DLT ingestion framework for VibeData pipelines. Provides config resolution, credential management (Azure Key Vault), pipeline execution, and observability.

Installation

# Core only
pip install vd-dlt

# With pipeline dependencies (dlt, pyarrow)
pip install vd-dlt[pipeline]

# With Notion connector
pip install vd-dlt[notion]

# With Notion schema (defaults, docs)
pip install vd-dlt[notion-schema]

# Everything for Notion
pip install vd-dlt[pipeline,notion,notion-schema]

Quick Start

from vd_dlt import PipelineRunner

runner = PipelineRunner(
    vault_url="https://my-vault.vault.azure.net/",
)
result = runner.run("my_source_name")
print(f"Loaded {result.total_rows_loaded} rows")

Architecture

The framework uses a 4-level config hierarchy (most specific wins):

1. Connector Defaults - from connector schema package
2. Source Config - from source YAML (including default_sync)
3. Group Config - optional grouping within source
4. Resource Config - per-table overrides