pipetree

Metadata-driven pipeline orchestration: describe what to move in declarative YAML, and a small package turns it into a runnable dependency tree.

YAML → config loader → typed model → graph builder → executor

Documentation: https://pipetree.dev. The package is the companion to the Pipeline Engineering blog series.

What it does

• One YAML block per table: a source or a SQL/PySpark file, a business key and a merge strategy. depends_on: auto reads the parents from the code.
• A dependency tree that runs: a table starts the moment its parents have succeeded. Transient errors are retried, a failed table marks its dependents upstream_failed and the rest finishes.
• Merge strategies on Delta: replace, append, scd1 and scd2, with sequence_by ordering across batches, soft or hard deletes, snapshot merges for sources that deliver a full extract, unknown members and surrogate keys.
• Notes for what a merge quietly changed: dropped duplicates, NULL keys, schema drift, deleted keys. In the log, the digest, the graph and the run log.
• A live picture of the run in a notebook or as an HTML file.
• Tags and partial runs: --select, --with-dependents, --with-ancestors, --tag.
• Databricks and Fabric: the same config runs on local Spark, Databricks and Fabric; secrets from Azure Key Vault without a secret, a failure email through Azure Communication Services, and a translation of scd tables to Databricks' AUTO CDC.

Installation

pip install pipetree-meta            # core: config loader, graph, executor, CLI
pip install "pipetree-meta[spark]"   # + PySpark and Delta Lake for local runs
pip install "pipetree-meta[azure]"   # + Key Vault secrets, token auth, failure email

Python 3.11 or newer. The import package and the command are pipetree. Local Spark needs a JDK (for example brew install openjdk@17). On Databricks and Fabric install without [spark], the runtime brings its own Spark and Delta.

Quick start

# pipeline.yaml
systems:
  crm:
    type: csv
    path: data/customer.csv
bronze:
  tables:
    customer:
      source: { system: crm, object: account }
      business_key: [accountid]
      strategy: scd1

mkdir -p data && printf 'accountid,name\n1,Contoso\n2,Fabrikam\n' > data/customer.csv   # a tiny source file
pipetree validate --config pipeline.yaml   # check the config, nothing runs
pipetree graph --config pipeline.yaml      # print the dependency tree
pipetree run --config pipeline.yaml        # run it and print a digest

The quickstart runs a seven-table example from a clone of the repository. examples/quickstart.ipynb is the same run as a notebook with its outputs, and examples/with_problems/ shows what pipetree reports when the data misbehaves.

Documentation

• Install and Quickstart
• Concepts: YAML structure, the dependency tree, running, tags, notes and the run log, seeing the run
• Merge strategies: replace, append, scd1, scd2, snapshot merge, merge options
• Guides: Databricks, Fabric, notifications, AUTO CDC, custom sources
• Reference: every YAML key and every CLI option
• Architecture: the parts of the package

examples/databricks/ (a Databricks Asset Bundle) and examples/fabric/ (a notebook) run the example pipeline against a real workspace.

Development

uv sync --all-extras --dev
uv run pytest              # full suite
uv run pytest -m "not spark"  # skip the JVM-backed integration tests
uv run ruff check . && uv run ruff format --check .
uv run pyright

See CONTRIBUTING.md.

License

MIT, see LICENSE.