dmp-af: distributed dbt runs on Airflow

Overview

dmp-af runs your dbt models in parallel on Airflow. Each model becomes an independent task while preserving dependencies across domains.

Built for scale. Designed for large dbt projects (1000+ models) and data mesh architecture. Works with any project size.

Why dmp-af?

1. Domain-driven architecture - Separate models by domain into different DAGs, run in parallel, perfect for data mesh
2. dbt-first design - All configuration in dbt model configs, analytics teams stay in dbt, no Airflow knowledge required
3. Flexible scheduling - Multiple schedules per model (@hourly, @daily, @weekly, @monthly, and more)
4. Enterprise features - Multiple dbt targets, configurable test strategies, built-in maintenance, Kubernetes support

Installation

To install dmp-af run pip install dmp-af.

To contribute we recommend to use uv to install package dependencies. Run uv sync --all-packages --all-groups --all-extras to install all dependencies.

dmp-af by Example

All tutorials and examples are located in the examples folder.

To get basic Airflow DAGs for your dbt project, you need to put the following code into your dags folder:

# LABELS: dag, airflow (it's required for airflow dag-processor)
from dmp_af.dags import compile_dmp_af_dags
from dmp_af.conf import Config, DbtDefaultTargetsConfig, DbtProjectConfig

# specify here all settings for your dbt project
config = Config(
    dbt_project=DbtProjectConfig(
        dbt_project_name='my_dbt_project',
        dbt_project_path='/path/to/my_dbt_project',
        dbt_models_path='/path/to/my_dbt_project/models',
        dbt_profiles_path='/path/to/my_dbt_project',
        dbt_target_path='/path/to/my_dbt_project/target',
        dbt_log_path='/path/to/my_dbt_project/logs',
        dbt_schema='my_dbt_schema',
    ),
    dbt_default_targets=DbtDefaultTargetsConfig(default_target='dev'),
    dry_run=False,  # set to True if you want to turn on dry-run mode
)

dags = compile_dmp_af_dags(
    manifest_path='/path/to/my_dbt_project/target/manifest.json',
    config=config,
)
for dag_name, dag in dags.items():
    globals()[dag_name] = dag

In dbt_project.yml you need to set up default targets for all nodes in your project (see example):

sql_cluster: "dev"
daily_sql_cluster: "dev"
py_cluster: "dev"
bf_cluster: "dev"

This will create Airflow DAGs for your dbt project.

Check out the documentation for more details here.

Key Features

Auto-generated DAGs

• Automatically creates Airflow DAGs from your dbt project
• Organizes by domain and schedule
• Handles dependencies across domains

Idempotent runs

• Each model is a separate Airflow task
• Date intervals passed to every run
• Reliable backfills and reruns

Team-friendly

• Analytics teams stay in dbt
• No Airflow DAG writing required
• Infrastructure handled automatically

Requirements

dmp-af is tested with:

Project Information

• Docs
• PyPI
• Contributing
• Original dbt-af Project

About this fork

This project is a fork of Toloka AI BV's original repository. It includes substantial modifications by IJKOS & PARTNERS LTD. This fork is not affiliated with or endorsed by Toloka AI BV.

The original project is licensed under the Apache License 2.0.

Migrating from dbt-af

If you're currently using dbt-af and want to migrate to dmp-af, see our Migration Guide for step-by-step instructions.