Skip to main content

DBT extension for Metaflow

Project description

DBT extension for Metaflow

This extension adds support for executing DBT models as part of steps in Metaflow Flows via decorators.

Basic usage

Having a dbt_project.yml as part of your Flow project will allow executing dbt run as a pre-step to any task by simply adding the decorator to a step.

@dbt
@step
def start(self):
    # DBT Models have been run when step execution starts

    self.next(self.second_step)

If you only want to run a specific model as part of a step, you can specify this with the models= attribute

@dbt(models="customers")

Configuration options

Project directory

You might want to keep the DBT project separately nested within the Flow project. In these cases you would need to specify the location of the DBT project folder, due to the way project lookup works. This can be done by specifying the project location as a relative or absolute path within the decorator

@dbt(models="customers", project_dir="./dbt_project")

Supplying credentials

When deploying a DBT flow to be executed remotely, we do not want to bundle up sensitive credentials into the code package. Therefore a plain text profiles.yml will not suffice. We can utilize the environment variable replacement that DBT offers to get around this.

example profiles:

dbt_decorator:
  outputs:
    dev:
      type: postgres
      threads: 1
      host: localhost
      port: 5432
      user: "{{ env_var('DBT_POSTGRES_USER') }}"
      pass: "{{ env_var('DBT_POSTGRES_PW') }}"
      dbname: dbt_decorator
      schema: dev_jaffle_schema

    prod:
      type: postgres
      threads: 1
      host: localhost
      port: 5432
      user: "{{ env_var('DBT_POSTGRES_USER') }}"
      pass: "{{ env_var('DBT_POSTGRES_PW') }}"
      dbname: dbt_decorator
      schema: prod_jaffle_schema

  target: dev

Note: any profiles.yml in the flow project folder will be packaged, so make sure that they do not contain sensitive secrets.

We can supply the environment variables in various ways, for example

  • having them already present in the execution environment
  • supplying them with the @environment decorator in the flow (this still ends up bundling secrets into the package, but is good for testing)
  • hydrating environment variables with the @secrets decorator from a secret manager.

Examples

The command metaflow dbt examples will pull example flows to your working directory under metaflow-dbt-examples.

Follow the README.md under the examples directory for more instructions.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

metaflow_dbt-1.0.5.tar.gz (95.4 kB view details)

Uploaded Source

Built Distribution

metaflow_dbt-1.0.5-py3-none-any.whl (108.6 kB view details)

Uploaded Python 3

File details

Details for the file metaflow_dbt-1.0.5.tar.gz.

File metadata

  • Download URL: metaflow_dbt-1.0.5.tar.gz
  • Upload date:
  • Size: 95.4 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/5.1.1 CPython/3.12.7

File hashes

Hashes for metaflow_dbt-1.0.5.tar.gz
Algorithm Hash digest
SHA256 48921a586547ad9f6f9bed561d9e52ca47ab047aa59c84269d8aa75d889a0b93
MD5 82cbf93beb5e428d457259681809cfd8
BLAKE2b-256 d31d2b28d781906daa5dc22e238589419faac61dd2531e6d573136d309ab9e3e

See more details on using hashes here.

Provenance

The following attestation bundles were made for metaflow_dbt-1.0.5.tar.gz:

Publisher: publish.yml on outerbounds/metaflow-dbt

Attestations:

File details

Details for the file metaflow_dbt-1.0.5-py3-none-any.whl.

File metadata

  • Download URL: metaflow_dbt-1.0.5-py3-none-any.whl
  • Upload date:
  • Size: 108.6 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/5.1.1 CPython/3.12.7

File hashes

Hashes for metaflow_dbt-1.0.5-py3-none-any.whl
Algorithm Hash digest
SHA256 7679b2a16ff2b60e2a5628aa86e1762b33212676c110e391a5d902e8bdb5df44
MD5 c867719e758c85fad0d1b01f02eee17f
BLAKE2b-256 fdfec3b74a7462b479ef64fa6b5a9217ade802febe48aa48e3fb18ed6bb6d383

See more details on using hashes here.

Provenance

The following attestation bundles were made for metaflow_dbt-1.0.5-py3-none-any.whl:

Publisher: publish.yml on outerbounds/metaflow-dbt

Attestations:

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page