Skip to main content

The Airflow provider for Paradime to run and manage dbt™ jobs in production. The provider enables interaction with Paradime’s Bolt scheduler and management APIs.

Project description

Paradime

airflow-provider-paradime-dbt

This is the provider for Paradime to run and manage dbt™ jobs in production. The provider enables interaction with Paradime’s Bolt scheduler and management APIs.

Usage

Create a connection

  1. Generate your API key, secret and endpoint from Paradime Workspace settings.
  2. Create a connection in Airflow, as shown below. Create a connection

Create a DAG

Here is one example:

from airflow.decorators import dag

from paradime_dbt_provider.operators.paradime import ParadimeBoltDbtScheduleRunArtifactOperator, ParadimeBoltDbtScheduleRunOperator
from paradime_dbt_provider.sensors.paradime import ParadimeBoltDbtScheduleRunSensor

PARADIME_CONN_ID = "your_paradime_conn_id"  # Update this to your connection id
BOLT_SCHEDULE_NAME = "your_schedule_name"  # Update this to your schedule name


@dag(
    default_args={"conn_id": PARADIME_CONN_ID},
)
def run_schedule_and_download_manifest():
    # Run the schedule and return the run id as the xcom return value
    task_run_schedule = ParadimeBoltDbtScheduleRunOperator(task_id="run_schedule", schedule_name=BOLT_SCHEDULE_NAME)

    # Get the run id from the xcom return value
    run_id = "{{ task_instance.xcom_pull(task_ids='run_schedule') }}"

    # Wait for the schedule to complete before continuing
    task_wait_for_schedule = ParadimeBoltDbtScheduleRunSensor(task_id="wait_for_schedule", run_id=run_id)

    # Download the manifest.json file from the schedule run and return the path as the xcom return value
    task_download_manifest = ParadimeBoltDbtScheduleRunArtifactOperator(task_id="download_manifest", run_id=run_id, artifact_path="target/manifest.json")

    # Get the path to the manifest.json file from the xcom return value
    output_path = "{{ task_instance.xcom_pull(task_ids='download_manifest') }}"

    task_run_schedule >> task_wait_for_schedule >> task_download_manifest


run_schedule_and_download_manifest()

Refer to the example DAGs in this repository for more examples.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

airflow-provider-paradime-dbt-1.0.5.tar.gz (13.0 kB view details)

Uploaded Source

Built Distribution

File details

Details for the file airflow-provider-paradime-dbt-1.0.5.tar.gz.

File metadata

File hashes

Hashes for airflow-provider-paradime-dbt-1.0.5.tar.gz
Algorithm Hash digest
SHA256 a5a2167f79ba9892adb8354ad25f416ceac7cc843d5a03db207a3ccfdb688d0e
MD5 6857a0d570ae87d49c8f0b7ef455b950
BLAKE2b-256 c1523220db62f35c2fe0573d552e59b6ceb9f3d91dc5f933caffbb9a51ca9656

See more details on using hashes here.

File details

Details for the file airflow_provider_paradime_dbt-1.0.5-py3-none-any.whl.

File metadata

File hashes

Hashes for airflow_provider_paradime_dbt-1.0.5-py3-none-any.whl
Algorithm Hash digest
SHA256 2e32b03104b64b4682b421a2281ce517643ea7733540f18fe9dde5876181aee7
MD5 9cd5502e1868abdd17d263dea8211709
BLAKE2b-256 91f4fa2b6e89136eb6eec966be806ccb4500c5973221030ccb03a4673bac08c1

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page