The Airflow provider for Paradime to run and manage dbt™ jobs in production. The provider enables interaction with Paradime’s Bolt scheduler and management APIs.
Project description
airflow-provider-paradime-dbt
This is the provider for Paradime to run and manage dbt™ jobs in production. The provider enables interaction with Paradime’s Bolt scheduler and management APIs.
Usage
Create a connection
- Generate your API key, secret and endpoint from Paradime Workspace settings.
- Create a connection in Airflow, as shown below.
Create a DAG
Here is one example:
from airflow.decorators import dag
from paradime_dbt_provider.operators.paradime import ParadimeBoltDbtScheduleRunArtifactOperator, ParadimeBoltDbtScheduleRunOperator
from paradime_dbt_provider.sensors.paradime import ParadimeBoltDbtScheduleRunSensor
PARADIME_CONN_ID = "your_paradime_conn_id" # Update this to your connection id
BOLT_SCHEDULE_NAME = "your_schedule_name" # Update this to your schedule name
@dag(
default_args={"conn_id": PARADIME_CONN_ID},
)
def run_schedule_and_download_manifest():
# Run the schedule and return the run id as the xcom return value
task_run_schedule = ParadimeBoltDbtScheduleRunOperator(task_id="run_schedule", schedule_name=BOLT_SCHEDULE_NAME)
# Get the run id from the xcom return value
run_id = "{{ task_instance.xcom_pull(task_ids='run_schedule') }}"
# Wait for the schedule to complete before continuing
task_wait_for_schedule = ParadimeBoltDbtScheduleRunSensor(task_id="wait_for_schedule", run_id=run_id)
# Download the manifest.json file from the schedule run and return the path as the xcom return value
task_download_manifest = ParadimeBoltDbtScheduleRunArtifactOperator(task_id="download_manifest", run_id=run_id, artifact_path="target/manifest.json")
# Get the path to the manifest.json file from the xcom return value
output_path = "{{ task_instance.xcom_pull(task_ids='download_manifest') }}"
task_run_schedule >> task_wait_for_schedule >> task_download_manifest
run_schedule_and_download_manifest()
Refer to the example DAGs in this repository for more examples.
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
File details
Details for the file airflow-provider-paradime-dbt-1.0.5.tar.gz
.
File metadata
- Download URL: airflow-provider-paradime-dbt-1.0.5.tar.gz
- Upload date:
- Size: 13.0 kB
- Tags: Source
- Uploaded using Trusted Publishing? Yes
- Uploaded via: twine/4.0.2 CPython/3.11.7
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | a5a2167f79ba9892adb8354ad25f416ceac7cc843d5a03db207a3ccfdb688d0e |
|
MD5 | 6857a0d570ae87d49c8f0b7ef455b950 |
|
BLAKE2b-256 | c1523220db62f35c2fe0573d552e59b6ceb9f3d91dc5f933caffbb9a51ca9656 |
File details
Details for the file airflow_provider_paradime_dbt-1.0.5-py3-none-any.whl
.
File metadata
- Download URL: airflow_provider_paradime_dbt-1.0.5-py3-none-any.whl
- Upload date:
- Size: 16.8 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? Yes
- Uploaded via: twine/4.0.2 CPython/3.11.7
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 2e32b03104b64b4682b421a2281ce517643ea7733540f18fe9dde5876181aee7 |
|
MD5 | 9cd5502e1868abdd17d263dea8211709 |
|
BLAKE2b-256 | 91f4fa2b6e89136eb6eec966be806ccb4500c5973221030ccb03a4673bac08c1 |