Versatile Data Kit Airflow provider
A set of Airflow operators, sensors and a connection hook intended to help schedule Versatile Data Kit jobs using Apache Airflow.
Usage
To install it simply run:
pip install airflow-provider-vdk
Then you can create a workflow of data jobs (deployed by VDK Control Service) like this:
from datetime import datetime
from airflow import DAG
from vdk_provider.operators.vdk import VDKOperator
with DAG(
"airflow_example_vdk",
schedule_interval=None,
start_date=datetime(2022, 1, 1),
catchup=False,
tags=["example", "vdk"],
) as dag:
trino_job1 = VDKOperator(
conn_id="vdk-default",
job_name="airflow-trino-job1",
team_name="taurus",
task_id="trino-job1",
)
trino_job2 = VDKOperator(
conn_id="vdk-default",
job_name="airflow-trino-job2",
team_name="taurus",
task_id="trino-job2",
)
transform_job = VDKOperator(
conn_id="vdk-default",
job_name="airflow-transform-job",
team_name="taurus",
task_id="transform-job",
)
[trino_job1, trino_job2] >> transform_job
Example
Demo
You can see demo during one of the community meetings here: https://www.youtube.com/watch?v=c3j1aOALjVU&t=690s
Architecture
1 maintainer
Metadata
1 maintainer
Release files for airflow-provider-vdk 0.0.602273476
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| airflow-provider-vdk-0.0.602273476.tar.gz | 9.5 kB | Details |
Release files / airflow-provider-vdk-0.0.602273476.tar.gz
| Download URL | airflow-provider-vdk-0.0.602273476.tar.gz |
|---|---|
| Size | 9.5 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
b0e8cfd9006a5f6ae0f640f3d187d2f191ea1bfa9f7a0aad0745c0c03075406d
|
|
BLAKE2b-256 checksum How to use checksums |
aec334fd3fa14128dcf8cce0e6e75d2638a76e535f9549db75f1000b881a72c6
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/4.0.1 CPython/3.7.13
|
