No project description provided
Project description
Dataverk airflow
Enkelt wrapperbibliotek rundt KubernetesPodOperator som lager Airflow task som kjører i en Kubernetes pod.
Våre operators
Alle våre operators lar deg klone et repo på forhånd, bare legg det til med repo="navikt/<repo>
.
Vi har også støtte for å installere Python pakker ved oppstart av Airflow task, spesifiser requirements.txt
-filen din med requirements_path="/path/to/requirements.txt"
.
Quarto operator
Denne kjører Quarto render for deg.
from airflow import DAG
from airflow.utils.dates import days_ago
from dataverk_airflow import quarto_operator
with DAG('navn-dag', start_date=days_ago(1), schedule_interval="*/10 * * * *") as dag:
t1 = quarto_operator(dag=dag,
name="<navn-på-task>",
repo="navikt/<repo>",
quarto={
"path": "/path/to/index.qmd",
"env": "dev/prod",
"id":"uuid",
"token":
"quarto-token"
},
slack_channel="<#slack-alarm-kanal>")
Notebook operator
Denne lar deg kjøre en Jupyter notebook.
from airflow import DAG
from airflow.utils.dates import days_ago
from dataverk_airflow import notebook_operator
with DAG('navn-dag', start_date=days_ago(1), schedule_interval="*/10 * * * *") as dag:
t1 = notebook_operator(dag=dag,
name="<navn-på-task>",
repo="navikt/<repo>",
nb_path="/path/to/notebook.ipynb",
slack_channel="<#slack-alarm-kanal>")
Python operator
Denne lar deg kjøre vilkårlig Python-scripts.
from airflow import DAG
from airflow.utils.dates import days_ago
from dataverk_airflow import python_operator
with DAG('navn-dag', start_date=days_ago(1), schedule_interval="*/10 * * * *") as dag:
t1 = python_operator(dag=dag,
name="<navn-på-task>",
repo="navikt/<repo>",
script_path="/path/to/script.py",
slack_channel="<#slack-alarm-kanal>")
Kubernetes operator
Vi tilbyr også vår egen Kubernetes operator som kloner et valg repo inn i containeren.
from airflow import DAG
from airflow.utils.dates import days_ago
from dataverk_airflow import kubernetes_operator
with DAG('navn-dag', start_date=days_ago(1), schedule_interval="*/10 * * * *") as dag:
t1 = kubernetes_operator(dag=dag,
name="<navn-på-task>",
repo="navikt/<repo>",
cmds=["/path/to/bin/", "script-name.sh", "argument1", "argument2"],
image="europe-north1-docker.pkg.dev/nais-management-233d/ditt-team/ditt-image:din-tag",
slack_channel="<#slack-alarm-kanal>")
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
dataverk_airflow-0.5.3.tar.gz
(7.3 kB
view hashes)
Built Distribution
Close
Hashes for dataverk_airflow-0.5.3-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | c6ba631025ebe069f8eee34cad90ad4e57001e02a05da3270d26ad64850d4818 |
|
MD5 | b22061b184b7bff023bc8200c2b2767a |
|
BLAKE2b-256 | ff0cff9b4c4cdc720bce63962d583cc41c2577788baed3920ef4d8c16ae85701 |