An opinionated implementation of exclusively using airflow DockerOperators for all Operators
Project description
airflow-docker
Description
An opinionated implementation of exclusively using airflow DockerOperators for all Operators.
Default Operator
from airflow_docker.operator import Operator
task = Operator(
image='some-image:latest',
...
)
Default Sensor
from airflow_docker.operator import Sensor
sensor = Sensor(
image='some-image:latest',
...
)
Task Code
from airflow_docker_helper import client
client.sensor(True)
Branch Operator
Dag Task
from airflow_docker.operator import BranchOperator
branching_task = BranchOperator(
image='some-image:latest',
...
)
Task Code
from airflow_docker_helper import client
client.branch_to_tasks(['task1', 'task2'])
Short Circuit Operator
Dag Task
from airflow_docker.operator import ShortCircuitOperator
short_circuit = ShortCircuitOperator(
image='some-image:latest',
...
)
Task Code
from airflow_docker_helper import client
client.short_circuit() # This task will short circuit if this function gets called
Context Usage
Dag Task
from airflow_docker.operator import Operator
task = Operator(
image='some-image:latest',
provide_context=True,
...
)
Task Code
from airflow_docker_helper import client
context = client.context()
Configuration
The following operator defaults can be set under the airflowdocker
namespace:
- force_pull (boolean true/false)
- auto_remove (boolean true/false)
- network_mode
For example, to set force_pull
to False by default set the following
environment variable like so:
export AIRFLOW__AIRFLOWDOCKER__FORCE_PULL=false
Plugin
This package works as an airflow plugin as well. When installed and running airflow, dags can import like so
from airflow.{type, like "operators", "sensors"}.{name specificed inside the plugin class} import *
i.e.
from airflow.operators.airflow_docker import Operator
Tests
We also ship an airflowdocker/tester
image to verify the integrity of
your DAG definitions before committing them.
One can run the tests against your own dags like so:
docker run -it -v "${pwd}/dags:/airflow/dags" airflowdocker/tester
or else see the
airflow-docker-compose
project which ships with a test
subcommand for precisely this purpose.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for airflow_docker-2.1.0-py2.py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 2a84fd28c1fc24461ffc4988ce2d2634ca0446c0de6d05bdd4e46c4aeb1e6d2e |
|
MD5 | a7d9974175facffe8bc4d3c547720eb3 |
|
BLAKE2b-256 | aa4fd4d2d3efc9d18026d9793dc7ce752365711c40f007293c35235adbd27deb |