An opinionated implementation of exclusively using airflow DockerOperators for all Operators
Project description
airflow-docker
Description
An opinionated implementation of exclusively using airflow DockerOperators for all Operators.
Default Operator
from airflow_docker.operator import Operator
task = Operator(
image='some-image:latest',
...
)
Default Sensor
from airflow_docker.operator import Sensor
sensor = Sensor(
image='some-image:latest',
...
)
Task Code
from airflow_docker_helper import client
client.sensor(True)
Branch Operator
Dag Task
from airflow_docker.operator import BranchOperator
branching_task = BranchOperator(
image='some-image:latest',
...
)
Task Code
from airflow_docker_helper import client
client.branch_to_tasks(['task1', 'task2'])
Short Circuit Operator
Dag Task
from airflow_docker.operator import ShortCircuitOperator
short_circuit = ShortCircuitOperator(
image='some-image:latest',
...
)
Task Code
from airflow_docker_helper import client
client.short_circuit() # This task will short circuit if this function gets called
Context Usage
Dag Task
from airflow_docker.operator import Operator
task = Operator(
image='some-image:latest',
provide_context=True,
...
)
Task Code
from airflow_docker_helper import client
context = client.context()
Configuration
The following operator defaults can be set under the airflowdocker
namespace:
- force_pull (boolean true/false)
- auto_remove (boolean true/false)
- network_mode
For example, to set force_pull
to False by default set the following
environment variable like so:
export AIRFLOW__AIRFLOWDOCKER__FORCE_PULL=false
Plugin
This package works as an airflow plugin as well. When installed and running airflow, dags can import like so
from airflow.{type, like "operators", "sensors"}.{name specificed inside the plugin class} import *
i.e.
from airflow.operators.airflow_docker import Operator
Tests
We also ship an airflowdocker/tester
image to verify the integrity of
your DAG definitions before committing them.
One can run the tests against your own dags like so:
docker run -it -v "${pwd}/dags:/airflow/dags" airflowdocker/tester
or else see the
airflow-docker-compose
project which ships with a test
subcommand for precisely this purpose.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
File details
Details for the file airflow-docker-2.1.1.tar.gz
.
File metadata
- Download URL: airflow-docker-2.1.1.tar.gz
- Upload date:
- Size: 17.5 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/3.3.0 pkginfo/1.7.0 requests/2.25.1 setuptools/49.3.1 requests-toolbelt/0.9.1 tqdm/4.56.2 CPython/3.6.11
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 28149435931b29ded3c196ad48c6a22219870df036cda164eddfb19c69a41f35 |
|
MD5 | 7a7da8074b41ad0bd43fd41498b8111e |
|
BLAKE2b-256 | 2913f54226f8d4d3d0840b661df0f7f7067cbedce39c311cd6572b30961a6c9b |
File details
Details for the file airflow_docker-2.1.1-py2.py3-none-any.whl
.
File metadata
- Download URL: airflow_docker-2.1.1-py2.py3-none-any.whl
- Upload date:
- Size: 20.7 kB
- Tags: Python 2, Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/3.3.0 pkginfo/1.7.0 requests/2.25.1 setuptools/49.3.1 requests-toolbelt/0.9.1 tqdm/4.56.2 CPython/3.6.11
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | f85b85f95c7a675c1159cd1f37f12d13344007eef005e45fb191bee81442eb47 |
|
MD5 | 44db3da4f67498671c1d16e6c5c8ed9f |
|
BLAKE2b-256 | 21ac91322f7b8a284a3f305255dad805f85ccd45365f87203a2fa6eb88f60cea |