Skip to main content

An opinionated implementation of exclusively using airflow DockerOperators for all Operators

Project description

airflow-docker

CircleCI Codacy Badge codecov

Description

An opinionated implementation of exclusively using airflow DockerOperators for all Operators.

Default Operator

from airflow_docker.operator import Operator

task = Operator(
    image='some-image:latest',
    ...
)

Default Sensor

from airflow_docker.operator import Sensor

sensor = Sensor(
    image='some-image:latest',
    ...
)

Task Code

from airflow_docker_helper import client

client.sensor(True)

Branch Operator

Dag Task

from airflow_docker.operator import BranchOperator

branching_task = BranchOperator(
    image='some-image:latest',
    ...
)

Task Code

from airflow_docker_helper import client

client.branch_to_tasks(['task1', 'task2'])

Short Circuit Operator

Dag Task

from airflow_docker.operator import ShortCircuitOperator

short_circuit = ShortCircuitOperator(
    image='some-image:latest',
    ...
)

Task Code

from airflow_docker_helper import client

client.short_circuit()  # This task will short circuit if this function gets called

Context Usage

Dag Task

from airflow_docker.operator import Operator

task = Operator(
    image='some-image:latest',
    provide_context=True,
    ...
)

Task Code

from airflow_docker_helper import client

context = client.context()

Configuration

The following operator defaults can be set under the airflowdocker namespace:

  • force_pull (boolean true/false)
  • auto_remove (boolean true/false)
  • network_mode

For example, to set force_pull to False by default set the following environment variable like so:

export AIRFLOW__AIRFLOWDOCKER__FORCE_PULL=false

Plugin

This package works as an airflow plugin as well. When installed and running airflow, dags can import like so

from airflow.{type, like "operators", "sensors"}.{name specificed inside the plugin class} import *

i.e.

from airflow.operators.airflow_docker import Operator

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for airflow-docker, version 1.1.3
Filename, size File type Python version Upload date Hashes
Filename, size airflow_docker-1.1.3-py2.py3-none-any.whl (16.2 kB) File type Wheel Python version py2.py3 Upload date Hashes View hashes
Filename, size airflow-docker-1.1.3.tar.gz (15.3 kB) File type Source Python version None Upload date Hashes View hashes

Supported by

Elastic Elastic Search Pingdom Pingdom Monitoring Google Google BigQuery Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN DigiCert DigiCert EV certificate StatusPage StatusPage Status page