Skip to main content

An opinionated implementation of exclusively using airflow DockerOperators for all Operators

Project description

airflow-docker

CircleCI Codacy Badge codecov

Description

An opinionated implementation of exclusively using airflow DockerOperators for all Operators.

Default Operator

from airflow_docker.operator import Operator

task = Operator(
    image='some-image:latest',
    ...
)

Default Sensor

from airflow_docker.operator import Sensor

sensor = Sensor(
    image='some-image:latest',
    ...
)

Task Code

from airflow_docker_helper import client

client.sensor(True)

Branch Operator

Dag Task

from airflow_docker.operator import BranchOperator

branching_task = BranchOperator(
    image='some-image:latest',
    ...
)

Task Code

from airflow_docker_helper import client

client.branch_to_tasks(['task1', 'task2'])

Short Circuit Operator

Dag Task

from airflow_docker.operator import ShortCircuitOperator

short_circuit = ShortCircuitOperator(
    image='some-image:latest',
    ...
)

Task Code

from airflow_docker_helper import client

client.short_circuit()  # This task will short circuit if this function gets called

Context Usage

Dag Task

from airflow_docker.operator import Operator

task = Operator(
    image='some-image:latest',
    provide_context=True,
    ...
)

Task Code

from airflow_docker_helper import client

context = client.context()

Configuration

The following operator defaults can be set under the airflowdocker namespace:

  • force_pull (boolean true/false)
  • auto_remove (boolean true/false)
  • network_mode

For example, to set force_pull to False by default set the following environment variable like so:

export AIRFLOW__AIRFLOWDOCKER__FORCE_PULL=false

Plugin

This package works as an airflow plugin as well. When installed and running airflow, dags can import like so

from airflow.{type, like "operators", "sensors"}.{name specificed inside the plugin class} import *

i.e.

from airflow.operators.airflow_docker import Operator

Tests

We also ship an airflowdocker/tester image to verify the integrity of your DAG definitions before committing them.

One can run the tests against your own dags like so:

docker run -it -v "${pwd}/dags:/airflow/dags" airflowdocker/tester

or else see the airflow-docker-compose project which ships with a test subcommand for precisely this purpose.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for airflow-docker, version 2.1.1
Filename, size File type Python version Upload date Hashes
Filename, size airflow_docker-2.1.1-py2.py3-none-any.whl (20.7 kB) File type Wheel Python version py2.py3 Upload date Hashes View
Filename, size airflow-docker-2.1.1.tar.gz (17.5 kB) File type Source Python version None Upload date Hashes View

Supported by

AWS AWS Cloud computing Datadog Datadog Monitoring DigiCert DigiCert EV certificate Facebook / Instagram Facebook / Instagram PSF Sponsor Fastly Fastly CDN Google Google Object Storage and Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Salesforce Salesforce PSF Sponsor Sentry Sentry Error logging StatusPage StatusPage Status page