Skip to main content

An opinionated implementation of exclusively using airflow DockerOperators for all Operators

Project description

airflow-docker

CircleCI Codacy Badge codecov

Description

An opinionated implementation of exclusively using airflow DockerOperators for all Operators.

Default Operator

from airflow_docker.operator import Operator

task = Operator(
    image='some-image:latest',
    ...
)

Default Sensor

from airflow_docker.operator import Sensor

sensor = Sensor(
    image='some-image:latest',
    ...
)

Task Code

from airflow_docker_helper import client

client.sensor(True)

Branch Operator

Dag Task

from airflow_docker.operator import BranchOperator

branching_task = BranchOperator(
    image='some-image:latest',
    ...
)

Task Code

from airflow_docker_helper import client

client.branch_to_tasks(['task1', 'task2'])

Short Circuit Operator

Dag Task

from airflow_docker.operator import ShortCircuitOperator

short_circuit = ShortCircuitOperator(
    image='some-image:latest',
    ...
)

Task Code

from airflow_docker_helper import client

client.short_circuit()  # This task will short circuit if this function gets called

Context Usage

Dag Task

from airflow_docker.operator import Operator

task = Operator(
    image='some-image:latest',
    provide_context=True,
    ...
)

Task Code

from airflow_docker_helper import client

context = client.context()

Configuration

The following operator defaults can be set under the airflowdocker namespace:

  • force_pull (boolean true/false)
  • auto_remove (boolean true/false)
  • network_mode

For example, to set force_pull to False by default set the following environment variable like so:

export AIRFLOW__AIRFLOWDOCKER__FORCE_PULL=false

Plugin

This package works as an airflow plugin as well. When installed and running airflow, dags can import like so

from airflow.{type, like "operators", "sensors"}.{name specificed inside the plugin class} import *

i.e.

from airflow.operators.airflow_docker import Operator

Tests

We also ship an airflowdocker/tester image to verify the integrity of your DAG definitions before committing them.

One can run the tests against your own dags like so:

docker run -it -v "${pwd}/dags:/airflow/dags" airflowdocker/tester

or else see the airflow-docker-compose project which ships with a test subcommand for precisely this purpose.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

airflow-docker-2.1.1.tar.gz (17.5 kB view details)

Uploaded Source

Built Distribution

airflow_docker-2.1.1-py2.py3-none-any.whl (20.7 kB view details)

Uploaded Python 2 Python 3

File details

Details for the file airflow-docker-2.1.1.tar.gz.

File metadata

  • Download URL: airflow-docker-2.1.1.tar.gz
  • Upload date:
  • Size: 17.5 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.7.0 requests/2.25.1 setuptools/49.3.1 requests-toolbelt/0.9.1 tqdm/4.56.2 CPython/3.6.11

File hashes

Hashes for airflow-docker-2.1.1.tar.gz
Algorithm Hash digest
SHA256 28149435931b29ded3c196ad48c6a22219870df036cda164eddfb19c69a41f35
MD5 7a7da8074b41ad0bd43fd41498b8111e
BLAKE2b-256 2913f54226f8d4d3d0840b661df0f7f7067cbedce39c311cd6572b30961a6c9b

See more details on using hashes here.

File details

Details for the file airflow_docker-2.1.1-py2.py3-none-any.whl.

File metadata

  • Download URL: airflow_docker-2.1.1-py2.py3-none-any.whl
  • Upload date:
  • Size: 20.7 kB
  • Tags: Python 2, Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.7.0 requests/2.25.1 setuptools/49.3.1 requests-toolbelt/0.9.1 tqdm/4.56.2 CPython/3.6.11

File hashes

Hashes for airflow_docker-2.1.1-py2.py3-none-any.whl
Algorithm Hash digest
SHA256 f85b85f95c7a675c1159cd1f37f12d13344007eef005e45fb191bee81442eb47
MD5 44db3da4f67498671c1d16e6c5c8ed9f
BLAKE2b-256 21ac91322f7b8a284a3f305255dad805f85ccd45365f87203a2fa6eb88f60cea

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page