airflow-run

Airflow Run

Python tool for deploying Airflow Multi-Node Cluster.

Requirements

Python >=3.6 (tested)

Goal

To provide a quick way to setup Airflow Multi-Node Cluster (a.k.a. Celery Executor Setup).

Steps

Generate config yaml file.
Run commands to start webserver, scheduler, worker, (rabbitmq, postgres).
Add dag files and run initdb.

Generate config file:

afr --generate_config

Running the tool in the same directory as config file:

afr --run postgresql
afr --run initdb
afr --run rabbitmq
afr --run webserver
afr --run scheduler
afr --run worker --queue {queue name}
afr --run flower

Or, running the tool specifying config path:

afr --run postgresql --config /path/config.yaml

Or, use this environment variable to set the config path:

export AIRFLOWRUN_CONFIG_PATH="/some_path/config.yaml"

After running webserver, scheduler and worker (postgres and rabbitmq if needed local instances), Add your dag files in the dags subdirectory in the directory you defined in the config file.

(* note: make sure you have the correct user permission in the dags, logs subdirectories.)

That is it!!

Default Config yaml file:

private_registry: False
registry_url: registry.hub.docker.com
username: ""
password: ""
repository: pkuong/airflow-run
image: airflow-run
tag: latest
local_dir: {local directory where you want to mount /dags and /logs folder}
webserver_port: 8000
flower_port: 5555
custom_mount_volumes: []
env:
  AIRFLOW__CORE__EXECUTOR: CeleryExecutor
  AIRFLOW__CORE__LOAD_EXAMPLES: "False"
  AIRFLOW__CORE__DAGS_FOLDER: /usr/local/airflow/airflow/dags
  AIRFLOW__CORE__LOGS_FOLDER: /usr/local/airflow/airflow/logs
  AIRFLOW_HOME: /usr/local/airflow
  AIRFLOW__CORE__FERNET_KEY: ""
rabbitmq:
  name: rabbitmq
  username: {username}
  password: {password}
  host: {IP}
  virtual_host: /
  image: rabbitmq:3-management
  home: /var/lib/rabbitmq
  ui_port: 15672
  port: 5672
  env:
    RABBITMQ_DEFAULT_USER: {username}
    RABBITMQ_DEFAULT_PASS: {password}
postgresql:
  name: postgresql
  username: {username}
  password: {password}
  host: {host}
  image: postgres
  data: /var/lib/postgresql/data
  port: 5432
  env:
    PGDATA: /var/lib/postgresql/data/pgdata
    POSTGRES_USER: {username}
    POSTGRES_PASSWORD: {password}

Custom mount volumes

You can specify custom mount volumes in the container, for example:

custom_mount_volumes:
  - host_path: /Users/bob/.aws
    container_path: /usr/local/airflow/.aws

Docker image

This tool is using the following public docker image by default.

https://hub.docker.com/repository/docker/pkuong/airflow-run

Building the image:

If you want to build your own image, you can run the following:

afd --build --config_path={absolute path to config.yaml} --dockerfile_path={absolute path to directory which contains Dockerfile}

Contributors

Paulo Kuong (@pkuong)

Development Status
- 3 - Alpha
License
- OSI Approved :: MIT License
Programming Language
- Python :: 3.4

Release files for airflow-run 0.4.9

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for airflow-run 0.4.9
File	Size	Uploaded
airflow-run-0.4.9.tar.gz	10.5 kB	Jul 26, 2021	Details

Built distribution (wheel)

Table of built distributions (wheels) for airflow-run 0.4.9
File	Interpreter	ABI	Platform	Reset
airflow_run-0.4.9-py3-none-any.whl 11.0 kB Jul 26, 2021	Python 3	none	any	Details

Total release size: 21.5 kB

Release files / airflow-run-0.4.9.tar.gz

Download URL	airflow-run-0.4.9.tar.gz
Size	10.5 kB
Tags	Source
SHA-256 checksum How to use checksums	`9a81fb756cb7f2660463a7588026bcc7a88bbabfc3998acf87049f828de157ec`
BLAKE2b-256 checksum How to use checksums	`430225e3fd96986f630c4e9e1b25bb7e41182bac022d53368b9407fc44bb0cc4`
Upload date	Jul 26, 2021
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/3.4.2 importlib_metadata/3.7.3 pkginfo/1.7.1 requests/2.25.1 requests-toolbelt/0.9.1 tqdm/4.59.0 CPython/3.7.5`

Release files / airflow_run-0.4.9-py3-none-any.whl

Download URL	airflow_run-0.4.9-py3-none-any.whl
Size	11.0 kB
Tags	Python 3
SHA-256 checksum How to use checksums	`15c9df7fa34e8b3a56a05ee4ea953476ce759d8a99e489494a5403a00a506c70`
BLAKE2b-256 checksum How to use checksums	`c0d644ce2e4ad0cd5dc3ab5bb417524fa40a37ced7c16cb0d5d72e8b517f0f74`
Upload date	Jul 26, 2021
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/3.4.2 importlib_metadata/3.7.3 pkginfo/1.7.1 requests/2.25.1 requests-toolbelt/0.9.1 tqdm/4.59.0 CPython/3.7.5`