Skip to main content

Simplified Airflow CLI Tool for Lauching CeleryExecutor Deployment

Project description

Airflow Run

Python tool for deploying Airflow Multi-Node Cluster.

Requirements

  • Python >=3.6 (tested)

Goal

To provide a quick way to setup Airflow Multi-Node Cluster (a.k.a. Celery Executor Setup).

Steps

  1. Generate config yaml file.

  2. Run commands to start webserver, scheduler, worker, (rabbitmq, postgres).

  3. Add dag files and run initdb.

Generate config file:

afr --generate_config

Running the tool in the same directory as config file:

afr --run postgresql
afr --run initdb
afr --run rabbitmq
afr --run webserver
afr --run scheduler
afr --run worker --queue {queue name}
afr --run flower

Or, running the tool specifying config path:

afr --run postgresql --config /path/config.yaml

Or, use this environment variable to set the config path:

export AIRFLOWRUN_CONFIG_PATH="/some_path/config.yaml"

After running webserver, scheduler and worker (postgres and rabbitmq if needed local instances), Add your dag files in the dags subdirectory in the directory you defined in the config file.

(* note: make sure you have the correct user permission in the dags, logs subdirectories.)

That is it!!

Default Config yaml file:

private_registry: False
registry_url: registry.hub.docker.com
username: ""
password: ""
repository: pkuong/airflow-run
image: airflow-run
tag: latest
local_dir: {local directory where you want to mount /dags and /logs folder}
webserver_port: 8000
flower_port: 5555
custom_mount_volumes: []
env:
  AIRFLOW__CORE__EXECUTOR: CeleryExecutor
  AIRFLOW__CORE__LOAD_EXAMPLES: "False"
  AIRFLOW__CORE__DAGS_FOLDER: /usr/local/airflow/airflow/dags
  AIRFLOW__CORE__LOGS_FOLDER: /usr/local/airflow/airflow/logs
  AIRFLOW_HOME: /usr/local/airflow
  AIRFLOW__CORE__FERNET_KEY: ""
rabbitmq:
  name: rabbitmq
  username: {username}
  password: {password}
  host: {IP}
  virtual_host: /
  image: rabbitmq:3-management
  home: /var/lib/rabbitmq
  ui_port: 15672
  port: 5672
  env:
    RABBITMQ_DEFAULT_USER: {username}
    RABBITMQ_DEFAULT_PASS: {password}
postgresql:
  name: postgresql
  username: {username}
  password: {password}
  host: {host}
  image: postgres
  data: /var/lib/postgresql/data
  port: 5432
  env:
    PGDATA: /var/lib/postgresql/data/pgdata
    POSTGRES_USER: {username}
    POSTGRES_PASSWORD: {password}

Custom mount volumes

You can specify custom mount volumes in the container, for example:
custom_mount_volumes:
  - host_path: /Users/bob/.aws
    container_path: /usr/local/airflow/.aws

Docker image

This tool is using the following public docker image by default.
https://hub.docker.com/repository/docker/pkuong/airflow-run

Building the image:

If you want to build your own image, you can run the following:
afd --build --config_path={absolute path to config.yaml} --dockerfile_path={absolute path to directory which contains Dockerfile}

Contributors

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

airflow-run-0.4.9.tar.gz (10.5 kB view details)

Uploaded Source

Built Distribution

airflow_run-0.4.9-py3-none-any.whl (11.0 kB view details)

Uploaded Python 3

File details

Details for the file airflow-run-0.4.9.tar.gz.

File metadata

  • Download URL: airflow-run-0.4.9.tar.gz
  • Upload date:
  • Size: 10.5 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.4.2 importlib_metadata/3.7.3 pkginfo/1.7.1 requests/2.25.1 requests-toolbelt/0.9.1 tqdm/4.59.0 CPython/3.7.5

File hashes

Hashes for airflow-run-0.4.9.tar.gz
Algorithm Hash digest
SHA256 9a81fb756cb7f2660463a7588026bcc7a88bbabfc3998acf87049f828de157ec
MD5 581a4a4132b6ba6be43fb50433292c51
BLAKE2b-256 430225e3fd96986f630c4e9e1b25bb7e41182bac022d53368b9407fc44bb0cc4

See more details on using hashes here.

File details

Details for the file airflow_run-0.4.9-py3-none-any.whl.

File metadata

  • Download URL: airflow_run-0.4.9-py3-none-any.whl
  • Upload date:
  • Size: 11.0 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.4.2 importlib_metadata/3.7.3 pkginfo/1.7.1 requests/2.25.1 requests-toolbelt/0.9.1 tqdm/4.59.0 CPython/3.7.5

File hashes

Hashes for airflow_run-0.4.9-py3-none-any.whl
Algorithm Hash digest
SHA256 15c9df7fa34e8b3a56a05ee4ea953476ce759d8a99e489494a5403a00a506c70
MD5 858ae842e282e84591f46287d97c7b4d
BLAKE2b-256 c0d644ce2e4ad0cd5dc3ab5bb417524fa40a37ced7c16cb0d5d72e8b517f0f74

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page