Simplified Airflow CLI Tool for Lauching CeleryExecutor Deployment
Project description
Airflow Run
Python tool for deploying Airflow Multi-Node Cluster.
Requirements
Python >=3.6 (tested)
Goal
Steps
Generate config yaml file.
Run commands to start webserver, scheduler, worker, (rabbitmq, postgres).
Add dag files and run initdb.
Generate config file:
afr --generate_config
Running the tool in the same directory as config file:
afr --run postgresql
afr --run initdb
afr --run rabbitmq
afr --run webserver
afr --run scheduler
afr --run worker --queue {queue name}
afr --run flower
Or, running the tool specifying config path:
afr --run postgresql --config /path/config.yaml
Or, use this environment variable to set the config path:
export AIRFLOWRUN_CONFIG_PATH="/some_path/config.yaml"
After running webserver, scheduler and worker (postgres and rabbitmq if needed local instances), Add your dag files in the dags subdirectory in the directory you defined in the config file.
(* note: make sure you have the correct user permission in the dags, logs subdirectories.)
That is it!!
Default Config yaml file:
private_registry: False
registry_url: registry.hub.docker.com
username: ""
password: ""
repository: pkuong/airflow-run
image: airflow-run
tag: latest
local_dir: {local directory where you want to mount /dags and /logs folder}
webserver_port: 8000
flower_port: 5555
custom_mount_volumes: []
env:
AIRFLOW__CORE__EXECUTOR: CeleryExecutor
AIRFLOW__CORE__LOAD_EXAMPLES: "False"
AIRFLOW__CORE__DAGS_FOLDER: /usr/local/airflow/airflow/dags
AIRFLOW__CORE__LOGS_FOLDER: /usr/local/airflow/airflow/logs
AIRFLOW_HOME: /usr/local/airflow
AIRFLOW__CORE__FERNET_KEY: ""
rabbitmq:
name: rabbitmq
username: {username}
password: {password}
host: {IP}
virtual_host: /
image: rabbitmq:3-management
home: /var/lib/rabbitmq
ui_port: 15672
port: 5672
env:
RABBITMQ_DEFAULT_USER: {username}
RABBITMQ_DEFAULT_PASS: {password}
postgresql:
name: postgresql
username: {username}
password: {password}
host: {host}
image: postgres
data: /var/lib/postgresql/data
port: 5432
env:
PGDATA: /var/lib/postgresql/data/pgdata
POSTGRES_USER: {username}
POSTGRES_PASSWORD: {password}
Custom mount volumes
custom_mount_volumes:
- host_path: /Users/bob/.aws
container_path: /usr/local/airflow/.aws
Docker image
https://hub.docker.com/repository/docker/pkuong/airflow-run
Building the image:
afd --build --config_path={absolute path to config.yaml} --dockerfile_path={absolute path to directory which contains Dockerfile}
Contributors
Paulo Kuong (@pkuong)
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
File details
Details for the file airflow-run-0.4.9.tar.gz
.
File metadata
- Download URL: airflow-run-0.4.9.tar.gz
- Upload date:
- Size: 10.5 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/3.4.2 importlib_metadata/3.7.3 pkginfo/1.7.1 requests/2.25.1 requests-toolbelt/0.9.1 tqdm/4.59.0 CPython/3.7.5
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 9a81fb756cb7f2660463a7588026bcc7a88bbabfc3998acf87049f828de157ec |
|
MD5 | 581a4a4132b6ba6be43fb50433292c51 |
|
BLAKE2b-256 | 430225e3fd96986f630c4e9e1b25bb7e41182bac022d53368b9407fc44bb0cc4 |
File details
Details for the file airflow_run-0.4.9-py3-none-any.whl
.
File metadata
- Download URL: airflow_run-0.4.9-py3-none-any.whl
- Upload date:
- Size: 11.0 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/3.4.2 importlib_metadata/3.7.3 pkginfo/1.7.1 requests/2.25.1 requests-toolbelt/0.9.1 tqdm/4.59.0 CPython/3.7.5
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 15c9df7fa34e8b3a56a05ee4ea953476ce759d8a99e489494a5403a00a506c70 |
|
MD5 | 858ae842e282e84591f46287d97c7b4d |
|
BLAKE2b-256 | c0d644ce2e4ad0cd5dc3ab5bb417524fa40a37ced7c16cb0d5d72e8b517f0f74 |