Custom cdk constructs for apache airflow
Project description
airflow-cdk
This project makes it simple to deploy airflow via ECS fargate using the aws cdk in Python.
Usage
pip install airflow-cdk
from aws_cdk import core
from airflow_cdk import FargateAirflow
app = core.App()
FargateAirflow(
app,
"airflow-cdk",
postgres_password="replacethiswithasecretpassword")
app.synth()
cdk deploy
That's it.
Components
The following aws resources will be deployed as ecs tasks within the same cluster and vpc by default:
- an airflow webserver task
- and an internet-facing application load-balancer
- an airflow scheduler task
- an airflow worker task
- (note) it will auto-scale based on cpu and memory usage up to a total of 16 instances at a time by default starting from 1
- a rabbitmq broker
- an rds instance
- an s3 bucket for logs
TODOs
- create a custom component to deploy airflow to an ec2 cluster as opposed to fargate
- improve documentation
- (possibly) subsume the airflow stable helm chart as a cdk8s chart
- (possibly) a flower service
Contributions Welcome!
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
airflow_cdk-0.3.2.tar.gz
(10.9 kB
view hashes)
Built Distribution
Close
Hashes for airflow_cdk-0.3.2-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | a695abd0cce07d7c12332bba6066c5ce0679e921ec467897459c5669eba98c44 |
|
MD5 | 8959ab8e64137b2c67071f3566c37c31 |
|
BLAKE2b-256 | 5415704dcc73e94bded11b95739d2437012d328197def8b7441aee9b9ee9a6f1 |