Custom cdk constructs for apache airflow
Project description
airflow-cdk
This project makes it simple to deploy airflow via ECS fargate using the aws cdk in Python.
Usage
pip install airflow-cdk
from aws_cdk import core
from airflow_cdk import FargateAirflow
app = core.App()
FargateAirflow(
app,
"airflow-cdk",
postgres_password="replacethiswithasecretpassword")
app.synth()
cdk deploy
That's it.
Components
The following aws resources will be deployed as ecs tasks within the same cluster and vpc by default:
- an airflow webserver task
- and an internet-facing application load-balancer
- an airflow scheduler task
- an airflow worker task
- (note) it will auto-scale based on cpu and memory usage up to a total of 16 instances at a time by default starting from 1
- a rabbitmq broker
- an rds instance
- an s3 bucket for logs
TODOs
- create a custom component to deploy airflow to an ec2 cluster as opposed to fargate
- improve documentation
- (possibly) subsume the airflow stable helm chart as a cdk8s chart
- (possibly) a flower service
Contributions Welcome!
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
airflow_cdk-0.3.1.tar.gz
(10.6 kB
view hashes)
Built Distribution
Close
Hashes for airflow_cdk-0.3.1-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | c76063fc85ab5f1adeeabf715ceec7fed1182af76dbfae47479cd2afff39d531 |
|
MD5 | 63da78b1c0ab3fcd1e964b6a9ab8d861 |
|
BLAKE2b-256 | 4a6575e25fd11472fdc3ee2bd330809aa7155e72d16b0117e8c7728cd798fade |