Custom cdk constructs for apache airflow
Project description
airflow-cdk
This project makes it simple to deploy airflow via ECS fargate using the aws cdk in Python.
It is meant for rapid prototyping, you will need to do some work to make it production-ready.
Usage
pip install airflow-cdk
from aws_cdk import core
from airflow_cdk.infra import FargateAirflow
app = core.App()
FargateAirflow(
app,
"airflow-cdk",
postgres_password="replacethiswithasecretpassword")
app.synth()
cdk deploy
That's it.
Components
The following aws resources will be deployed as ecs tasks within the same cluster and vpc by default:
- an airflow webserver task
- and an internet-facing application load-balancer
- an airflow scheduler task
- an airflow worker task
- (note) it will auto-scale based on cpu and memory usage up to a total of 16 instances at a time by default starting from 1
- a rabbitmq broker
- an rds instance
- an s3 bucket for logs
TODOs
- create a custom component to deploy airflow to an ec2 cluster as opposed to fargate
- improve documentation
- (possibly) subsume the airflow stable helm chart as a cdk8s chart
- (possibly) a flower service
Contributions Welcome!
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
airflow_cdk-0.1.8.tar.gz
(9.4 kB
view hashes)
Built Distribution
Close
Hashes for airflow_cdk-0.1.8-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 887b17b49baf5c03894e736d37c281fda2c0988ffbb24f73423b381fbaa4af92 |
|
MD5 | f23d518af697cd659043e326830cbd2b |
|
BLAKE2b-256 | 8fce8e1e0c775bb9252e4ceb11090a3844aa6600ad7f6bdd698b7e3cc759fef4 |