CDK pipelines provides constructs for Waves, Stages using only native CDK stack dependencies
Project description
CDK Express Pipeline
Introduction
CDK Express Pipelines is a library that allows you to define your pipelines in CDK native method. It is built on top of the AWS CDK and is an alternative to AWS CDK Pipelines that is build system agnostic.
Sponsors
Features
- Works on any system for example your local machine, GitHub, GitLab, etc.
- Uses the
cdk deploy
command to deploy your stacks - It's fast. Make use of concurrent/parallel Stack deployments
- Stages and Waves are plain classes, not constructs, they do not change nested Construct IDs (like CDK Pipelines)
- Supports TS and Python CDK
How does it work?
This library makes use of the fact that the CDK CLI computes the dependency graph of your stacks and deploys them in
the correct order. It creates the correct dependency graph between Waves, Stages and Stacks with the help of the
native .addDependency
method of the CDK Stack. The cdk deploy '**'
command will deploy all stacks in the correct
order.
Deployment Order
The Wave, Stage and Stack order is as follows:
- Waves are deployed sequentially, one after the other.
- Stages within a Wave are deployed in parallel.
- Stacks within a Stage are deployed in order of stack dependencies within a Stage.
For example, the following definition of Waves, Stages and Stacks as in CDK Express Pipelines:
Will create a dependency graph as follows:
When used with cdk deploy '**' --concurrency 10
, it will deploy all stacks in parallel, 10 at a time, where possible
while still adhering to the dependency graph. Stacks will be deployed in the following order:
✨ Deployment order visualized ✨
Selective Deployment
Leverages a consistent and predictable naming convention for Stack IDs. A Stack ID consists of the Wave, Stage and original Stack ID. This enables us to target Waves, Stages or individual stacks for deployment. For example, given the following stack IDs:
Wave1_Stage1_StackA
Wave1_Stage1_StackB
Wave1_Stage1_StackC
Wave1_Stage2_StackD
Wave2_Stage1_StackE
Wave2_Stage1_StackF
It makes targeted deployments easy:
- Deploy Wave1:
cdk deploy 'Wave1_*'
deploys all stacks inWave1
- Deploy Wave1 Stage1:
cdk deploy 'Wave1_Stage1_*'
deploys all stacks inWave1_Stage1
- Deploy Wave1 Stage1 StackA:
cdk deploy 'Wave1_Stage1_StackA'
deploys onlyWave1_Stage1_StackA
[!IMPORTANT] When targeting specific stacks be sure to pass the
--exclusively
flag to thecdk deploy
command to only deploy the specified stacks and not its dependencies.
Benefits of selecting a specific Wave, Stage or Stack over the all '**'
method:
- While developing, you can speed up deployments from your local machine by deploying only what you are working on.
- When deploying with a CI/CD system, you can have additional logic between them. For example, you can place a
manual approval step between
Wave1
andWave2
.
Installation
TS
npm install cdk-express-pipelines
Then import the library in your code:
import { CdkExpressPipeline } from 'cdk-express-pipelines';
Python
pip install cdk-express-pipelines
Then import the library in your code:
from cdk_express_pipelines import CdkExpressPipeline
Usage
The ExpressStack
extends the cdk.Stack
class and has a very similar signature, only taking an extra stage
parameter. There are multiple ways to build your pipeline, it involves creating the Pipeline, adding Waves, Stages and
Stacks to your Stages and then calling .synth()
on the Pipeline. See the alternative expand sections for other
methods.
Stack Definition:
class StackA extends ExpressStack {
constructor(scope: Construct, id: string, stage: ExpressStage, stackProps?: StackProps) {
super(scope, id, stage, stackProps);
new cdk.aws_sns.Topic(this, 'MyTopic');
// ... more resources
}
}
class StackB extends ExpressStack {
//... similar to StackA
}
class StackC extends ExpressStack {
//... similar to StackA
}
1️⃣ Pipeline Definition:
const app = new App();
const expressPipeline = new CdkExpressPipeline();
// === Wave 1 ===
const wave1 = expressPipeline.addWave('Wave1');
// --- Wave 1, Stage 1---
const wave1Stage1 = wave1.addStage('Stage1');
const stackA = new StackA(app, 'StackA', wave1Stage1);
const stackB = new StackB(app, 'StackB', wave1Stage1);
stackB.addExpressDependency(stackA);
// === Wave 2 ===
const wave2 = expressPipeline.addWave('Wave2');
// --- Wave 2, Stage 1---
const wave2Stage1 = wave2.addStage('Stage1');
new StackC(app, 'StackC', wave2Stage1);
expressPipeline.synth([
wave1,
wave2,
]);
The stack deployment order will be printed to the console when running cdk
commands:
ORDER OF DEPLOYMENT
🌊 Waves - Deployed sequentially
🔲 Stages - Deployed in parallel, all stages within a wave are deployed at the same time
📄 Stack - Dependency driven, will be deployed after all its dependent stacks, denoted by ↳ below it, is deployed
🌊 Wave1
🔲 Stage1
📄 StackA (Wave1_Stage1_StackA)
📄 StackB (Wave1_Stage1_StackB)
↳ StackA
🌊 Wave2
🔲 Stage1
📄 StackC (Wave2_Stage1_StackC)
2️⃣ Pipeline Definition Alternative - Stacks Nested in Stages:
const app = new App();
class Wave1 extends ExpressWave {
constructor() {
super('Wave1');
}
}
class Wave1Stage1 extends ExpressStage {
constructor(wave1: Wave1) {
super('Stage1', wave1);
const stackA = new StackA(app, 'StackA', this);
const stackB = new StackB(app, 'StackB', this);
stackB.addExpressDependency(stackA);
}
}
class Wave2 extends ExpressWave {
constructor() {
super('Wave2');
}
}
class Wave2Stage1 extends ExpressStage {
constructor(wave2: Wave2) {
super('Stage1', wave2);
new StackC(app, 'StackC', this);
}
}
const expressPipeline = new CdkExpressPipeline();
const wave1 = new Wave1();
new Wave1Stage1(wave1);
const wave2 = new Wave2();
new Wave2Stage1(wave2);
expressPipeline.synth([wave1, wave2]);
3️⃣ Pipeline Definition Alternative - Extending all without nesting:
const app = new App();
// --- Custom Wave Class ---
class MyExpressWave extends ExpressWave {
constructor(props: ExpressWaveProps) {
super('My' + props.id);
}
}
// --- Custom Stage Class ---
class MyExpressStage extends ExpressStage {
constructor(id: string, wave: MyExpressWave, stacks?: MyExpressStack[]) {
super('My' + id, wave, stacks);
}
}
// --- Custom Stack Class ---
class MyExpressStack extends ExpressStack {
constructor(scope: Construct, id: string, stage: MyExpressStage, stackProps?: StackProps) {
super(scope, 'My' + id, stage, stackProps);
}
}
const expressPipeline = new CdkExpressPipeline();
const wave1 = new MyExpressWave({ id: 'Wave1' });
const wave1Stage1 = new MyExpressStage('Stage1', wave1);
const stackA = new MyExpressStack(app, 'StackA', wave1Stage1);
expressPipeline.synth([wave1]);
expect(stackA.id).toBe('MyWave1_MyStage1_MyStackA');
Legacy Usage
The CdkExpressPipelineLegacy
class can be used when you do not want/can not use the ExpressStack
class and have to
stick to the CDK Stack
class.
[!WARNING] Always use non-legacy classes for greenfield projects. Only use the Legacy classes if you have no other choice.
The following features are not available when using the Legacy classes:
- Enforcing Wave, Stage and Stack names do not include the
separator
character. - Enforcing that a Stack in Stage 1 can not depend on a Stack in Stage 2.
- Printing stack dependencies within a Stage. Since we do not know what stage a stack belongs to, it's not possible to print the dependencies of stacks of only that stage and not others.
- If a consistent naming convention has not been followed for Stacks, it might not be possible to target all stacks in a
stage or a wave. Deployment will have to always target all stacks with
"**"
. - Stack ids are not changed and have to be unique across all stacks in the CDK app, whereas with the non-legacy classes, stack ids only have to be unique within a Wave.
Stack Definition:
class StackA extends cdk.Stack {
constructor(scope: Construct, id: string, props?: cdk.StackProps) {
super(scope, id, props);
new cdk.aws_sns.Topic(this, 'MyTopicA');
// ... more resources
}
}
class StackB extends cdk.Stack {
// ... similar to StackA
}
class StackC extends cdk.Stack {
}
1️⃣ Pipeline Definition:
const app = new App();
const expressPipeline = new CdkExpressPipelineLegacy();
/* === Wave 1 === */
/* --- Wave 1, Stage 1--- */
const stackA = new StackA(app, 'StackA');
const stackB = new StackB(app, 'StackB');
stackB.addDependency(stackA);
// === Wave 2 ===
/* --- Wave 2, Stage 1--- */
const stackC = new StackC(app, 'StackC');
expressPipeline.synth([
{
id: 'Wave1',
stages: [{
id: 'Stage1',
stacks: [
stackA,
stackB,
],
}],
},
{
id: 'Wave2',
stages: [{
id: 'Stage1',
stacks: [
stackC,
],
}],
},
]);
The stack deployment order will be printed to the console when running cdk
commands:
ORDER OF DEPLOYMENT
🌊 Waves - Deployed sequentially
🔲 Stages - Deployed in parallel, all stages within a wave are deployed at the same time
📄 Stack - Dependency driven
🌊 Wave1
🔲 Stage1
📄 StackA
📄 StackB
🌊 Wave2
🔲 Stage1
📄 StackC
2️⃣ Pipeline Definition Alternative - method builder:
const app = new App();
const expressPipeline = new CdkExpressPipelineLegacy();
/* === Wave 1 === */
const wave1 = expressPipeline.addWave('Wave1');
/* --- Wave 1, Stage 1--- */
const wave1Stage1 = wave1.addStage('Stage1');
const stackA = wave1Stage1.addStack(new StackA(app, 'StackA'));
const stackB = wave1Stage1.addStack(new StackB(app, 'StackB'));
stackB.addDependency(stackA);
// === Wave 2 ===
const wave2 = expressPipeline.addWave('Wave2');
/* --- Wave 2, Stage 1--- */
const wave2Stage1 = wave2.addStage('Stage1');
wave2Stage1.addStack(new StackC(app, 'StackC'));
expressPipeline.synth([
wave1,
wave2,
]);
Builds System Templates/Examples
Local
These examples all assume a project created with the default structure of the CDK CLI
command cdk init app --language typescript
.
These example are taken from the demo TS project: https://github.com/rehanvdm/cdk-express-pipeline-demo-ts
Diff commands
# Diffs all stacks
cdk diff '**' --profile YOUR_PROFILE
# Diffs only specific stacks in a Wave
cdk diff 'Wave1_*' --profile YOUR_PROFILE --exclusively
# Diffs only specific stacks of a Stage in a Wave
cdk diff 'Wave1_Stage1_*' --profile YOUR_PROFILE --exclusively
# Diffs only a specific stack
cdk diff 'Wave1_Stage1_StackA' --profile YOUR_PROFILE --exclusively
Deploy commands
# Deploys all stacks in correct order
cdk deploy '**' --profile YOUR_PROFILE --concurrency 10 --require-approval never
# Deploys only specific stacks in a Wave in correct order
cdk deploy 'Wave1_*' --profile YOUR_PROFILE --exclusively --concurrency 10 --require-approval never
# Deploys only specific stacks of a Stage in a Wave in correct order
cdk deploy 'Wave1_Stage1_*' --profile YOUR_PROFILE --exclusively --concurrency 10 --require-approval never
# Deploys only a specific stack
cdk deploy 'Wave1_Stage1_StackA' --profile YOUR_PROFILE --exclusively --concurrency 10 --require-approval never
GitHub Workflows
These examples all assume a project created with the default structure of the CDK CLI
command cdk init app --language typescript
.
These example are taken from the demo TS project: https://github.com/rehanvdm/cdk-express-pipeline-demo-ts
.github/workflows/diff.yml
Does a build and CDK Diff on PR open and push, the cdk diff
output can be viewed in the action run logs.
name: Diff
on:
pull_request:
types: [ opened, synchronize ]
workflow_dispatch: { }
env:
FORCE_COLOR: 1
jobs:
deploy:
name: CDK Diff and Deploy
runs-on: ubuntu-latest
permissions:
actions: write
contents: read
id-token: write
steps:
- name: Checkout repo
uses: actions/checkout@v4
- name: Set up node
uses: actions/setup-node@v3
with:
node-version: 20
cache: npm
- name: Install dependencies
run: npm install ci
# TODO: Alternatively use an AWS IAM user and set the credentials in GitHub Secrets (less secure than GH OIDC below)
- name: Configure AWS credentials
uses: aws-actions/configure-aws-credentials@v4
with:
role-to-assume: # TODO: Your role to assume
aws-region: # TODO: your region
- name: CDK diff
run: npm run cdk -- diff '**'
Produces the following output in the GitHub Action logs:
.github/workflows/deploy.yml
Does a build, CDK Diff and Deploy when a push happens on the main
branch.
name: Deploy
on:
push:
branches:
- main
env:
FORCE_COLOR: 1
jobs:
deploy:
name: CDK Diff and Deploy
runs-on: ubuntu-latest
permissions:
actions: write
contents: read
id-token: write
steps:
- name: Checkout repo
uses: actions/checkout@v4
- name: Set up node
uses: actions/setup-node@v3
with:
node-version: 20
cache: npm
- name: Install dependencies
run: npm install ci
# TODO: Alternatively use an AWS IAM user and set the credentials in GitHub Secrets (less secure than GH OIDC below)
- name: Configure AWS credentials
uses: aws-actions/configure-aws-credentials@v4
with:
role-to-assume: # TODO: Your role to assume
aws-region: # TODO: your region
- name: CDK diff
run: npm run cdk -- diff '**'
- name: CDK deploy
run: npm run cdk -- deploy '**' --require-approval never --concurrency 10
Produces the following output in the GitHub Action logs:
.github/workflows/deploy-advance.yml
The synth
job builds the CDK app and saves the cloud assembly to the ./cloud_assembly_output
directory. The whole
repo with installed NPM packages and the cloud assembly is then cached. This job of the pipeline does not have access
to any AWS Secrets, the installing of packages and building is decoupled from the deployment improving security.
The wave1
and wave2
jobs fetches the cloud assembly from the cache and then does a CDK Diff and Deploy on only their
stacks. The wave1
job targets all the stacks that start with Wave1_
and the wave2
job targets all the stacks that
start with Wave2_
. It is important to add the --exclusively
flag to only focus on the specified stacks and not its
dependencies.
name: Deploy Advance
on:
push:
branches:
- main
workflow_dispatch: { } # While testing only
env:
FORCE_COLOR: 1
jobs:
synth:
name: Build and CDK Synth
runs-on: ubuntu-latest
permissions:
actions: write
contents: read
id-token: write
steps:
- name: Checkout repo
uses: actions/checkout@v4
- name: Set up node
uses: actions/setup-node@v3
with:
node-version: 20
cache: npm
- name: Install dependencies
run: npm install ci
- name: CDK Synth
run: npm run cdk -- synth --output ./cloud_assembly_output
- name: Cache CDK Assets
uses: actions/cache/save@v4
with:
path: ./
key: "cdk-assets-${{ github.sha }}"
wave1:
name: Wave 1
needs:
- synth
runs-on: ubuntu-latest
permissions:
actions: write
contents: read
id-token: write
steps:
- name: Fetch CDK Assets
uses: actions/cache/restore@v4
with:
path: ./
key: "cdk-assets-${{ github.sha }}"
- name: Configure AWS credentials
uses: aws-actions/configure-aws-credentials@v4
with:
role-to-assume: arn:aws:iam::581184285249:role/githuboidc-git-hub-deploy-role
aws-region: eu-west-1
- name: CDK diff
run: npm run cdk -- diff 'Wave1_*' --exclusively --app ./cloud_assembly_output
- name: CDK deploy
run: npm run cdk -- deploy 'Wave1_*' --require-approval never --concurrency 10 --exclusively --app ./cloud_assembly_output
# Manual approval
wave2:
name: Wave 2
needs:
- wave1
runs-on: ubuntu-latest
permissions:
actions: write
contents: read
id-token: write
steps:
- name: Fetch CDK Assets
uses: actions/cache/restore@v4
with:
path: ./
key: "cdk-assets-${{ github.sha }}"
# TODO: Alternatively use an AWS IAM user and set the credentials in GitHub Secrets (less secure than GH OIDC below)
- name: Configure AWS credentials
uses: aws-actions/configure-aws-credentials@v4
with:
role-to-assume: # TODO: Your role to assume
aws-region: # TODO: your region
- name: CDK diff
run: npm run cdk -- diff 'Wave2_*' --exclusively --app ./cloud_assembly_output
- name: CDK deploy
run: npm run cdk -- deploy 'Wave2_*' --require-approval never --concurrency 10 --exclusively --app ./cloud_assembly_output
Produces the following output in the GitHub Action logs:
GitLab
TODO...
Any other build system
...
Demo Projects
Docs
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for cdk_express_pipeline-1.0.2.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | 8423e78c94f73dddcaee7b498561fe1c277720e8e58d5be15a75256f8daa0e73 |
|
MD5 | 714b49add834c0857ac3a6b17a814450 |
|
BLAKE2b-256 | 51863c31b2f6c9c2c9bd07829718aebcb7b477ac7f9ec1ae5d9a2702dc35337c |
Hashes for cdk_express_pipeline-1.0.2-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | a6936c44ab772b8f60de95ad63d904b81aa9f1451154b46c355e6cbc087ae5fe |
|
MD5 | 3648cb89b66b554376b66af59354c013 |
|
BLAKE2b-256 | 1d0d0e2af685490748b8e17483dc620f5230176b5f585ae25a2a0f462e82cb01 |