Dynamically generate Buildkite pipelines
Project description
Buildpipe
A Buildkite plugin to dynamically generate pipelines. Especially useful for monorepos where you want to create dependencies between projects.
Example
initial_pipeline.yml
steps:
- label: ":pipeline:"
plugins:
- jwplayer/buildpipe#v0.7.3:
dynamic_pipeline: dynamic_pipeline.yml
projects:
- label: project1
path: project1/ # changes in this dir will trigger steps for project1
- label: project2
skip: deploy* # skip steps with label deploy* (e.g. deploy-prd)
path: project2/
- label: project3
skip:
- test
- build
path:
- project3/
- project2/somedir/ # project3 steps will also be triggered by changes in this dir
dynamic_pipeline.yml
steps:
- label: test
env:
BUILDPIPE_SCOPE: project # this variable ensures a test step is generated for each project
command:
- cd $$BUILDPIPE_PROJECT_PATH
- make test
- wait
- label: build
branches: "master"
env:
BUILDPIPE_SCOPE: project
command:
- cd $$BUILDPIPE_PROJECT_PATH
- make build
- make publish-image
agents:
- queue=build
- wait
- label: tag
branches: "master"
command:
- make tag-release
- wait
- label: deploy-staging
branches: "master"
env:
BUILDPIPE_SCOPE: project
command:
- cd $$BUILDPIPE_PROJECT_PATH
- make deploy-staging
- wait
- block: ":rocket: Release!"
- wait
- label: deploy-prod
branches: "master"
env:
BUILDPIPE_SCOPE: project
command:
- cd $$BUILDPIPE_PROJECT_PATH
- make deploy-prod
The above pipelines specify the following:
- There are three projects to track in the repository.
- The env variable
BUILDPIPE_SCOPE: project
tells buildpipe to generate a step for each project if that project changed. - The
skip
option will skip any step label matchingdeploy*
. - The env variable
BUILDPIPE_PROJECT_PATH
is created by buildpipe as the project's path. If multiple paths are specified for a project, it's the first path.
Configuration
Plugin
Option | Required | Type | Default | Description |
---|---|---|---|---|
dynamic_pipeline | Yes | string | The name including the path to the pipeline that contains all the actual steps | |
diff | No | string | Can be used to override the default commands (see below for a better explanation of the defaults) | |
log_level | No | string | INFO | The Level of logging to be used by the python script underneath; pass DEBUG for verbose logging if errors occur |
projects | Yes | array | List of projects that buildpipe will run steps for |
Project
Option | Required | Type | Default | Description |
---|---|---|---|---|
label | Yes | string | Project label | |
path | Yes | array | The path(s) that specify changes to a project | |
skip | No | array | Exclude steps that have labels that match the rule |
Other useful things to note:
- Option
skip
make use of Unix shell-style wildcards (Look at .gitignore files for inspiration) - If multiple paths are specified, the environment variable
BUILDPIPE_PROJECT_PATH
will be the first path.
diff
command
Depending on your merge strategy, you might need to use different diff command.
Buildpipe assumes you are using a merge strategy on the master branch.
Requirements
Python3 and pip3 are currently required, but we have plant to convert it into a binary with Golang.
Just make sure to install them in your agent bootstrap script or Dockerfile.
Cloudformation bootstrap script
# Install python3
yum -y install python3 python3-pip
pip3 install -U setuptools wheel
Agent Dockerfile
FROM buildkite/agent:3.0
RUN apk add --no-cache \
# Languages
python3 py-setuptools
Troubleshooting
Buildpipe is incorrectly showing project as changed
Buildkite doesn't by default do clean checkouts. To enable clean
checkouts set the BUILDKITE_CLEAN_CHECKOUT
environment variable. An
example is to modify the pre-checkout hook,
.buildkite/hooks/pre-checkout
:
#!/bin/bash
set -euo pipefail
echo '--- :house_with_garden: Setting up pre-checkout'
export BUILDKITE_CLEAN_CHECKOUT="true"
Testing
make test
License
MIT
Acknowledgements
The rewrite to a plugin was inspired by git-diff-conditional-buildkite-plugin.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for buildpipe-0.7.3-py2.py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | a9325f20f3d377f5af0f2029c65182cc18d4977bca2fc766ad96c4e238c685c6 |
|
MD5 | 6a5177f3692fc68bda5665f30636e962 |
|
BLAKE2b-256 | b69faf0db87a4425f9365b20cd6321fa77f1f6b9b41d3dbb9628a79dbd13d689 |