Dynamically generate Buildkite pipelines
A Buildkite plugin to dynamically generate pipelines. Especially useful for monorepos where you want to create dependencies between projects.
steps: - label: ":buildkite:" plugins: - jwplayer/buildpipe#v0.8.0: dynamic_pipeline: dynamic_pipeline.yml
projects: - label: project1 path: project1/ # changes in this dir will trigger steps for project1 skip: deploy* # skip steps with label matching deploy* (e.g. deploy-prd) - label: project2 skip: test path: - project2/ - project1 # you can trigger a project using multiple paths - label: project3 skip: # you can skip a list of projects - test - deploy-stg path: project3/somedir/ # subpaths can also be triggered steps: # the same schema as regular buildkite pipeline steps - label: test env: BUILDPIPE_SCOPE: project # this variable ensures a test step is generated for each project command: - cd $$BUILDPIPE_PROJECT_PATH # BUILDPIPE_PROJECT_PATH will be set by buildpipe - make test - wait - label: build branches: "master" env: BUILDPIPE_SCOPE: project command: - cd $$BUILDPIPE_PROJECT_PATH - make build - make publish-image agents: - queue=build - wait - label: tag branches: "master" command: - make tag-release - wait - label: deploy-stg branches: "master" env: BUILDPIPE_SCOPE: project command: - cd $$BUILDPIPE_PROJECT_PATH - make deploy-staging - wait - block: ":rocket: Release!" branches: "master" - wait - label: deploy-prd branches: "master" env: BUILDPIPE_SCOPE: project command: - cd $$BUILDPIPE_PROJECT_PATH - make deploy-prod
The above pipelines specify the following:
- There are three projects to track in the repository.
- The env variable
BUILDPIPE_SCOPE: projecttells buildpipe to generate a step for each project if that project changed.
skipoption will skip any step label matching
- The env variable
BUILDPIPE_PROJECT_PATHis created by buildpipe as the project's path. If multiple paths are specified for a project, it's the first path.
Full working example
For a full working example, check out Buildkite Monorepo Example.
|default_branch||No||string||master||Default branch of repository|
|diff_pr||No||string||Override command for non-default branch (see below for a better explanation of the defaults)|
|diff_default||No||string||Override command for default branch (see below for a better explanation of the defaults)|
|dynamic_pipeline||Yes||string||The name including the path to the pipeline that contains all the actual steps|
|log_level||No||string||INFO||The Level of logging to be used by the python script underneath; pass DEBUG for verbose logging if errors occur|
|path||Yes||array||The path(s) that specify changes to a project|
|skip||No||array||Exclude steps that have labels that match the rule|
Other useful things to note:
skipmake use of Unix shell-style wildcards (Look at .gitignore files for inspiration)
- If multiple paths are specified, the environment variable
BUILDPIPE_PROJECT_PATHwill be the first path.
Depending on your merge strategy, you might need to use different diff command.
Buildpipe assumes you are using a merge strategy on the default branch, which is assumed to be
The command for the non-default branch (e.g. when you have a PR up) is:
git log --name-only --no-merges --pretty=format: origin..HEAD
The command for the default branch you merge to is currently:
git log -m -1 --name-only --pretty=format: $BUILDKITE_COMMIT
Python3 is currently required, but we are planning to convert buildpipe to a binary using Go.
Just make sure to install Python3 in your agent bootstrap script or Dockerfile.
Cloudformation bootstrap script
# Install python3 yum -y install python3 python3-pip pip3 install -U setuptools wheel
FROM buildkite/agent:3.0 RUN apk add --no-cache \ # Languages python3 py-setuptools
Buildpipe is incorrectly showing project as changed
Buildkite doesn't by default do clean checkouts. To enable clean
checkouts set the
BUILDKITE_CLEAN_CHECKOUT environment variable. An
example is to modify the pre-checkout hook,
#!/bin/bash set -euo pipefail echo '--- :house_with_garden: Setting up pre-checkout' export BUILDKITE_CLEAN_CHECKOUT="true"
The rewrite to a plugin was inspired by git-diff-conditional-buildkite-plugin.
Release history Release notifications | RSS feed
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Hashes for buildpipe-0.8.0-py2.py3-none-any.whl