Skip to main content

Dynamically generate Buildkite pipelines

Project description

Buildpipe

A Buildkite plugin to dynamically generate pipelines. Especially useful for monorepos where you want to create dependencies between projects.

Example

Update projects

initial_pipeline.yml

steps:
  - label: ":buildkite:"
    plugins:
      - jwplayer/buildpipe#v0.8.0:
          dynamic_pipeline: dynamic_pipeline.yml

dynamic_pipeline.yml

projects:
 - label: project1
   path: project1/  # changes in this dir will trigger steps for project1
   skip: deploy*  # skip steps with label matching deploy* (e.g. deploy-prd)
 - label: project2
   skip: test
   path:
      - project2/
      - project1  # you can trigger a project using multiple paths
 - label: project3
   skip:  # you can skip a list of projects
     - test
     - deploy-stg
   path: project3/somedir/  # subpaths can also be triggered
steps:  # the same schema as regular buildkite pipeline steps
  - label: test
    env:
      BUILDPIPE_SCOPE: project  # this variable ensures a test step is generated for each project
    command:
      - cd $$BUILDPIPE_PROJECT_PATH  # BUILDPIPE_PROJECT_PATH will be set by buildpipe
      - make test
  - wait
  - label: build
    branches: "master"
    env:
      BUILDPIPE_SCOPE: project
    command:
      - cd $$BUILDPIPE_PROJECT_PATH
      - make build
      - make publish-image
    agents:
      - queue=build
  - wait
  - label: tag
    branches: "master"
    command:
      - make tag-release
  - wait
  - label: deploy-stg
    branches: "master"
    env:
      BUILDPIPE_SCOPE: project
    command:
      - cd $$BUILDPIPE_PROJECT_PATH
      - make deploy-staging
  - wait
  - block: ":rocket: Release!"
    branches: "master"
  - wait
  - label: deploy-prd
    branches: "master"
    env:
      BUILDPIPE_SCOPE: project
    command:
      - cd $$BUILDPIPE_PROJECT_PATH
      - make deploy-prod

The above pipelines specify the following:

  • There are three projects to track in the repository.
  • The env variable BUILDPIPE_SCOPE: project tells buildpipe to generate a step for each project if that project changed.
  • The skip option will skip any step label matching deploy*.
  • The env variable BUILDPIPE_PROJECT_PATH is created by buildpipe as the project's path. If multiple paths are specified for a project, it's the first path.

Full working example

For a full working example, check out Buildkite Monorepo Example.

Configuration

Plugin

Option Required Type Default Description
default_branch No string master Default branch of repository
diff_pr No string Override command for non-default branch (see below for a better explanation of the defaults)
diff_default No string Override command for default branch (see below for a better explanation of the defaults)
dynamic_pipeline Yes string The name including the path to the pipeline that contains all the actual steps
log_level No string INFO The Level of logging to be used by the python script underneath; pass DEBUG for verbose logging if errors occur

Project schema

Option Required Type Default Description
label Yes string Project label
path Yes array The path(s) that specify changes to a project
skip No array Exclude steps that have labels that match the rule

Other useful things to note:

  • Option skip make use of Unix shell-style wildcards (Look at .gitignore files for inspiration)
  • If multiple paths are specified, the environment variable BUILDPIPE_PROJECT_PATH will be the first path.

diff_ commands

Depending on your merge strategy, you might need to use different diff command.

Buildpipe assumes you are using a merge strategy on the default branch, which is assumed to be master.

The command for the non-default branch (e.g. when you have a PR up) is:

git log --name-only --no-merges --pretty=format: origin..HEAD

The command for the default branch you merge to is currently:

git log -m -1 --name-only --pretty=format: $BUILDKITE_COMMIT

Requirements

Python3 is currently required, but we are planning to convert buildpipe to a binary using Go.

Just make sure to install Python3 in your agent bootstrap script or Dockerfile.

Cloudformation bootstrap script

# Install python3
yum -y install python3 python3-pip
pip3 install -U setuptools wheel

Agent Dockerfile

FROM buildkite/agent:3.0

RUN apk add --no-cache \
  # Languages
  python3 py-setuptools

Troubleshooting

Buildpipe is incorrectly showing project as changed

Buildkite doesn't by default do clean checkouts. To enable clean checkouts set the BUILDKITE_CLEAN_CHECKOUT environment variable. An example is to modify the pre-checkout hook, .buildkite/hooks/pre-checkout:

#!/bin/bash
set -euo pipefail

echo '--- :house_with_garden: Setting up pre-checkout'

export BUILDKITE_CLEAN_CHECKOUT="true"

Testing

make test

License

MIT

Acknowledgements

The rewrite to a plugin was inspired by git-diff-conditional-buildkite-plugin.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for buildpipe, version 0.8.0
Filename, size File type Python version Upload date Hashes
Filename, size buildpipe-0.8.0-py2.py3-none-any.whl (7.3 kB) File type Wheel Python version py2.py3 Upload date Hashes View
Filename, size buildpipe-0.8.0.tar.gz (110.4 kB) File type Source Python version None Upload date Hashes View

Supported by

Pingdom Pingdom Monitoring Google Google Object Storage and Download Analytics Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN DigiCert DigiCert EV certificate StatusPage StatusPage Status page