Skip to main content
Help us improve PyPI by participating in user testing. All experience levels needed!

Dynamically generate Buildkite pipelines

Project description

Build Status Documentation Status PyPI Version

Buildpipe allows you to dynamically generate your Buildkite pipelines so that you can:

  • Manage continuous deployment logic such as only deploying during business hours
  • Maintain monorepos by only looking at git changes in specified projects
  • Specify dependencies between projects so that their steps are concurrent

Install

pip install buildpipe

Example

# trigger deploy steps on master during business hours
deploy:
  branch: master
  timezone: US/Eastern
  allowed_hours_regex: '9|1[0-7]'
  allowed_weekdays_regex: '[1-5]'
  blacklist_dates_regex: '\d{4}\-(01\-01|12\-31)'
# ignore certain files from triggering steps with fnmatch
ignore:
  - '*.md'
  - 'pyproject/*.ini'
stairs:
  - name: test
    scope: project
    buildkite:
      command:
        - cd $$PROJECT_PATH
        - make test
  - name: build
    scope: project
    emoji: ":docker:"
    tags:
      - docker-only
    buildkite:
      agents:
        - queue=build
      branches: master
      command:
        - make build
        - make publish-image
  - name: tag
    scope: stair
    emoji: ":github:"
    buildkite:
      branches: master
      command: make tag
  - name: deploy-staging
    scope: project
    emoji: ":shipit:"
    deploy: true
    buildkite:
      branches: master
      command:
        - cd $$PROJECT_PATH
        - make deploy-staging
  - name: deploy-prod
    scope: project
    emoji: ":shipit:"
    deploy: true
    buildkite:
      branches: master
      command:
        - cd $$PROJECT_PATH
        - make deploy-prod
projects:
  - name: pyproject
    path: pyproject
    emoji: ":python:"
    tags:
      - docker-only
    dependencies:
      - jsproject
  - name: jsproject
    path: jsproject
    emoji: ":javascript:"
    skip_stairs:
      - deploy-staging

The above buildpipe config file specifies the following:

  • There are two projects to track in the repo: jsproject and pyproject.
  • A stair is a group of steps. It can have a scope of “project” or “stair”. Scope “project” creates a step for each project changed while scope “stair” creates only one step.
  • You can also limit a stair’s scope using tag rules. For example, pyproject has tag “docker-only” and so will include the build step; but jsproject won’t have that step.
  • Any git file changes that are subpaths of either project’s path will trigger steps for each project.
  • In addition, pyproject has path jsproject as a dependency: any changes in jsproject will trigger steps for pyproject to be included in the pipeline. Note dependencies are paths and not projects.
  • Stairs with “deploy: true” will only happen in master branch between 9am and 5pm ET during weekdays that are not New Year’s Eve or Day.
  • Project jsproject will never create step deploy-staging.
  • Files ending with .md or .ini files under pyproject will be ignore from triggering deploy steps.

In the above config, if only files under pyproject were touched and the merge happened during business hours, then buildpipe would create the following steps:

steps:
  - wait
  - command:
    - cd $$PROJECT_PATH
    - make test
    env:
      PROJECT_NAME: pyproject
      PROJECT_PATH: pyproject
      STAIR_NAME: test
      STAIR_SCOPE: project
    label: 'test pyproject :python:'
  - wait
  - agents:
    - queue=build
    branches: master
    command:
    - cd $$PROJECT_PATH
    - make build
    - make publish-image
    env:
      PROJECT_NAME: pyproject
      PROJECT_PATH: pyproject
      STAIR_NAME: build
      STAIR_SCOPE: project
    label: 'build pyproject :docker:'
  - wait
  - branches: master
    command: make tag
    env:
      STAIR_NAME: tag
      STAIR_SCOPE: stair
    label: 'tag :github:'
  - wait
  - branches: master
    command:
    - cd $$PROJECT_PATH
    - make deploy-staging
    concurrency: 1
    concurrency_group: deploy-staging-pyproject
    env:
      PROJECT_NAME: pyproject
      PROJECT_PATH: pyproject
      STAIR_NAME: deploy-staging
      STAIR_SCOPE: project
    label: 'deploy-staging pyproject :shipit:'
  - wait
  - branches: master
    command:
    - cd $$PROJECT_PATH
    - make deploy-prod
    concurrency: 1
    concurrency_group: deploy-prod-pyproject
    env:
      PROJECT_NAME: pyproject
      PROJECT_PATH: pyproject
      STAIR_NAME: deploy-prod
      STAIR_SCOPE: project
    label: 'deploy-prod pyproject :shipit:'

Set up

In the Buildkite pipeline settings UI you just have to add the following in “Commands to run”:

buildpipe -i path/to/buildpipe.yml -o pipeline.yml
buildkite-agent pipeline upload pipeline.yml

Testing

make test

License

MIT

Project details


Release history Release notifications

This version
History Node

0.2.5

History Node

0.2.4

History Node

0.2.3

History Node

0.2.2

History Node

0.2.1

History Node

0.2.0

History Node

0.1.3

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Filename, size & hash SHA256 hash help File type Python version Upload date
buildpipe-0.2.5-py2.py3-none-any.whl (7.1 kB) Copy SHA256 hash SHA256 Wheel 3.6 May 14, 2018
buildpipe-0.2.5.tar.gz (18.9 kB) Copy SHA256 hash SHA256 Source None May 14, 2018

Supported by

Elastic Elastic Search Pingdom Pingdom Monitoring Google Google BigQuery Sentry Sentry Error logging CloudAMQP CloudAMQP RabbitMQ AWS AWS Cloud computing Fastly Fastly CDN DigiCert DigiCert EV certificate StatusPage StatusPage Status page