Skip to main content

Guard gitlab jobs from multiple simultaneous executions

Project description

pipeline status

gitlab-job-guard

Guard pipeline jobs from multiple simultaneous executions

$ PRIVATE_TOKEN="$GITLAB_API_TOKEN" gitlab-job-guard -w 3600
$ my-unguarded-deployment-task --to=production

gitlab-job-guard will block if it detects other pipelines running for the current project to avoid multiple pipelines from clobbering up a deployment/environment.

While gitlab will auto-cancel redundant, pending pipelines for the same branch by default - this is not the case for multiple pipelines from different branches targeting a particular deployment/environment. Gitlab has no way to detect or control these user-defined branch-to-environment mappings and this means environments can easily be left in an unsafe/broken state. (e.g. terraform apply or ansible, etc from different pipelines running at the same time).

gitlab-job-guard uses the Gitlab API to determine if existing pipelines are scheduled and to backoff-and-retry until it is safe to proceed. Conflicts are detected by user-defined matches on pipeline ref names (branch, tag, etc) and/or pipeline status.

Usage

The simplest usage would likely be placing gitlab-job-guard in a before_script section in your gitlab-ci.yml to protect all jobs (though this can slow things down).

before_script:
  - PRIVATE_TOKEN="$GITLAB_API_TOKEN" gitlab-job-guard

Though often, this is only needed to guard jobs that share common state/data (i.e. a deployment environment, an artifact build/release, etc).

deploy-production:
  stage: deploy
  script:
    - PRIVATE_TOKEN="$GITLAB_API_TOKEN" gitlab-job-guard
    - my-unguarded-deployment-task --to=production

or to guard something like a terraform job running for tags.

provision-infrastructure:
  stage: provision
  script:
    - export PRIVATE_TOKEN="$GITLAB_API_TOKEN"
    - gitlab-job-guard --guard-ref-regex='^v[0-9\.]+'  # Regex matches tags
    - terraform plan  ...
    - terraform apply ...
  only:
    - tags

Other usages

To hold jobs for a collisions on pattern matches on the ref/branch name.

gitlab-job-guard -c=^master$                # Match branch names matching 'master' exactly

gitlab-job-guard -c=^(master|dev(elop)?)$   # Match any of the mainline branches

gitlab-job-guard -c=^(feature|release|hotfix)/  # Match any gitflow transient branch prefixes

gitlab-job-guard -c=^[0-9]\-                # Match branch names beginning with a number
                                            # and dash ignoring all other text.
                                            # e.g. a gitlab branch made from an issue

gitlab-job-guard -c=^v?[\d.]+$              # Match (semver) tags like v1.0.9, 2.0

gitlab-job-guard -c=^environment/           # Match any environment deployments?

gitlab-job-guard -c=^environment/dc1.+      # Match environment deployments to DC1?

gitlab-job-guard -c="$CI_BUILD_REF_NAME"    # Match current branch name (partially).
                                            # i.e. 'master' matches 'feature/master-document'

gitlab-job-guard -c="^$CI_BUILD_REF_NAME$"  # Match current branch name (exactly).
                                            # i.e. 'master' does not match 'master-deployment'

gitlab-job-guard -c='.+' -s='running|pending'  # Match any pipeline in running or pending state

To hold a job for a collision on part of the ref name (e.g. on branch prefix such as feature/ or hotfix/ or release/, etc a la gitflow).

# Assuming CI_BUILD_REF_NAME=feature/foo

CI_BUILD_REF_PREFIX=$(echo "$CI_BUILD_REF_NAME" | sed -r 's@(.+/)(.+)@\1@')
# CI_BUILD_REF_PREFIX now contains 'feature/'

gitlab-job-guard -c="^$CI_BUILD_REF_PREFIX" -s='running|pending'

TODO

For long pipelines, this solution can have subtle consequences with growing queues and increased contention and unpredictability as to which pipeline is the first-past-the-post. An older pipeline taking precedence over newer commits if often not desired and newer pipelines always winning is probably desired.

  • Handle existing conflicting pipelines - cancel them or give-way.
  • Narrow down conflicts to jobs (CI_JOB_NAME) or stages (CI_JOB_STAGE) so that other parts of the pipelines that do not share state are allowed to run freely.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for gitlab-job-guard, version 0.0.3
Filename, size File type Python version Upload date Hashes
Filename, size gitlab_job_guard-0.0.3-py3-none-any.whl (19.7 kB) File type Wheel Python version py3 Upload date Hashes View
Filename, size gitlab-job-guard-0.0.3.tar.gz (7.9 kB) File type Source Python version None Upload date Hashes View

Supported by

AWS AWS Cloud computing Datadog Datadog Monitoring DigiCert DigiCert EV certificate Facebook / Instagram Facebook / Instagram PSF Sponsor Fastly Fastly CDN Google Google Object Storage and Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Salesforce Salesforce PSF Sponsor Sentry Sentry Error logging StatusPage StatusPage Status page