gitlab-ci-generator

This project builds a dynamic gitlab ci file that is intended to be used for monorepos. The concept is to simplify the amount of work that is needed to manage the monorepo.

Installation

You can download and run directly with python or install using:

pip install gitlab-ci-generator

This installs like an executable and you can simply run:

gitlab-ci-generator

Usage

The project takes the following parameters:

This program will load the input yaml file into two distinct dictionaries that are available to the template:

• pipeline_info
• jobs

To run without specifying an output file you can write stdout to a redirect file like so:

gitlab-ci-generator --inputfile ./job-dependency.yml 1> /tmp/test.out

pipeline_info

This section is gets loaded as is into a dictionary so you can easily extend it for your needs. The built-in template uses the following:

pipeline_info:
  shared_includes: <<-- Used by the template to build the include: section of the gitlab ci file.
    - name: rules <<-- List of includes and can be of type local or project.  The rules example here is used in the template in concert with the share_reference_rules: section to define dynamic rules for jobs.
      local: "example/example-config/rules.yml" 
  shared_variables: << -- Used by the template to build the variables: section of the gitlab ci under each job.
    - name: MYVAR <-- The variable name to include
      value: MYVAR_VALUE <-- The value string or a variable for the value 
  shared_reference_rules: <<-- Used by the template to build the rules: section of the gitlab ci under each job.
    - rule_name: standard-rule-if <- The rule_source and rule_name will create a !reference[rule_source, rule_name] value under the rules section of each job.
      rule_source: .job_changes_rules
      only_changes: true
  workflow_rules: <<-- Optional. Used to build the workflow:rules section. Omit this key entirely to keep the built-in MR/schedule/push default rules (needed for MR-triggered child pipelines); supply a list to override with !reference rules; supply an empty list to omit the workflow: section entirely.
    - rule_name: <<-- The rule_source and rule_name will create a !reference[rule_source, rule_name] value under workflow:rules.
      rule_source: <<-- Points the rule_name in the !reference
      when: <<-- Optional.
  pre_job: <<-- Optional section that can be used to specify a pre trigger job to run at the start of your pipeline.
    name: <<-- The name of the pre job.
    stage: <<-- The stage that you want the pre job to run in.
    trigger_job_info: <<-- Info around the pre job.
      project: <<-- gitlab project location for pre job.
      ref: <<-- Pre job branch reference
      file: <<-- Path to the yml file to be triggered.
    rules: <<-- Custom rules for the pre job.
      - rule_name: <<-- The rule_source and rule_name will create a !reference[rule_source, rule_name] value under the rules section of each job.
      - rule_source: <<-- Points the rule_name in the !reference
      only_changes: true <<-- True will run this job only if changes occur.
  final_job: <<-- Optional section that can be used to specify a final trigger job to run at the end of your pipeline.
    name: <<-- The name of the final job.
    stage: <<-- The stage that you want the final job to run in.
    trigger_job_info: <<-- Info around the final job.
      project: <<-- gitlab project location for final job.
      ref: <<-- Final job branch reference
      file: <<-- Path to the yml file to be triggered.
    rules: <<-- Custom rules for the final job.
      - rule_name: <<-- The rule_source and rule_name will create a !reference[rule_source, rule_name] value under the rules section of each job.
      - rule_source: <<-- Points the rule_name in the !reference
      only_changes: true <<-- True will run this job only if changes occur.

jobs

This section is used to define the hierarchy for the gitlab jobs. This section is not extensible and should be structured as follows:

jobs: <<-- List of jobs and dependencies that will build the appropriate changes/needs logic in the generated gitlab ci file.
  - name: example-1 <<-- name: will be the job name in the generated gitlab file and is required. 
    folder: example/example-folder-1 <<-- folder: The subfolder in the gitlab repo that will be monitored for changes and contains the sub-repo in the monorepo.
    gitlab_yml_file: .gitlab-example.yml <<--  gitlab_yml_file: is the pipeline file that will be triggered in the folder: that is specified
    dependent_jobs: <<-- dependent_jobs: is a list of dependent jobs and can have dependent_jobs on its jobs.  In other words, this can go multiple levels deep with dependencies.  
      - name: example-2
        folder: example/example-folder-2
        gitlab_yml_file: .gitlab-example.yml

Sample Input yaml File

This sample can also be found in the gitlab repo for this project at the location example/example-config/job-dependency.yml

---
pipeline_info:
  shared_includes:
    - name: rules
      local: "example/example-config/rules.yml"
  shared_variables:
    - name: MYVAR
      value: MYVAR_VALUE
  workflow_rules:
    - rule_name: merge-request-rule-if
      rule_source: .job_changes_rules
    - rule_name: schedule-rule-if
      rule_source: .job_changes_rules
  shared_reference_rules:
    - rule_name: standard-rule-if
      rule_source: .job_changes_rules
      only_changes: true
    - rule_name:  manual-rule-if
      rule_source: .job_changes_rules
      when: manual
      only_changes: true
    - rule_name: schedule-rule-if
      rule_source: .job_changes_rules
      only_changes: false
  pre_job:
    name: pre-job-example
    stage: pre-build
    trigger_job_info:
      local: 'example/example-pre-job/.gitlab-example.yml'
    rules:
      - rule_name: standard-rule-if
        rule_source: .job_changes_rules
        only_changes: true
  final_job:
    name: cut-tag
    stage: post-deploy
    trigger_job_info:
      project: gary.schaetz/private/pipelines
      ref: main
      file: /path/to/child-pipeline.yml
    rules:
      - rule_name: standard-rule-if
        rule_source: .job_changes_rules
        only_changes: true
jobs:
  - name: example-1
    folder: example/example-folder-1
    gitlab_yml_file: .gitlab-example.yml
    dependent_jobs:
      - name: example-2
        folder: example/example-folder-2
        gitlab_yml_file: .gitlab-example.yml

Sample Output

Using the jinja template here and the data from example/example-config/job-dependency.yml

include:
    - local: 'example/example-config/rules.yml'

workflow:
  rules:
    - if: !reference [.job_changes_rules,merge-request-rule-if]
    - if: !reference [.job_changes_rules,schedule-rule-if]

stages:
  - pre-build
  - build
  - post-deploy

pre-job-example:
  stage: pre-build
  variables:
    MYVAR: MYVAR_VALUE
  rules:
    - if: !reference [.job_changes_rules,standard-rule-if]
      changes:
        - example/example-folder-1/**/*
        - example/example-folder-2/**/*
  trigger:
    include: 
      - local: 'example/example-pre-job/.gitlab-example.yml'
    strategy: depend
    forward:
      pipeline_variables: true

example-1:
  stage: build
  variables:
    PROJECT_SUBDIR: example/example-folder-1
    PROJECT_JOB_NAME: example-1
    PROJECT_SUBDIR_LAST_FOLDER: example-folder-1
    MYVAR: MYVAR_VALUE
  rules:
    - if: !reference [.job_changes_rules,standard-rule-if]
      changes:
        - example/example-folder-1/**/*
    - if: !reference [.job_changes_rules,manual-rule-if]
      when: manual
      changes:
        - example/example-folder-1/**/*
    - if: !reference [.job_changes_rules,schedule-rule-if]
  trigger:
    include: example/example-folder-1/.gitlab-example.yml    
    strategy: depend
    forward:
      pipeline_variables: true
 
example-2:
  stage: build
  variables:
    PROJECT_SUBDIR: example/example-folder-2
    PROJECT_JOB_NAME: example-2
    PROJECT_SUBDIR_LAST_FOLDER: example-folder-2
    MYVAR: MYVAR_VALUE
  rules:
    - if: !reference [.job_changes_rules,standard-rule-if]
      changes:
        - example/example-folder-2/**/*
        - example/example-folder-1/**/*
    - if: !reference [.job_changes_rules,manual-rule-if]
      when: manual
      changes:
        - example/example-folder-2/**/*
        - example/example-folder-1/**/*
    - if: !reference [.job_changes_rules,schedule-rule-if]
  needs: 
    - job: example-1
      optional: true 
  trigger:
    include: example/example-folder-2/.gitlab-example.yml    
    strategy: depend
    forward:
      pipeline_variables: true
 
cut-tag:
  stage: post-deploy
  variables:
    MYVAR: MYVAR_VALUE
  rules:
    - if: !reference [.job_changes_rules,standard-rule-if]
      changes:
        - example/example-folder-1/**/*
        - example/example-folder-2/**/*
  trigger:
    include: 
      - project: 'gary.schaetz/private/pipelines'
        ref: main
        file: '/path/to/child-pipeline.yml' 
    strategy: depend
    forward:
      pipeline_variables: true

How to use in your pipeline

A variable is passed into your pipeline that you can see in the previous section called PROJECT_SUBDIR. Use this in before scripts to cd into the active project.

Two other variables are exposed:

• PROJECT_JOB_NAME which contains the value from the name tag in your input yaml
• PROJECT_SUBDIR_LAST_FOLDER the last folder in PROJECT_SUBDIR

Here is an example of a pipeline that will use this program in one job to generate the ci file and then trigger it in the next job. Notice how we set PARENT_PIPELINE_SOURCE variable so that you can use this in your downstream rules as a trigger overrides the original CI_PIPELINE_SOURCE.

include: 
  - project: "gary.schaetz/private/pipelines"
    ref: main
    file: 'templates/rules/rules.yml'

stages:
  - build_config
  - run

generate-pipeline:
  image: gschaetz/base-alpine:latest
  stage: build_config
  tags: 
    - docker 
    - kubernetes
  rules:
    - !reference [.job_rules, standard-rule]  
  before_script:
    - pip install gitlab-ci-generator
  script:
    - gitlab-ci-generator -f job-dependency.yml -o generated-ci.yml
  artifacts:
    paths:
      - generated-ci.yml

trigger-child-pipeline:
  stage: run
  rules:
    - if: !reference [.job_changes_rules,standard-rule-if]
      changes:
        - "*/**/*"
  trigger:
    include:
      - artifact: generated-ci.yml
        job: generate-pipeline
    strategy: depend
  variables:
    PARENT_PIPELINE_ID: $CI_PIPELINE_ID
    PARENT_PIPELINE_SOURCE: $CI_PIPELINE_SOURCE

License

MIT License