Remote Pipeline Test Framework

End-to-end test framework for GitLab pipeline template authors.

Trigger pipelines, make assertions.

Why

As a GitLab pipeline author, your pipeline and jobs will be used across several projects under varying configurations. Your pipeline template understands the context (branches, tags, vars, source files etc.) and responds accordingly.

The resultant pipeline's status might vary, the number of stages and jobs might change, statuses of the jobs might vary, artifacts may or may not be produced and environments may or may not be provisioned etc.

Should you not setup proper tests for each of these scenarios? Should these tests not be executed on every change of the pipeline template?

Thus, rptf.

Table of Contents

1. Quickstart
2. Using RPTF with GitLab
3. Setup GitLab Access Token
4. Usage With Self-Managed GitLab
5. Authoring Tests
6. Detailed Example
7. Test Output
8. CLI Options
9. JUnit Reports
10. License
11. Contributing
12. Who Uses RPTF

Quickstart

Being authored in Python, rptf is distributed via PyPi: https://pypi.org/project/rptf/

Install rptf locally with pip.

pip install rptf

This makes rptf available for local execution:

rptf --help

Test cases are defined in a Yaml file, which by default is called rptf.yml.

# rptf.yml

should_succeed_with_25_jobs:
    trigger:
        project_id: ...
    assertions:
        pipeline_status: success
        job_count: 25

should_fail_with_1_job:
    trigger:
        project_id: ...
    assertions:
        pipeline_status: failed
        job_count: 1

should_fail_with_3_jobs_deploy_skipped:
    trigger:
        project_id: ...
    assertions:
        pipeline_status: failed
        job_count: 3
        job_status:
            build: success
            test: failed
            deploy: skipped

Each test case contains two sections: trigger and assertions. The example above shows two test project's pipelines being triggered and asserts being make expect the pipelines have 25 jobs and 1 job respectively.

Pipeline status is also asserted, and the second test case expects a failed pipeline.

Using RPTF with GitLab

We can execute rptf within a GitLab pipeline, thus, you have a primary test project which contains the rptf.yml file and includes a .gitlab-ci.yml definition that triggers pipelines in other GitLab projects and makes assertions.

# .gitlab-ci.yml

stages:
    - test

rptf:
    stage: test
    image: python:3
    before_script:
        - pip install rptf
    script:
        - rptf

The rptf job above triggers pipelines defined in rptf.yml as long as the GITLAB_ACCESS_TOKEN provided has sufficient permissions. It's advisable to set the token at the project or group level via GitLab -> Settings -> CICD.

Setup GitLab Access Token

The project containing the rptf job needs to provide GITLAB_ACCESS_TOKEN. This token must be authorized to execute pipelines across all projects defined in rptf.yml.

Usage With Self-Managed GitLab

Environment variable GITLAB_API_URL is read to determine the GitLab instance where the pipelines are created.

You can set this to get rptf to work with your self-managed GitLab instance.

# set GITLAB_API_URL
export GITLAB_API_URL=https://path-to-your-instance/api/v4

# execute rptf 
rptf

Authoring Tests

The Yaml specification declares the following attributes that can be used to author test cases:

• trigger defines the target project where pipeline should be triggered
  • project_id required
  • ref defaults to master
  • branch deprecated in favor of ref
  • variables defaults to an emppty list `[]'
• assertions defines assertions to make after pipeline finishes execution
  • pipeline_status required, expected pipeline status
  • job_count expected number of jobs
  • job_status dict of expected job status
  • TODO artifact_count expected number of artifacts
  • TODO environment_status dict of expected environment status

Detailed Example

two_jobs_two_artifacts:
    trigger:
        project_id: 11112222
        ref: feature-branch-abc
        variables:
            hello: world
            foo: bar
    assertions:
        pipeline_status: success
        job_count: 2
        job_status:
            artifact-one: success
            artifact-two: success
        artifact_count: 2               # TODO
        environment_status:             # TODO
            success: available          # TODO
            review_env_one: stopped     # TODO

CLI Options

# help
rptf --help             # help

# run tests
rptf                    # by default, executes rptf.yml
rptf --target FILE      # path to test definition yaml

# toggle JUnit test reporting
rptf                    # generates JUnit report
rptf --junit            # generates JUnit report
rptf --no-junit         # JUnit reporting skipped

Test Output

JUnit Reports

By default, all test failures are reported in a JUnit compatible XML file called rptf-report.junit.xml.

This can be integrated in the rptf job to enable deep integration with GitLab (or any other tool that supports JUnit reporting format).

# .gitlab-ci.yml

stages:
    - test

rptf:
    stage: test
    image: python:3
    before_script:
        - pip install rptf
    script:
        - rptf
    artifacts:
        reports:
            junit: rptf-report.junit.xml

Example of RPTF has failing tests integrated back into a GitLab Merge Request:

• https://gitlab.com/sri-at-gitlab/projects/remote-pipeline-test-framework/framework/-/merge_requests/1

JUnit reporting can be turned off by passing the rptf --no-junit flag.

License

MIT

Contributing

Contributions welcome here in the form of feature requests, feature implementations, documentation improvements, examples etc.

# execute locally
pipenv install
pipenv run python -m rptf.rptf

Who Uses RPTF

In-exhaustive list of RPTF users:

• GitLab Five Minute Production App
• LeapMotion
• APNIC
• Brainbits
• Everbridge
• Bolt

Add your organization.