qubership-pipelines-declarative-executor 2.1.0

Qubership Pipelines Declarative Executor

Qubership Pipelines Declarative Executor

Open-source python pipelines orchestrator and executor.

This application is distributed as a library on PyPI, Docker image on GitHub Packages, and as a Reusable Workflow for your existing GitHub workflows. You can use this application in any of the provided forms that best suits your needs.

Structure

Docker image

Provided Docker image with this application includes:

• Pipelines Declarative Executor app itself
• Sample "Python Modules" to execute commands in your "Atlas Pipeline"
• SOPS binaries to securely process configuration files and output parameters

Using your own "Python Modules"

Sample "Python Modules", included in distributed Docker image, are only showcasing the intended way of working via Execution Commands and Execution Context. You might need to have your custom commands executed in "Atlas Pipelines" - then you will need to create and distribute your own implementation. There's a Development Guide available in CLI Samples repository.

A few ways you can include your own "Python Modules" into executor Docker image:

• Building new image with required modules, and putting path to them to PIPELINES_DECLARATIVE_EXECUTOR_PYTHON_MODULE_PATH env variable
• Mounting directory with your modules into the image, while also upgrading PIPELINES_DECLARATIVE_EXECUTOR_PYTHON_MODULE_PATH env variable to mounted path

Multiple "Python Modules" are supported via stage.path property (check syntax guide for more information)

Python Dependency

You can also use this application as a python dependency, by installing it from PyPI:

pip install qubership-pipelines-declarative-executor

Or adding it to your dependency list:

qubership-pipelines-declarative-executor = "^2.0.0"

Common CI Workflows (GitHub/GitLab)

Alternative usage scenario (although much less configurable than creating your own workflow, since we can't pass custom env variables into it) is using provided Reusable Workflow in your repository. There is an example of how it can be invoked from GitHub Workflow: pipeline.yml

There's also an example of a similar implementation for GitLab

When you've created your own Docker image with necessary "Python Modules" (non-sample ones), you might want to create your own workflows using that image. Provided workflows serve as a starting point, but custom workflows will allow full control and ability to pass env variables and local files.

Features

Atlas Pipeline definitions

"AtlasPipelines" are intended to work via Execution Commands, packed into "Python Modules".

Pipeline itself describes data flow between sequentially executed stages, while also supporting invoking nested pipelines (for reusing configuration) and parallel stages.

This repository uses actual "AtlasPipelines" in its tests, you can check them here.

Separate article on syntax with examples is available here.

Reporting

Executor collects and can upload report (intended for UI representation) of currently executed pipeline.

This feature is configured via env variables in Report section. You can select REPORT_SEND_MODE (either ON_COMPLETION or PERIODIC), send intervals, and endpoint configs:

Report configuration example is here

Auth Rules

Orchestrator can fetch remote AtlasPipeline and AtlasConfig files from various sources. To access private repositories or authenticated endpoints, you can configure authentication rules.

Example Auth Rules are present here

Rules are processed in order they are defined, and first applicable rule will be used (in case when multiple would've matched).

If no rules match, requests are made without authentication.

CI Wrapper configuration

You can pass your CI (GitHub, GitLab, Jenkins, etc.) execution instance parameters (user who triggered pipeline, pipeline's URL) to make them available in the report via environment variables

SOPS Encryption

Executor decrypts input files (pipelines and configs) if they are encrypted with SOPS, and can also encrypt any output secure files. This feature is configured via a set of environment variables

ENV Configuration

Other parameters are available and documented in the General section

Performance

Performance tests and comparisons of different ways to invoke imported "Python Modules" will be available here