Qubership Pipelines Declarative Executor

Open-source Python orchestrator and executor for Atlas Pipelines - declarative YAML workflows for CI/CD-style automation.

You describe a pipeline as a sequence of stages (shell commands or packaged Python CLI modules), define how parameters flow between steps, and optionally run stages in parallel or trigger nested sub-pipelines. The executor loads configs and pipeline definitions (local paths or remote URLs), evaluates conditions and variable substitution, runs each stage as a subprocess, and collects outputs - with built-in retries, SOPS encryption for secrets, execution state save/resume, and optional report upload to HTTP or S3 endpoints.

It is distributed as a library on PyPI, a Docker image on GitHub Container Registry, and a reusable GitHub workflow. Use whichever packaging fits your environment.

Structure

Docker image

Provided Docker image with this application includes:

• Pipelines Declarative Executor app itself
• Sample "Python Modules" to execute commands in your "Atlas Pipeline"
• SOPS binaries to securely process configuration files and output parameters

Using your own "Python Modules"

Sample "Python Modules", included in distributed Docker image, are only showcasing the intended way of working via Execution Commands and Execution Context. You might need to have your custom commands executed in "Atlas Pipelines" - then you will need to create and distribute your own implementation. There's a Development Guide available in CLI Samples repository.

A few ways you can include your own "Python Modules" into executor Docker image:

• Building new image with required modules, and putting path to them to PIPELINES_DECLARATIVE_EXECUTOR_PYTHON_MODULE_PATH env variable
• Mounting directory with your modules into the image, while also upgrading PIPELINES_DECLARATIVE_EXECUTOR_PYTHON_MODULE_PATH env variable to mounted path

Multiple "Python Modules" are supported via stage.path property (check syntax guide for more information)

Python Dependency

You can also use this application as a python dependency, by installing it from PyPI:

pip install qubership-pipelines-declarative-executor

Or adding it to your dependency list:

qubership-pipelines-declarative-executor = "^2.0.0"

Common CI Workflows (GitHub/GitLab)

Alternative usage scenario (although much less configurable than creating your own workflow, since we can't pass custom env variables into it) is using provided Reusable Workflow in your repository. There is an example of how it can be invoked from GitHub Workflow: pipeline.yml

There's also an example of a similar implementation for GitLab

When you've created your own Docker image with necessary "Python Modules" (non-sample ones), you might want to create your own workflows using that image. Provided workflows serve as a starting point, but custom workflows will allow full control and ability to pass env variables and local files.

Cancellation of running pipeline via native GH/GL UI button is also supported.

Features

Atlas Pipeline definitions

"AtlasPipelines" are intended to work via Execution Commands, packed into "Python Modules".

Pipeline itself describes data flow between sequentially executed stages, while also supporting invoking nested pipelines (for reusing configuration) and parallel stages.

This repository uses actual "AtlasPipelines" in its tests, you can check them here.

Separate article on syntax with examples is available here.

Reporting

Executor collects and can upload report (intended for UI representation) of currently executed pipeline.

Remote uploads are configured via Remote Delivery params using PIPELINES_DECLARATIVE_EXECUTOR_REMOTE_DELIVERIES.

Delivery configuration example is here

Auth Rules

Orchestrator can fetch remote AtlasPipeline and AtlasConfig files from various sources. To access private repositories or authenticated endpoints, you can configure authentication rules.

Example Auth Rules are present here

Rules are processed in order they are defined, and first applicable rule will be used (in case when multiple would've matched).

If no rules match, requests are made without authentication.

CI Wrapper configuration

You can pass your CI (GitHub, GitLab, Jenkins, etc.) execution instance parameters (user who triggered pipeline, pipeline's URL) to make them available in the report via environment variables

SOPS Encryption

Executor decrypts input files (pipelines and configs) if they are encrypted with SOPS, and can also encrypt any output secure files. This feature is configured via a set of environment variables

ENV Configuration

Other parameters are available and documented in the General section

Performance

Performance tests and comparisons of different ways to invoke imported "Python Modules" will be available here