laboro 0.0.57

The Laboro workflow framework

---

gitea: none include_toc: true

Laboro

Laboro is a workflow manager that helps you to build and run workflows with the lesser possible code typing.

Development status

Laboro is in early alpha stage and is not ready for production.

Status of this documentation

This documentation is an incomplete work in progress and changes may occur as new versions of Laboro are released.

Install

Laboro is intended to be run within a container. See the container/README.md file to build a custom Laboro container image.

The present documentation will only consider the Laboro configuration and usage using the Laboro container.

However, Laboro is available through the Python Package Index for adventurous and expert users willing to simply install Laboro on their computer using pip3 install laboro and configure it manually.

To get the latest Laboro version, run the following command line:

docker pull mcosystem/laboro:latest

The Laboro version matches the Python Laboro package version. Thus, to install Laboro in a specific version, use the corresponding tag:

docker pull mcosystem/laboro:1.0.0

Configuration

Laboro has two configuration levels:

• The global configuration level:

  This level is set up with the mandatory /etc/laboro/laboro.yml.

  The default global configuration file is auto-generated at build time from the parameters given to container build command. See the container/README.md for further details on the available configuration parameters and how to build a custom Laboro container image.

  Passing parameters at build time ascertains that all needed directories are created with the expected permissions.

  Even if Laboro is a container based app, modification of the default global configuration file outside build time (i.e. by mounting a volume at run time) is not expected nor supported.

  If you choose to do so, please, don't forget to create the directories according to your configuration file and apply the right permissions to them.

  Example of global configuration file:

  ---
  laboro:
    histdir: /opt/laboro/log/hist
    workspacedir: /opt/laboro/workspaces
    workflowdir:  /opt/laboro/workflows
    log:
      dir:  /opt/laboro/log
      level: DEBUG
  

• The workflow configuration level:

  Each workflow must have a YAML configuration file in the ${LABORO_WORKFLOWDIR}.

  The name of the configuration filename must reflect the workflow name. Thus, if you workflow name is my_workflow, the workflow configuration file must be ${LABORO_WORKFLOWDIR}/my_workflow.yml

  The ${LABORO_WORKFLOWDIR} is expected to be mounted as a container volume at run time.

  Thus, to run the two my_workflow_1 and my_workflow_2, the volume must contain my_workflow_1.yml and my_workflow_2.yml file and be mounted under the ${LABORO_WORKFLOWDIR} directory which defaults to ${LABORO_HOMEDIR}/workflows.

  Using the default global configuration file, the following command line would run the my_workflow_1 and my_workflow_2 workflows with their corresponding configuration files situated on the container host in the /path/to/local/workflowdir directory:

  docker run --name laboro \
             -e TZ=Pacific/Noumea \
             -v /path/to/local/workflowdir:/opt/laboro/workflows:ro \
             mcosystem/laboro \
             -r my_workflow_1 my_workflow_2
  

Workflows

Sessions

Each execution of a specific workflow is identified by a unique identifier called a session. The session identifier is a string representation of an uuid.uuid4 object. This session identifier is stored in the workflow execution history file.

The workflow log file and the session workspace are named after this property.

Specification

The workflow configuration file must validate against the specification found in the source code

Configuration

A workflow is described by a simple YAML file describing all details of the workflow steps and actions.

An example of a valid workflow configuration file can be found in the test suite

Workflow properties

name: The workflow name. This property is used to identify the workflow. The workflow main workspace, the workflow log file and the workflow execution history file are named after this property.

Sections

packages

The packages section lists all Python packages that should be installed to run the workflow.

All Python packages declared in the packages section will be automatically installed using pip and made available to the workflow.

However, to avoid hazardous python package installation, any valid Laboro package name must match the ^laboro_.*$ regular expression and must be available through the Python Package Index

workspace

The workspace section provides the ability to manage the automatic workspace deletion with the delete_on_exit sub-property.

When delete_on_exit is set to True, the session workspace is deleted the workflow execution ends.

steps

The steps section describes all task that will be sequentially executed in the same order that they are listed. Each step has its own set of properties.

steps.actions

steps.action.object

steps.action.object.module

steps.action.object.class

steps.action.object.args

steps.action.object.args

steps.action.object.instantiate

steps.action.object.methods

Conditional execution

Loops

Optional modules loading

A Laboro module is a Python package specifically designed to run as a Laboro extension and provides various classes derived from the laboro.module.Module base class (see the module section for further details on Laboro modules).

Laboro modules needed by the workflow should be listed in the packages section of the workflow configuration file.

Run your workflows

Once the Laboro container is built or installed from the official image, you can run you workflows by summoning the container and pass your workflows names as arguments.

The workflows will be run sequentially in the specified order.

docker pull mcosystem/laboro:latest
docker run --name laboro \
           -v /path/to/local/workflowdir:/opt/laboro/workflows:ro \
           laboro \
           --run my_workflow_1 my_workflow_2

Example of a Laboro container running a workflow named my_workflow_2 :

Workspaces

Each workflow has a file storage space named after workflow name the and situated under the ${workflowdir} directory.

The workspace is the sub-directory named after the workflow session id created under the main workflow directory for each workflow execution. Laboro modules can use this workspace to store file for later use.

Note: When the workspace.delete_on_exit workflow property is set to True, the workspace and its content is deleted at the end of the workflow execution.

-+-- workspacedir/
       |
       +-- my_workflow_1/
       |      |
       |      +-- workflow_1_session_1/
       |      |        |
       |      |        + file_1
       |      |
       |      +-- workflow_1_session_2/
       |               |
       |               + file_1
       |
       +-- my_workflow_2/
              |
              +-- workflow_2_session_1/
              |        |
              |        + file_1
              |
              +-- workflow_ 2_session_2/

Logging

Laboro logging configuration is set from the main laboro configuration file.

The default container configuration set the "${LABORO_LOGDIR}" variable to /var/log/laboro so the log directory can be easily mounted at runtime from the host with the following command:

docker run --name laboro \
           -e TZ=Pacific/Noumea \
           -v /path/to/local/workflowdir:/opt/laboro/workflows:ro \
           -v /path/to/local/logdir:/var/log/laboro:rw \
           mcosystem/laboro \
           -r my_workflow_1 my_workflow_2

All workflows logs are stored in the laboro.log.dir directory and each workflow has a specific directory where its session logs will be stored.

-+-- logdir/
       |
       +-- my_workflow_1/
       |      |
       |      +-- workflow_1_session_1.log
       |      |
       |      +-- workflow_1_session_2.log
       |
       +-- my_workflow_2/
              |
              +-- workflow_2_session_1.log

Log levels

Log level is a workflow configuration and must be set to one of the following values:

Log levels
DEBUG
INFO
WARNING
ERROR
CRITICAL

Log date and time

Secrets

Execution history

Modules

Available Modules

Build your own module

Write the module specification