airflow-unicore-integration 0.5.0

Using the UnicoreExecutor

To use the UnicoreExecutor, this library needs to be installed in your airflow environment and then some configuration work needs to be done.

How to configure these settigns is up to your deployment, as it uses the standard airflow configuration mechanism. In case of a helm deployemnt via the official helm chart, you will need to use environment variables, as all unicore related options are not present in the chart and will cause schema-validation to fail.

All options fall under the [unicore.executor] section in airflow.cfg, or have the AIRFLOW__UNICORE_EXECUTOR__ prefix as an environment variable (or formatted as AIRFLOW___TEAM__UNICORE_EXECUTOR__).

The default env is loaded via . default_env.sh, and must enable an environment, where python is available in a suitable version, and the apache-airflow-task-sdk and apache-airflow-providers-git packages are available. All other dependencies depend on the dags to be run, but must already be included in the environment.

A simple solution for this may be the “activate” script for a python venv. If the target systems requires additional commands to enable python (e.g. module load), these may be added to the top of the activate script.

AIRFLOW__UNICORE_EXECUTOR__SITES_CONFIG: |
    [
      [
        "local",
        "https://unicore:8080/DEMO-SITE/rest/core",
        "group:groupname",
        "",
        "False"
      ],
      [
        "juwels",
        "https://unicore:8080/DEMO-SITE/rest/core",
        "",
        "",
        ""
      ],
      [
        "juwels-booster",
        "https://unicore:8080/DEMO-SITE/rest/core",
        "",
        "activate_proxy_env_command",
        "True"
      ]
    ]

Using the Unicore Operators

There are multiple Unicore operators provided by this package. The most versatile one is the UnicoreGenericOperator, which supports a lot of job parameters. All other operators are intended to offer a slightly less complex constructor, and therefore simpler usage, but all generic parameters are still available to be used.

All operators support all possible parameters of the Unicore job description. Here is an excerpt containing some commonly used parameters:

For imports and exports go here for details.

The UnicoreGenericOperator supports the following additional parameters:

The UnicoreScriptOperator offers a way to more easily submit a script as a job, where the script content can be provided as a string.

The UnicoreBSSOperator offers a way to directly submit batch-scripts from their content-strings.

The UnicoreExecutableOperator offers a reduced constructor that only requires an executable.

The UnicoreDateOperator is more of a testing operator, since it will only run the date executable.

Behaviour on Errors and Success

The Unicore Operators do not do a lot of error and exception handling, and mostly just forward any problems to be handled by airflow. All of the Unicore logic is handled by the pyunicore library.

While some validation of the resulting Unicore job description is done automatically, it may still be possible to build an invalid job description with the operators. This may lead to a submission failure with Unicore. In this case, an exception is thrown to be handled by airflow.

For a successful job submission, the job exit code is returned as the task return value, so that airflow can handle non-zero exit codes. All operators will also append the content of the job-log-file from Unicore to the airflow task log. Also, some job results and values will be uploaded via airflow-x-coms as well:

Example DAGs

There are some example DAGs in this repository under project-dir/dags.

• unicore-test-1.py just shows basic date and executable usage.

• unicore-test-2.py has some basic examples for the generic operator.

• unicore-test-3.py also includes script-operator examples.

• unicore-test-4.py has some examples with more arguments.

• unicore-test-bss.py shows how bss submission can be done (very simple example).

• unicore-test-credentials.py demonstrates that not only the credentials from the airflow connections backend can be used, but they can also be provided in the constructor of the operator.

• unicore-test-import-export.py gives short examples for the imports and exports usage.

Setup testing env

Ensure a current version of docker is installed.

Run python3 -m build to build the python package.

Run the testing-env/build-image.sh script to create the customized airflow image, which will contain the newly build python package.

Run testing-env/run-testing-env.sh init to initialize the airflow containers, database etc. This only needs to be done once.

Run testing-env/run-testing-env.sh up to start the local airflow and Unicore deployment. Airflow will be available on port 8080, Unicore on port 8081.

The run-testing-env.sh script supports the commands up, down, start, stop, ps and init for matching docker compose functions.

Install package via pip

pip install airflow-unicore-integration