CTAO Workload Management System

This project provides Helm chart to deploy WMS with all necessary dependencies and services.

Chart

The WMS Helm chart aims to provide a complete deployment of DIRAC with all necessary dependencies. It contains:

• DIRAC Master CS
• DIRAC services and agents
• DIRAC Web App
• DIRAC Databases
• Authentication services
• Bootstrap jobs
• External dependencies

The WMS chart is integrated in the global DPPS deployment.

Chart Dependencies

The chart includes the following external dependencies:

• diracx: complete DiracX deployment
• mariadb: databases for DIRAC and DiracX data storage

For testing, several components are also included as subcharts, but are not needed are expected to be provided as external components integrated production:

• gcert-issuer: a CTAO grid certificate issuer for testing deployments
• cvmfs: CVMFS stratum-0 deployment for testing
• indigo-iam: IAM deployment provided by CTAO for testing

Components description

DIRAC Master CS

The DIRAC Master Configuration Server deployment consists of:

• statefulset (master-cs/statefulset.yaml): to ensure the uniqueness of the deployment
• secret (master-cs/secret.yaml): contains specific DIRAC configuration which shouldn't be exposed in the remote configuration

DIRAC components

All DIRAC components (services, agents and executors) deployment:

• deployments (deployment.yaml): iteratively generated deployment and service for each enabled DIRAC component
• configuration files (dirac-configuration-configmap.yaml): a configmap containing DIRAC configuration files, separate into DIRAC sections (Operations, Resources, Registry, ...). Each installed component as its own specific Systems section configuration
• environment configuration (dirac-env-configmap.yaml): environment specific configuration

DIRAC Web App

A deployment of the DIRAC Web App:

• deployment (web-app/deployment.yaml)
• configuration (web-app/configmap.yaml): web app specific configuration

Computing Element

A simple SSH computing element:

• deployment (computing-element/deployment.yaml): a ssh server acting as a CE

Ingress

TODO.

Bootstrap sequence diagram

The bootstrap sequence is composed of several Jobs to make sure DIRAC is correctly configured and running.

1. DIRAC configuration initialization (bootstraps/configure-dirac-job.yaml)

Apply the DIRAC configuration contained in the configmap to the Master CS. This job runs once MariaDB and MasteCS statefulsets are running.

2. DIRAC DB initialization (bootstraps/init-dirac-db-job.yaml)

Install the DIRAC DB and the DiracX AuthDB. The Job runs once the ComponentMonitoring service is running.

3. Synchronize DIRAC configuration (bootstraps/sync-dirac-job.yaml)

Synchronize RSS configuration, then synchronize DIRAC and DiracX configuration and finally initialize/synchronize IAM users. This job runs after Databases initialization and when DiracX is running.

4. First Proxy initialization (bootstraps/first-proxy-job.yaml)

Initialize DIRAC proxies, at installation is necessary to initialize a first time the pilot proxy. This job runs once the ProxyManager, BundleDelivery and SystemAdministrato services are running.

---
title: WMS Bootstrap sequence
---
flowchart TB
    subgraph mariadb ["MariaDB Statefulset"]
        direction LR
        mdb["dirac-db"]
    end
    subgraph mastercs ["Master CS statefulset"]
        direction LR
        mcs["dirac-master-cs"]
    end
    subgraph bootdiracconf ["Bootstrap DIRAC Conf Job"]
        direction LR
        confdirac("bootstrap-dirac-configure")
    end
    subgraph compmon ["Component Monitoring deployment"]
        direction LR
        bootcomp["bootstrap-component-monitoring"]-->compmondep["component-monitoring"]
    end
    subgraph bootdb ["Bootstrap DIRAC DB Job"]
        direction LR
        authdb("install-diracx-auth-db")-->compdb("install-dirac-db")
    end
    subgraph bootcnf ["Sync DIRAC Conf Job"]
        direction LR
        syncrss("bootstrap-dirac-sync-rss")-->syncdiracxcs("sync-diracx-cs")
        syncdiracxcs-->synciamusers("sync-iam-users")
    end
    subgraph diracx ["DiracX deployment"]
        direction LR
        dx("DiracX")
    end
    subgraph dcomp ["DIRAC Components deployment"]
        direction TB
        proxyman["dirac-proxy-manager"]
        bundledeliv["dirac-bundle-delivery"]
        systemadmin["dirac-system-admin"]
        rest["..."]
    end
    subgraph fproxy ["First proxy init Job"]
        direction LR
        initproxy("first-proxy")
    end
    mariadb --wait for--> mastercs
    mastercs --wait for--> bootdiracconf
    bootdiracconf --wait for--> compmon
    compmon --wait for--> bootdb
    bootdb --wait for--> bootcnf
    bootcnf --wait for--> dcomp
    syncdiracxcs --needs--> diracx
    diracx --wait for--> fproxy
    proxyman --wait for--> fproxy
    bundledeliv --wait for--> fproxy
    systemadmin --wait for--> fproxy

Storage Volumes

• CS backup (master-cs/statefulset.yaml): persistent volume to keep CS backup while restarting the Master CS statefulset
• Storage Element Data (pvc.yaml): persistent storage for data stored in the test SE

Configuration Structure

DIRAC server configuration

All the configuration related to DIRAC server is contained in the values.yaml under diracServer section. This section will allow you to configure master-cs, web-app, components and configurations.

• Master CS .diracServer.masterCS
• WebApp .diracServer.webApp
• DIRAC components .diracServer.diracComponents: in this section each components can be enabled or disabled, configured (port, specific configuration), specify the replica counts. For each componennt a default configuration is provided but can be overwritten.

Example:

diracServer:
    diracComponents:
        componentMonitoring:
            port: 9190 # the port on which contact the service
            cmd: Framework/ComponentMonitoring # the command run by `dirac-service`
            replicaCount: 1
            config: | # to overwrite the default configuration
                Systems
                {
                    Framework
                    {
                        Services
                        {
                            ComponentMonitoring
                            {
                                ...
                            }
                        }
                    }
                }

Note: the name of the subsection (e.g. componentMonitoring) is used to generate deployment, service and associated configmap names.

• bootstrap: bootstrap jobs can be individually disabled or enabled, and configured under diracServer.bootstrap

• DIRAC databases: the list of the DIRAC databases to be installed must be provided under diracServer.diracDatabases

• DIRAC configuration: To set specific DIRAC configuration there is diracServer.diracConfig. Since the full DIRAC configuration is very long, the configurability of it is separated into sections: Systems, Registry, Resources, Operations, DiracX. For each a file in a configmap is generated which is then apply at bootstrap.

One can write a section using the yaml format, which will be automatically translated, or using a string template. One can decide to provide its own full DIRAC configuration by overwriting the configmap itself using diracServer.diracConfig.configmap.create: false and diracServer.diracConfig.configmap.name: config-map-name

Test Report

The WMS test report can be retrieve from the CI pipeline artifact.

Deploy WMS locally

The first time you want to run the local cluster:

pixi run install-tools

To start a Kind local cluster, deploy the chart in it, and run the test, do:

pixi run make -e dev