Agent for integrating service provider's site with Waldur Mastermind.
Project description
Agent for Service Provider Integration
Agent for Mastermind integration with a provider's site. The main purpose of the agent is data syncronization between Waldur instance and an application (for example SLURM or MOAB cluster). The application uses order-related information from Waldur to manage accounts in the site and accounting-related info from the site to update usage data in Waldur. For now, the agent supports only SLURM cluster as a site.
Architecture
This is a stateless application, which is deployed on a machine having access to SLURM cluster data. The agent consists of two sub-applications:
- agent-pull, which fetches data from Waldur and updates a state of a SLURM cluster correspondingly (e.g. creation of SLURM accounts ordered in Waldur);
- agent-push, which sends data from SLURM cluster to Waldur (e.g. update of resource usages).
Integration with Waldur
For this, the agent uses Waldur client based on Python and REST communication with Waldur backend. Agent-pull
application pulls data of orders created for a specific offering linked to the site and creates/updates/removes SLURM accounts based on this info. Agent-push
fetches data of usage, limits and associations from the site and pushes it to Waldur.
Integration with the site
SLURM cluster
For this, the agent uses SLURM command line utilities (e.g. sacct
and sacctmgr
).
The access to the binaries can be either direct or using docker client.
In the latter case, the agent is required to have access to docker
binary and to docker socket (e.g. /var/run/docker.sock
).
Setup
The application supports the following environmental variables (required ones formatted with bold font):
WALDUR_API_URL
- URL of Waldur Mastermind API (e.g.http://localhost:8081/api/
).WALDUR_API_TOKEN
- token for access to Mastermind API.WALDUR_SYNC_DIRECTION
- accepts two values:push
andpull
. Ifpull
, then application sends data from SLURM cluster to Waldur, vice versa ifpush
.WALDUR_OFFERING_UUID
- UUID of corresponding offering in Waldur.REQUESTS_VERIFY_SSL
- flag for SSL verification for Waldur client, default istrue
.SLURM_TRES_CONFIG_PATH
- a path to the SLURM TRES configuration, default is./config-components.yaml
.SLURM_DEPLOYMENT_TYPE
- type of SLURM deployment. accepts two values:docker
andnative
, default isdocker
.SLURM_CUSTOMER_PREFIX
- prefix used for customer's accounts, default ishpc_
.SLURM_PROJECT_PREFIX
- prefix used for project's accounts, default ishpc_
.SLURM_ALLOCATION_PREFIX
- prefix used for allocation's accounts, default ishpc_
.SLURM_ALLOCATION_NAME_MAX_LEN
- maximum length of account name created by the application.SLURM_DEFAULT_ACCOUNT
- default account name existing in SLURM cluster for creation of new accounts. Default iswaldur
.SLURM_CONTAINER_NAME
- name of a headnode SLURM container; must be set if SLURM_DEPLOYMENT_TYPE is docker.ENABLE_USER_HOMEDIR_ACCOUNT_CREATION
- whether to create home directories for users related to accounts.SENTRY_DSN
- Data Source Name for Sentry (more info here).SENTRY_ENVIRONMENT
- name of the Sentry environment.
Alternatively, the agent can serve several offerings. For this:
-
Create a config file for offerings with the following format:
offerings: - name: example-offering-01 waldur_api_url: https://waldur1.exmaple.com/api/ waldur_api_token: <token1> waldur_offering_uuid: <uuid1> - name: example-offering-02 waldur_api_url: https://waldur2.exmaple.com/api/ waldur_api_token: <token2> waldur_offering_uuid: <uuid2>
-
Add a variable
WALDUR_CONFIG_FILE_PATH
to the environment, for example:export WALDUR_CONFIG_FILE_PATH=/etc/waldur-site-agent/offerings.yaml
NB: Environment variables take precedence over the config file, so if you define WALDUR_API_URL
, WALDUR_API_TOKEN
and WALDUR_OFFERING_UUID
, the file is ignored.
Deployment
Test environment
In order to test the agent, a user should deploy 2 separate instances of it. The first one (called agent-pull) is for fetching data from Waldur with further processing and the second one (called agent-push) is for sending data from SLURM cluster to Waldur. Both instances must be configured with environment variables (from e.g. .env-file), file for computing components and an optional file for multiple offerings.
The example of .env-file
for agent-pull:
WALDUR_SYNC_DIRECTION=pull # The setup for agent-pull
WALDUR_API_URL=http://waldur.example.com/api/ # Waldur API URL
WALDUR_API_TOKEN=9e1132b9616ebfe943ddf632ca32bbb7e1109a32 # Token of a service provider in Waldur
WALDUR_OFFERING_UUID=e21a0f0030b447deb63bedf69db6742e # UUID of SLURM offering in Waldur
SLURM_DEFAULT_ACCOUNT=root # Default account for SLURM
SLURM_CONTAINER_NAME=slurmctld # Name of SLURM namenode container
The example of .env-file for agent-push:
WALDUR_SYNC_DIRECTION=push # The setup for agent-push
WALDUR_API_URL=http://waldur.example.com/api/ # Waldur API URL
WALDUR_API_TOKEN=9e1132b9616ebfe943ddf632ca32bbb7e1109a32 # Token of a service provider in Waldur
WALDUR_OFFERING_UUID=e21a0f0030b447deb63bedf69db6742e # UUID of SLURM offering in Waldur
SLURM_CONTAINER_NAME=slurmctld # Name of SLURM namenode container
Docker-based deployment
You can find the Docker Compose configuration for testing in examples/docker-compose/ folder:
In order to test it, you need to execute following commands in your terminal app:
cd examples/docker-compose
docker-compose up -d
Systemd deployment
In case of native deployment, you need to setup and run the a systemd service executing Python module.
SLURM agent
The agent requires sacct
and sacctmgr
to be accessible on a machine, so it should run on a headnode of the SLURM cluster.
Firstly, install the waldur-site-agent:
pip install waldur-site-agent
Secondly, put systemd unit, environment and and TRES config files to the corresponding locations. Don't forget to modify Waldur-related values the env files.
agent-pull files for a SLURM agent
systemd unit
: waldur-site-agent-pull.serviceexample .env
: waldur-site-agent-pull.env
agent-push files for a SLURM agent
systemd unit
: waldur-site-agent-push.serviceexample .env
: waldur-site-agent-push.env
Common files
# For agent-pull
cp systemd-conf/agent-pull/waldur-site-agent-pull.service /etc/systemd/system/
mkdir /etc/waldur-site-agent/
cp systemd-conf/agent-pull/waldur-site-agent-pull.env /etc/waldur-site-agent/pull.env
cp ./config-components.yaml.example /etc/waldur-site-agent/tres.yaml # you can use a different path and set SLURM_TRES_CONFIG_PATH to it
# For agent-push
cp systemd-conf/agent-push/waldur-site-agent-push.service /etc/systemd/system/
cp systemd-conf/agent-push/waldur-site-agent-push.env /etc/waldur-site-agent/push.env
After these preparation steps, run the following script to apply the changes.
systemctl daemon-reload
systemctl start waldur-site-agent-pull
systemctl enable waldur-site-agent-pull # to start after reboot
systemctl start waldur-site-agent-push
systemctl enable waldur-site-agent-push # to start after reboot
Older systemd versions
If you want to deploy the agents on a machine with systemd revision older than 240, you should use files with legacy configuration:
- systemd legacy unit file for agent-pull: waldur-site-agent-pull-legacy.service
- systemd legacy unit file for agent-push: waldur-site-agent-push-legacy.service
# For pulling agent
cp systemd-conf/agent-pull/waldur-site-agent-pull-legacy.service /etc/systemd/system/waldur-site-agent-pull.service
# For pushing agent
cp systemd-conf/agent-push/waldur-site-agent-push-legacy.service /etc/systemd/system/waldur-site-agent-push.service
TRES configuration
To setup TRES-related info, the agent uses the corresponding configuration file configured by SLURM_TRES_CONFIG_PATH
environment variable (config-components.yaml
by default). Each entry of the file incudes key-value-formatted data.
A key is a type of TRES (with optional name if type is gres
) and the value contains limit, measured unit, type of accounting and label.
The script waldur_slurm_load_components
sends this data to Waldur:
waldur_slurm_load_components
If a user wants to change this information, a path of a custom config file should be set for SLURM_TRES_CONFIG_PATH
variable.
Service provider configuration
SLURM
The agents require existing offering data in Waldur. As a service provider owner, you should create an offering in the marketplace:
- Go to
Provider
section on the left tab ->Add new offering
button - Input a name, choose a category, select
SLURM remote allocation
from the drop-down list on the bottom and clickCreate
button
- Open the offering page and create a plan in the
Accounting
section: clickAdd plan
and input the necessary details - Go to
Integration
section, clickShow integration steps
and ensure they are completed within your SLURM cluster.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for waldur_site_agent-0.2.4-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 1c63faea5516ba429096342ae00a43c2f261c0a1ab6a7d12eb38790757372c7d |
|
MD5 | c14fb020f650f075a4ded101d49c4c86 |
|
BLAKE2b-256 | 2bf118e433f977e6bb62c69f2e792d2a871f3317d58fad28e705ba7657d39ccf |