The Kafka Slurm Agent is a tool for submitting computing tasks to the Slurm queues on multiple clusters. It uses Kafka to asynchronously communicate with an agent installed on the cluster.It contains a monitoring tool and a job submitter.
Project description
Kafka Slurm Agent
The Kafka Slurm Agent is a tool for submitting computing tasks to the Slurm queues on multiple clusters. It uses Kafka to asynchronously communicate with an agent installed on each cluster. It contains a monitoring tool and a job submitter.
Installation.
Use the standard pip
tool to install. The recommended way is to use a Python virtual environment:
python3 -m venv venv
source venv/bin/activate
pip install kafka-slurm-agent
Using
In the folder in which you created the venv
subfolder run the following command:
kafka-slurm create .
This will generate a configuration file kafkaslurm_cfg.py
, startup scripts and a module (my_monitor_agent.py
)
for adding your own implementation of the monitoring agent. This implementation may react to the incoming events
with computed jobs and handle them.
Test connectivity #docker run --add-host zalman:172.17.0.1 -p 8088:8080 -e KAFKA_CLUSTERS_0_NAME=local -e KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS=172.17.0.1:9092 -d provectuslabs/kafka-ui:latest
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distributions
Built Distribution
Hashes for kafka_slurm_agent-1.1.3-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | a176c356e0092686b95d28df102ac471ed0c0dd73f959b28672e42e00a0121a4 |
|
MD5 | 0a28bfa7bfed4c5820673001af7daf02 |
|
BLAKE2b-256 | 6caf0d9d7bbc27ecef5f21a492f6aaf630ade5c2dd7f038c616c889f8a024371 |