The Kafka Slurm Agent is a distributed computing and stream processing engine that can be used to run python code acrossmultiple SLURM managed HPC clusters or individual workstations.It uses Kafka to asynchronously communicate with agents installed on clusters and workstations.It contains a monitoring tool with a Web JSON API and a job submitter.It is a pure Python implementation using faust stream processing
Project description
Kafka Slurm Agent
The Kafka Slurm Agent is a tool for submitting computing tasks to the Slurm queues on multiple clusters. It uses Kafka to asynchronously communicate with an agent installed on each cluster. It contains a monitoring tool and a job submitter.
Installation.
Use the standard pip
tool to install. The recommended way is to use a Python virtual environment:
python3 -m venv venv
source venv/bin/activate
pip install kafka-slurm-agent
Using
In the folder in which you created the venv
subfolder run the following command:
kafka-slurm create-project --folder .
This will generate a configuration file kafkaslurm_cfg.py
, startup scripts and a module (my_monitor_agent.py
)
for adding your own implementation of the monitoring agent. This implementation may react to the incoming events
with computed jobs and handle them.
Creating topics on Kafka with appropriate partitions
kafka-slurm --new-topic-partitions 4 topics-create kafka-slurm --new-topic-partitions 4 topics-create
Submitting jobs
js = JobSubmitter() prots = ['E9Q4N7-F1', 'E9Q7C4-F1', 'E9Q842-F1', 'E9Q9R9-F1', 'Q15149-F7', 'Q15149-F8', 'Q15149-F9']
check (default: True) - don't submit if was already computed
ignore_error_status (default: False) - don't submit if previously generated an error
results = js.send_many(prots, 'run.py', {'RESOURCES_REQUIRED': 1, 'JOB_TYPE': 'cpu'}, ignore_error_status=True, check=False) print(results)
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distributions
Built Distribution
Hashes for kafka_slurm_agent-1.2.1-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 4d6acc9974d26f2d7f792d014229bab682c3be28033d8b1a4e782692e2e5b61f |
|
MD5 | b740de6a4549f9a1a1f49e96bd432080 |
|
BLAKE2b-256 | 346c71dcdedbf302486910dda0d5080c642d5122993b39985f17fa5ea9876916 |