kafka-slurm-agent 1.2.1

The Kafka Slurm Agent is a distributed computing and stream processing engine that can be used to run python code acrossmultiple SLURM managed HPC clusters or individual workstations.It uses Kafka to asynchronously communicate with agents installed on clusters and workstations.It contains a monitoring tool with a Web JSON API and a job submitter.It is a pure Python implementation using faust stream processing

Kafka Slurm Agent

The Kafka Slurm Agent is a tool for submitting computing tasks to the Slurm queues on multiple clusters. It uses Kafka to asynchronously communicate with an agent installed on each cluster. It contains a monitoring tool and a job submitter.

Installation.

Use the standard pip tool to install. The recommended way is to use a Python virtual environment: python3 -m venv venv source venv/bin/activate pip install kafka-slurm-agent

Using

In the folder in which you created the venv subfolder run the following command: kafka-slurm create-project --folder . This will generate a configuration file kafkaslurm_cfg.py, startup scripts and a module (my_monitor_agent.py) for adding your own implementation of the monitoring agent. This implementation may react to the incoming events with computed jobs and handle them.

Creating topics on Kafka with appropriate partitions

kafka-slurm --new-topic-partitions 4 topics-create kafka-slurm --new-topic-partitions 4 topics-create

Submitting jobs

js = JobSubmitter() prots = ['E9Q4N7-F1', 'E9Q7C4-F1', 'E9Q842-F1', 'E9Q9R9-F1', 'Q15149-F7', 'Q15149-F8', 'Q15149-F9']

check (default: True) - don't submit if was already computed

ignore_error_status (default: False) - don't submit if previously generated an error

results = js.send_many(prots, 'run.py', {'RESOURCES_REQUIRED': 1, 'JOB_TYPE': 'cpu'}, ignore_error_status=True, check=False) print(results)