myqueue 22.11.2

Simple job queue

MyQueue

MyQueue is a tool for submitting and keeping track of tasks running on a cluster of computers. It uses SLURM, PBS or LSF as a backend and makes handling of tasks easy. It has a command-line interface called mq with a number of sub-commands and a Python interface for managing workflows. Simple to set up: no system administrator or database required.

Features

• Easy task submission:

  • from the command line: mq submit <task> -R <cores>:<time>

  • from Python: myqueue.submit(...)

• Automatic restarting of timed-out/out-of-memory tasks with more time/cores

• Remembers your finished and failed tasks

• Powerful list command for monitoring

• Can be used together with Python venv‘s

• Folder-based Workflows

Quick links:

• Documentation: https://myqueue.readthedocs.io/

• Code: https://gitlab.com/myqueue/myqueue/

• Issues: https://gitlab.com/myqueue/myqueue/issues/

• Chat: https://matrix.to/#/#myqueue:matrix.org

Examples

Submit Python script to 32 cores for 2 hours:

$ mq submit script.py -R 32:2h

Submit Python module abc.run in two folders:

$ mq submit abc.run F1/ F2/ -R 16:30m

Check results of tasks in current folder and its sub-folders:

$ mq list  # or mq ls
id  folder name      res.   age     state   time    error
--- ------ --------- ------ ------- ------- ------- ------
117 ./     script.py 32:2h  5:28:43 TIMEOUT 2:00:03
118 ./F1/  abc.run   16:30m 5:22:16 done      12:12
119 ./F2/  abc.run   16:30m 5:22:16 done      17:50
--- ------ --------- ------ ------- ------- ------- ------
done: 2, TIMEOUT: 1, total: 3

Resubmit with more resources (1 day):

$ mq resubmit -i 117 -R 32:1d

See more examples of use here:

• Quick-start

• Documentation

• How it works

• Command-line interface

• Workflows

• Python API

Installation

MyQueue has only one dependency: Python version 3.7 or later.

Install MyQueue from PyPI with pip:

$ python3 -m pip install myqueue

Enable bash tab-completion for future terminal sessions like this:

$ mq completion >> ~/.profile

Now, configure your system as described here.

Release notes

See the release notes for a history of notable changes to MyQueue.

Help, support and feedback

If you need help, want to report a bug or suggest a new feature then you are welcome to get in touch via MyQueue’s issue tracker or the #myqueue room on Matrix.

Contributing

We welcome contributions to the code and documentation, preferably as merge-requests. More information here.