Simple job queue
MyQueue is a tool for submitting and keeping track of tasks running on a cluster of computers. It uses SLURM, PBS or LSF as a backend and makes handling of tasks easy. It has a command-line interface called mq with a number of sub-commands and a Python interface for managing workflows. Simple to set up: no system administrator or database required.
- Easy task submission:
- from the command line: mq submit <task> -R <cores>:<time>
- from Python: myqueue.submit(...)
- Automatic restarting of timed-out/out-of-memory tasks with more time/cores
- Remembers your finished and failed tasks
- Powerful list command for monitoring
- Can be used together with Python venv‘s
- Folder-based Workflows
- Documentation: https://myqueue.readthedocs.io/
- Code: https://gitlab.com/myqueue/myqueue/
- Issues: https://gitlab.com/myqueue/myqueue/issues/
- Chat: https://matrix.to/#/#myqueue:matrix.org
Submit Python script to 32 cores for 2 hours:
$ mq submit script.py -R 32:2h
Submit Python module abc.run in two folders:
$ mq submit abc.run F1/ F2/ -R 16:30m
Check results of tasks in current folder and its sub-folders:
$ mq list # or mq ls id folder name res. age state time error --- ------ --------- ------ ------- ------- ------- ------ 117 ./ script.py 32:2h 5:28:43 TIMEOUT 2:00:03 118 ./F1/ abc.run 16:30m 5:22:16 done 12:12 119 ./F2/ abc.run 16:30m 5:22:16 done 17:50 --- ------ --------- ------ ------- ------- ------- ------ done: 2, TIMEOUT: 1, total: 3
Resubmit with more resources (1 day):
$ mq resubmit -i 117 -R 32:1d
See more examples of use here:
MyQueue has only one dependency: Python version 3.7 or later.
Install MyQueue from PyPI with pip:
$ python3 -m pip install myqueue
Enable bash tab-completion for future terminal sessions like this:
$ mq completion >> ~/.profile
Now, configure your system as described here.
See the release notes for a history of notable changes to MyQueue.
Help, support and feedback
Release history Release notifications | RSS feed
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
|Filename, size||File type||Python version||Upload date||Hashes|
|Filename, size myqueue-22.1.0.tar.gz (68.4 kB)||File type Source||Python version None||Upload date||Hashes View|
|Filename, size myqueue-22.1.0-py3-none-any.whl (82.8 kB)||File type Wheel||Python version py3||Upload date||Hashes View|