Skip to main content

Simple job queue

Project description

https://gitlab.com/myqueue/myqueue/badges/master/coverage.svg https://badge.fury.io/py/myqueue.svg https://joss.theoj.org/papers/10.21105/joss.01844/status.svg Documentation Status

MyQueue

MyQueue is a tool for submitting and keeping track of tasks running on a cluster of computers. It uses SLURM, PBS or LSF as a backend and makes handling of tasks easy. It has a command-line interface called mq with a number of sub-commands and a Python interface for managing workflows. Simple to set up: no system administrator or database required.

Quick links:

Examples

Submit Python script to 32 cores for 2 hours:

$ mq submit script.py -R 32:2h

Submit Python module abc.run in two folders:

$ mq submit abc.run F1/ F2/ -R 16:30m

Check results of tasks in current folder and its sub-folders:

$ mq list  # or mq ls
id  folder name      res.   age     state   time    error
--- ------ --------- ------ ------- ------- ------- ------
117 ./     script.py 32:2h  5:28:43 TIMEOUT 2:00:03
118 ./F1/  abc.run   16:30m 5:22:16 done      12:12
119 ./F2/  abc.run   16:30m 5:22:16 done      17:50
--- ------ --------- ------ ------- ------- ------- ------
done: 2, TIMEOUT: 1, total: 3

Resubmit with more resources (1 day):

$ mq resubmit -i 117 -R 32:1d

See more examples of use here:

Installation

MyQueue has only one dependency: Python version 3.7 or later.

Install MyQueue from PyPI with pip:

$ python3 -m pip install myqueue

Enable bash tab-completion for future terminal sessions like this:

$ mq completion >> ~/.profile

Now, configure your system as described here.

Release notes

See the release notes for a history of notable changes to MyQueue.

Help, support and feedback

If you need help, want to report a bug or suggest a new feature then you are welcome to get in touch via MyQueue’s issue tracker or the #myqueue room on Matrix.

Contributing

We welcome contributions to the code and documentation, preferably as merge-requests. More information here.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

myqueue-22.12.0.tar.gz (73.4 kB view hashes)

Uploaded Source

Built Distribution

myqueue-22.12.0-py3-none-any.whl (93.0 kB view hashes)

Uploaded Python 3

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page