Skip to main content

Simple job queue

Project description

https://gitlab.com/myqueue/myqueue/badges/master/coverage.svg https://badge.fury.io/py/myqueue.svg https://joss.theoj.org/papers/10.21105/joss.01844/status.svg Documentation Status

MyQueue

MyQueue is a tool for submitting and keeping track of tasks running on a cluster of computers. It uses SLURM, PBS or LSF as a backend and makes handling of tasks easy. It has a command-line interface called mq with a number of sub-commands and a Python interface for managing workflows. Simple to set up: no system administrator or database required.

Features

  • Easy task submission:
    • from the command line: mq submit <task> -R <cores>:<time>
    • from Python: myqueue.submit(...)
  • Automatic restarting of timed-out/out-of-memory tasks with more time/cores
  • Remembers your finished and failed tasks
  • Powerful list command for monitoring
  • Can be used together with Python venv‘s
  • Folder-based Workflows

Quick links:

Examples

Submit Python script to 32 cores for 2 hours:

$ mq submit script.py -R 32:2h

Submit Python module abc.run in two folders:

$ mq submit abc.run F1/ F2/ -R 16:30m

Check results of tasks in current folder and its sub-folders:

$ mq list  # or mq ls
id  folder name      res.   age     state   time    error
--- ------ --------- ------ ------- ------- ------- ------
117 ./     script.py 32:2h  5:28:43 TIMEOUT 2:00:03
118 ./F1/  abc.run   16:30m 5:22:16 done      12:12
119 ./F2/  abc.run   16:30m 5:22:16 done      17:50
--- ------ --------- ------ ------- ------- ------- ------
done: 2, TIMEOUT: 1, total: 3

Resubmit with more resources (1 day):

$ mq resubmit -i 117 -R 32:1d

See more examples of use here:

Installation

MyQueue has only one dependency: Python version 3.6 or later.

Install MyQueue from PyPI with pip:

$ python3 -m pip install myqueue

Enable bash tab-completion for future terminal sessions like this:

$ mq completion >> ~/.profile

Now, configure your system as described here.

Release notes

See the release notes for a history of notable changes to MyQueue.

Help, support and feedback

If you need help, want to report a bug or suggest a new feature then you are welcome to get in touch via MyQueue’s issue tracker or the #myqueue stream on Zulip.

Contributing

We welcome contributions to the code and documentation, preferably as merge-requests. More information here.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for myqueue, version 20.11.3
Filename, size File type Python version Upload date Hashes
Filename, size myqueue-20.11.3-py3-none-any.whl (67.6 kB) File type Wheel Python version py3 Upload date Hashes View
Filename, size myqueue-20.11.3.tar.gz (54.3 kB) File type Source Python version None Upload date Hashes View

Supported by

Pingdom Pingdom Monitoring Google Google Object Storage and Download Analytics Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN DigiCert DigiCert EV certificate StatusPage StatusPage Status page