Automatic DRMAA scheduling with resource management for Jug
Project description
This project is currently experimental so bug reports are welcome.
Requirements
Installation
Install jug_schedule with:
pip install jug-schedule
and then simply add the following to your ~/.jug/config/jug_user_commands.py:
try: from jug_schedule.schedule import ScheduleCommand schedule = ScheduleCommand() except Exception as e: sys.stderr.write("Couldn't import schedule, error was {0}\n".format(e))
If you are running directly from git you can instead use:
import os import sys sys.path.insert(0, "/path/to/clone/of/jug_schedule/") try: from jug_schedule.schedule import ScheduleCommand schedule = ScheduleCommand() except Exception as e: sys.stderr.write("Couldn't import schedule, error was {0}\n".format(e))
Usage
If installed properly, running jug should now include a schedule subcommand.
Running it will try to detect a queueing system and submit jobs to it.
You can also append --verbose debug for additional information.
Configuration
jug_schedule relies on DRMAA for interaction with the queueing system.
In order to use jug_schedule your environment needs to define DRMAA_LIBRARY_PATH. If running env | grep DRMAA_LIBRARY_PATH returns no match, ask your system administrators for the location of this library.
Then use:
export DRMAA_LIBRARY_PATH=/usr/ge62u5/lib/lx24-amd64/libdrmaa.so
You only need to set this option on the environment that runs jug schedule.
Resources
An additional feature of jug_schedule is the ability to define job resources.
If you already know jug’s TaskGenerator decorator you can simply replace it with the following where applicable:
from jug_schedule.resources import ResourcesTaskGenerator @ResourcesTaskGenerator(cpu=10, mem=10, queue="default") def func(...): ...
Supported arguments include: cpu, mem (in MB) and queue.
Command-line options
The following options are available:
--script - command used to run jug on the cluster. Point this to a shell script if you need to setup jug's environment prior to execution --max-jobs - how big is the pool of jug jobs (max number of simultaneous jobs) --logs - where to write job logs. Defaults to a directory 'jug_logs' in the current directory. --cycle-time - how many seconds to wait between every interaction with the queue system. Defaults to 60 --exit-on-error - jug_schedule will abort as soon as one job fails. Default is to only stop when all work is done or all jobs fail.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for jug_schedule-0.1.3-py2.py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | dbdb2a0e4951aaa5bc6c2a78e71491939ca99ce218681f36ac0c1eae2e24c7ac |
|
MD5 | 736719756768b3f39ea084080f11d95d |
|
BLAKE2b-256 | b1648b5c63060f4900ed8d7fcea6a1e17b537326656f21ce2076ece428ddf863 |