torque-submit 0.0.3

Python script to submit torque jobs in a very hackish way

This is as very simple hackish hack, that allows you to any serializable python callable as a torque job.

Features:

• Allows you to send enviorment on the other side

• Allows to execute any callable function as a torque task

• Allows you to launch a bunch of taska as a torque array job.

Dependencies:

• Working python 2.7/3.4 enviorment

• Some python dependencies (see REQUIREMENTS)

• Working qsub command (no need for other PBS/torque stuff).

Works by serializing the enviorment and callable function to the enviorment variables (or enviorment variables and files). For serialization we use dill if avilable or pickle.

Enviorment can be initialized using arbirtary bash script — this script will be sourced before running provided python callable.

Example

Example without enviorment:

 callable = partial(print, "Hello World!")

from torqsubmit import Submitter

s = Submitter()
s.tasks = [callable]
s.submit()

Example with enviorment:

callable = partial(print, "Hello World!")

from torqsubmit._submit import Submitter

ENV = """
source ${HOME}/.bashrc
workon torque-submit
export MSG="Hello World!"
"""

def print_from_env():
    import os
    print(os.environ["MSG"])

s = Submitter()
s.tasks = [callable]
s.enviorment = ENV
s.submit()

Example submitting many tasks:

from __future__ import print_function
from functools import partial

callable = partial(print, "Hello World!")

from torqsubmit._submit import Submitter


ENV = """
source ${HOME}/.bashrc
workon torque-submit
export MSG="Hello World!"
"""


def print_from_env():
    import os
    print(os.environ["MSG"])


s = Submitter()
s.enviorment = ENV
s.tasks = [print_from_env, print_from_env]
s.submit()