Skip to main content

Python script to submit torque jobs in a very hackish way

Project description

This is as very simple hackish hack, that allows you to any serializable python callable as a torque job.

Features:

  • Allows you to send enviorment on the other side
  • Allows to execute any callable function as a torque task
  • Allows you to launch a bunch of taska as a torque array job.

Dependencies:

  • Working python 2.7/3.4 enviorment
  • Some python dependencies (see REQUIREMENTS)
  • Working qsub command (no need for other PBS/torque stuff).

Works by serializing the enviorment and callable function to the enviorment variables (or enviorment variables and files). For serialization we use dill if avilable or pickle.

Enviorment can be initialized using arbirtary bash script — this script will be sourced before running provided python callable.

Example

Example without enviorment:

 callable = partial(print, "Hello World!")

from torqsubmit import Submitter

s = Submitter()
s.tasks = [callable]
s.submit()

Example with enviorment:

callable = partial(print, "Hello World!")

from torqsubmit._submit import Submitter

ENV = """
source ${HOME}/.bashrc
workon torque-submit
export MSG="Hello World!"
"""

def print_from_env():
    import os
    print(os.environ["MSG"])

s = Submitter()
s.tasks = [callable]
s.enviorment = ENV
s.submit()

Example submitting many tasks:

from __future__ import print_function
from functools import partial

callable = partial(print, "Hello World!")

from torqsubmit._submit import Submitter


ENV = """
source ${HOME}/.bashrc
workon torque-submit
export MSG="Hello World!"
"""


def print_from_env():
    import os
    print(os.environ["MSG"])


s = Submitter()
s.enviorment = ENV
s.tasks = [print_from_env, print_from_env]
s.submit()

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Filename, size & hash SHA256 hash help File type Python version Upload date
torque-submit-0.0.3.tar.gz (5.7 kB) Copy SHA256 hash SHA256 Source None

Supported by

Elastic Elastic Search Pingdom Pingdom Monitoring Google Google BigQuery Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN SignalFx SignalFx Supporter DigiCert DigiCert EV certificate StatusPage StatusPage Status page