This is a pre-production deployment of Warehouse, however changes made here WILL affect the production instance of PyPI.
Latest Version Dependencies status unknown Test status unknown Test coverage unknown
Project Description
Farmpy
======


This is a Python3 package to handle job submission to a compute farm. Currently supports Platform's LSF. It has a script to make submitting jobs easier and also an API to run jobs from within scripts.


Installation
------------

We assume that you are using Platform's LSF, i.e. it is installed already with bsub and lsadmin in your path.

Run the tests:

python3 setup.py test

Install:

python3 setup.py install


Synposis - command line
-----------------------

### Submitting jobs ###

To submit a job called "name" that asks for 1GB of memory, runs the script foo.sh and writes stdout and stderr to name.o and name.e:

bsub.py 1 name foo.sh

To submit a job that will only run when job with ID 42 has finished, use:

bsub.py --done 42 1 name foo.sh

If you want 10GB of /tmp space on the node:

bsub.py --tmp_space 10 1 name foo.sh

There are many more options. Use -h or --help to see the full list of options

bsub.py --help

### Getting stats from finished jobs ###

To get a tsv file of CPU, memory usage etc of a job:

bsub_out_to_stats bsub.output

This works with muliple files. e.g.:

bsub_out_to_stats bsub.output1 bsub.output2

or

bsub_out_to_stats *.output

Each file can contain the output of more than one job.


Synposis - running within a script
----------------------------------

Amongst other things, you run a job, set dependencies, change queues and resources and run arrays. Use

help(lsf)

to find out more.

### Make a job and run it ###

job1 = lsf.Job('out', 'err', 'name', 'normal', 1, 'foo.sh')
job1.run()

This created a job called 'name', which will run in the normal queue. It asked for 1GB of memory and will write stdout/stderr to the files out/err. It runs the script run.sh.

### Dependencies ###

When the job was run, the variable job1.job_id was set, so that you can use dependencies. For example:

job2 = lsf.Job('out2', 'err2', 'job2', 'normal', 1, 'run2.sh')
job2.add_dependency(job1.job_id)
job2.run()

Alternatively, dependcies can be set using job names, but these might not be unique so IDs are safer. If you want to use a name instead:

job2.add_dependency(job1.name)

### Job arrays ###

You can run a job array using start= and end= when constructing a Job. Example:

job = ('out', 'err', 'name', 'normal', 1, 'run.sh INDEX', start=1, end=10)

This sets up a job array with 10 elements. stdout files will be called out.1, out.2, ....etc and similarly for stderr. Every appearance of 'INDEX' in the command is translated to be the job index (technically, the $LSB_JOBINDEX environment variable). So this would submit 10 jobs to LSF:

run.sh 1
run.sh 2
...
run.sh 10
Release History

Release History

0.3.1

This version

History Node

TODO: Figure out how to actually get changelog content.

Changelog content for this version goes here.

Donec et mollis dolor. Praesent et diam eget libero egestas mattis sit amet vitae augue. Nam tincidunt congue enim, ut porta lorem lacinia consectetur. Donec ut libero sed arcu vehicula ultricies a non tortor. Lorem ipsum dolor sit amet, consectetur adipiscing elit.

Show More

0.2.1

History Node

TODO: Figure out how to actually get changelog content.

Changelog content for this version goes here.

Donec et mollis dolor. Praesent et diam eget libero egestas mattis sit amet vitae augue. Nam tincidunt congue enim, ut porta lorem lacinia consectetur. Donec ut libero sed arcu vehicula ultricies a non tortor. Lorem ipsum dolor sit amet, consectetur adipiscing elit.

Show More

Download Files

Download Files

TODO: Brief introduction on what you do with files - including link to relevant help section.

File Name & Checksum SHA256 Checksum Help Version File Type Upload Date
farmpy-0.3.1.tar.gz (13.4 kB) Copy SHA256 Checksum SHA256 Source Mar 29, 2016

Supported By

WebFaction WebFaction Technical Writing Elastic Elastic Search Pingdom Pingdom Monitoring Dyn Dyn DNS HPE HPE Development Sentry Sentry Error Logging CloudAMQP CloudAMQP RabbitMQ Heroku Heroku PaaS Kabu Creative Kabu Creative UX & Design Fastly Fastly CDN DigiCert DigiCert EV Certificate Rackspace Rackspace Cloud Servers DreamHost DreamHost Log Hosting