Python suite to manage jobs on a Sun Grid Engine computing cluster. Based on [pySGE](https://github.com/jiahao/PySGE) it adds the ability to span a parameter space.
pyGRID uses setuptoos to create an installer to be distributed. Type
python setup.py sdist
in the repository directory in order to create a package. You can grab the archived file produced by setuptools in the `dist` subdirectory of the repository to distribute pyGRID.
If you don't have root privileges on the cluster and you want to install pyGRID in your home directory follow these steps first:
* Create the path `~/lib/python2.6/site-packages` (assuming you installed python version 2.6) in your home directory.
* Create the `~/bin` path.
* Add `export PYTHONPATH=~/lib/python2.6/site-packages:$PYTHONPATH` and `export PATH=~/bin:$PATH` to your `.bashrc` file (and run `. ~/.bashrc`).
Transfer the pyGRID archive file on the cluster and from a terminal run
$ tar -xzvf pyGRID-0.2.tar.gz
$ cd pyGRID-0.2
$ python setup.py develop --prefix=~
where `0.2` is the current version number of pyGRID.
From now on you can run pyGRID by simply typing `pyGRID` from anywhere in the cluster.
pyGRID accepts the path of an xml file as input and the name of simulation defined within it as a `sim_element` xml tag. The `sim_element` xml tag specifies the options of the job you want to run and is used by pyGRID to
* write a bash file to pass to qsub for running the job
* pass options to qsub, included the values of parameters as environment variables
* create an auxilliary xml file, with extentsion `.grid`, holding the properties of a submitted job
* detect crashed jobs and resubmit them
The syntax for the xml file used by pyGRID is documented in `pyGRID/examples/basicExample.xml` but in short pyGRID conforms to the syntax used by `qsub` to specify job options.
pyGRID command line options can be divided in three groups:
* XML file option. This option is *Required*.
* `-f`, specifies the path of the file to load.
* Simulation options. These are mutually *exlusive* and at least one is *Required*.
* `-s`, specifies the name of the simulation to load from the XML file.
* `-a`, loads all the simulations in the file.
* Action options. Tell pyGRID which action to perform woth the specified simulation. These are mutually *exclusive*.
* `-w`, write the shell script for the simulation.
* `-b`, write the shell script and submit a job for every combination of the parameters of the simulation
* `-c`, scan the output sctream of finished jobs and detect the crashed ones. The `JOB_ID` and `TASK_ID` of the crashed job is saved in the `.grid` auxilliary file.
* `-r`, resubmit crashed jobs.
To invoke the documentation for pyGRID command line option type `pyGRID --help`.
I have created some tests for pyGRID in order to ensure that basic functions are not broken during development.
Both these tasks require some sort of system in place to emulate `qsub` and its responses.
To run the tests install `nose` with `pip install nose` and type
in the repository main directory.
TODO: Brief introduction on what you do with files - including link to relevant help section.