Skip to main content

Utilities for remote submission and collection of GridEngine jobs.

Project description

# remote_sge: Remotely submit jobs to GridEngine

* Remotely submit jobs to a GridEngine cluster.
* Retrieve job results.
* Automated transfer of jobs from one cluster to another.

Remote SGE is a tool for transferring queued jobs to a remote SGE cluster,
as well as for submitting jobs to SGE from a machine other than the cluster's
master host.

* [See here]( for common configuration problems.

# Usage

# Holds jobs with ids in the given range, collects all the required data, and
# sends them to the configured remote.
remote_sge shuttle 12300-12399

# Later...
# Downloads results from any remote jobs that have finished, and unpacks the
# results in the working directory of the local jobs.
# Finally, terminates (deletes) the original jobs from the local queue.
remote_sge collect

# Installation

The installation utility targets Centos 6.x and Amazon Linux 1 (cfncluster).

## Install Base Packages (Client AND Server)

# Feel free to change PY_VERSION to the latest, or to pypy or your favorite
# version available within pyenv.
curl -o \
-L \
&& chmod +x
./ <path to wheel file>

# Restart your shell, you can log out and log in again, or else:

This will:

* Install YUM packages required for pyenv.
* Install latest version of pyenv.
* Install Python 3.6.4 and create a virtualenv called `remote_sge`, based on that version.

## Install the Remote SGE package. (Client AND Server)

You can also type `pyenv activate remote_sge` and then `pip` and other executables will be added
to your path.

First, download the latest wheel from

# Activate your virtualenv:
pyenv activate remote_sge
# Install the package:
pip install /path/to/remote-sge-xxx.whl

## Install the Remote SGE Server Components.

# Activate your virtualenv if not already active:
pyenv activate remote_sge
# Type remote_sge install_server -h for a full list of command-line options.
# The below assumes that vim is installed and you know how to use it.
remote_sge install_server -i

The installer will ask you to edit a number of configuration files. Make changes as needed,
save and quit the editor, and the file will be copied into place.

Here are the particular settings to pay attention to:

### Main Config File:

* Client Section:
* You can safely remove this section from the server component; won't be used.
* Server Section:
* `work_root` should reflect the location where each job's working directory should run.
Each job will be given a randomly-named working directory within this location.
* `completed_files_root` should reflect the location where job results will be placed.
* `nginx_listen` can be a port number, or an `ip_address:port` assignment.
* Command Environment Section:
* A list of environment variables. Use this section to configure locations of any resources
your jobs might use, which are located outside of the job's working directory.

## Test the function of the server component:

# Change the location of the config dir, if you changed the default for config files.
remote_sge test_server ~/.config/remote_sge -P 443 -h localhost

If the server is configured correctly and running, you'll see `It works!`, and you should continue.

## Configure the client component:

The server install will have placed the required client certificates in a tgz file at

# Copy the certificate files to the client machine
ssh <client_machine>
scp me@server_machine:~/remote_sge_client_certs.tgz .

# Do the base install (see Install Base Packages above):
curl -L | PY_VERSION=3.6.4 bash

# Restart your shell:

pyenv activate remote_sge
# Ensure that the wheel file has been downloaded to this machine, and then:
pip install /path/to/remote-sge-xxx.whl

remote_sge install_client -i -c /path/to/remote_sge_client_certs.tgz

You'll be prompted to correct the hostname and port number of your server instance.
The other settings can likely be left alone.

## Test client connectivity

Just to make sure that you have connectivity to the server:

remote_sge test_server ~/.config/remote_sge -P 443 -h

# Set up for development:

``pip install -e .[testing]``

Project details

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for remote-sge, version 0.2.8
Filename, size File type Python version Upload date Hashes
Filename, size remote_sge-0.2.8-py3-none-any.whl (50.5 kB) File type Wheel Python version py3 Upload date Hashes View

Supported by

AWS AWS Cloud computing Datadog Datadog Monitoring DigiCert DigiCert EV certificate Facebook / Instagram Facebook / Instagram PSF Sponsor Fastly Fastly CDN Google Google Object Storage and Download Analytics Pingdom Pingdom Monitoring Salesforce Salesforce PSF Sponsor Sentry Sentry Error logging StatusPage StatusPage Status page