Skip to main content

Shepherd GridEngine

Project description

Build Status Coverage Status PyPi Version License

Make Grid Engine a bit more useful from Python.


On the host:

On the worker nodes:

  • No requirements beyond the standard library
  • Tested on Python 2.7, Python 3.3
  • Should also work on Python 2.6 and 3.2, 3.4


MIT, see LICENSE file.


Running large map style operations on a Grid Engine cluster can be frustrating. Array jobs can only give scripts an input like some range() function call, but this is rarely sufficient. Collecting results is also a huge pain. Suddenly there are shell scripts and result files everywhere and you feel an overwhelming sense of mediocracy.

Sheepdog aims to make life better for a somewhat specific use case:

  1. You’re using Python. Hopefully even Python 3.3.
  2. You’ve got access to a Grid Engine cluster on some remote machines. They can also run Python, somehow or other. The cluster computers and your client computer can all communicate over a network.
  3. You have a function of several parameters and you want to run it many times with different arguments. Results should come back nicely collated, and are reasonably small (you’re not too worried if argument or result objects get copied in memory).
  4. You’re a PhD student in Div F at CUED desperately trying to use fear effectively.

To accomplish these aims, Sheepdog:

  1. Takes your function and N tuples of arguments, marshals both
  2. Creates a mapping range(N) to arguments
  3. Starts a network interface (over HTTP)
  4. Starts a size N array job on the Grid Engine cluster, running the client
  5. Each client talks to the server to map its array job ID into an actual set of arguments, and fetches the Python function to execute as well
  6. The function is executed with the arguments
  7. The result is sent back over the network
  8. Results are collated against arguments

This is very similar to:

  • pythongrid. Almost identical. Sheepdog doesn’t have to be run on the cluster head, though. And can’t resubmit jobs or anything fancy like that. And isn’t dead.
  • gridmap. A fork of pythongrid that is actually active and looks quite nice! Maybe look at gridmap.
  • Celery. Yes. Pretty similar.
  • rq. Quite similar.
  • Resque. But Resque is written in Ruby, boo.
  • Every other distributed map compute queue thing ever written.


Ensure the GridEngine workers have Python available.


import sheepdog

def f(a, b):
    return a + b

args = [(1, 1), (1, 2), (2, 2)]
config = {"host": "fear"}

results =, args, config)
print("Received results:", results)
# Received results: [2, 3, 4]

There is also support for transferring other functions and variables (using the namespace parameter ns of map) and imports can be handled using global, for example:

def f(a, b):
    import numpy as np
    global np
    return g(a, b)

def g(a, b):
    return np.array((a, b)) ** 2

args = [(1, 2), (3, 4)]
namespace = {"g": g}
config = {"host": "fear"}

results =, args, config, namespace)

See the documentation for full details.


View Sheepdog on ReadTheDocs.

Project details

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for Sheepdog, version 0.2.2
Filename, size File type Python version Upload date Hashes
Filename, size Sheepdog-0.2.2.tar.gz (21.4 kB) File type Source Python version None Upload date Hashes View hashes

Supported by

Elastic Elastic Search Pingdom Pingdom Monitoring Google Google BigQuery Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN SignalFx SignalFx Supporter DigiCert DigiCert EV certificate StatusPage StatusPage Status page