Skip to main content

SAGA-based Pilot-Job Implementation

Project description


The SAGA BigJob framework is a SAGA-based pilot job implementation. The Simple API for Grid Applications (SAGA) is a high-level, easy-to-use API for accessing distributed resources. SAGA BigJob supports a wide range of application types, and is usable over a broad range of infrastructures, i.e., it is general-purpose, extensible and interoperable. Unlike other common pilot job systems SAGA BigJob
(i) natively supports MPI job and
(ii) works on a variety of back-end systems, generally reflecting the advantage of using a SAGA-based approach.

Web Page & Mailing List

Web page:

Mailing list:




$ curl -o
$ python $HOME/.bigjob/python/
$ . $HOME/.bigjob/python/bin/activate

See for further information. The Wiki includes among many things information on how to setup BigJob on various machines, e.g. on XSEDE (Ranger, Kraken) or FutureGrid.

Configuration of Redis Coordination Backend

BigJob requires [Redis]( for communication between BigJob manager/agent. Use the following steps to install Redis.

Mac Homebrew:

brew install redis redis-tools hiredis

Download Redis
Currently only tested with Redis 2.2.12

$ tar -xzvf redis-2.2.12.tar.gz
$ cd redis-2.2.12
$ make
$ make install (if root)

Start Redis (redis-server executable is located in src/ dir):

cd src

Alternative coordination services (EXPERIMENTAL - NOT RECOMMENDED!)

BigJob supports different other communication & coordination backends via a plugin mechanism.

### A) SAGA Advert Service

1) Utilise coordination url parameter of BigJob to utilize the Advert Service:

advert:// (Postgresql)
advert://localhost/ (SQLite)

2) Utilise coordination URL parameter of BigJob to connect to Redis instance:


Standard settings are:


### B) ZMQ

1.) Install ZeroMQ and pyzmq

1.1 Install ZMQ:

$ wget
$ tar -xzvf zeromq-2.1.7.tar.gz
$ cd zeromq-2.1.7
$ ./configure --prefix=$HOME/sw/zeromq-2.1.7
$ make install

1.2 Activate BigJob virtualenv:

. $HOME/.bigjob/python/bin/activate

1.3 Install Python bindings:

$ export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$HOME/sw/zeromq-2.1.7/lib/
$ export ZMQ_DIR=$HOME/sw/zeromq-2.1.9/

$ pip install pyzmq

2) Utilize coordination url parameter of BigJob to utilize ZMQ:


will start a local tcp server.



* setuptools >0.6c11,

* setuptools-git >0.3.3,

Building PyPi package

python build

Upload to PyPi

python sdist upload

Project details

Release history Release notifications | RSS feed

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

BigJob-0.4.122.tar.gz (227.4 kB view hashes)

Uploaded source

Supported by

AWS AWS Cloud computing Datadog Datadog Monitoring Facebook / Instagram Facebook / Instagram PSF Sponsor Fastly Fastly CDN Google Google Object Storage and Download Analytics Huawei Huawei PSF Sponsor Microsoft Microsoft PSF Sponsor NVIDIA NVIDIA PSF Sponsor Pingdom Pingdom Monitoring Salesforce Salesforce PSF Sponsor Sentry Sentry Error logging StatusPage StatusPage Status page