Skip to main content

SAGA-based Pilot-Job Implementation

Project description

The SAGA BigJob framework is a SAGA-based pilot job implementation. The Simple API for Grid Applications (SAGA) is a high-level, easy-to-use API for accessing distributed resources. SAGA BigJob supports a wide range of application types, and is usable over a broad range of infrastructures, i.e., it is general-purpose, extensible and interoperable. Unlike other common pilot job systems SAGA BigJob (i) natively supports MPI job and (ii) works on a variety of back-end systems, generally reflecting the advantage of using a SAGA-based approach.

Installation

$ curl https://raw.github.com/saga-project/BigJob/master/bootstrap/bigjob-bootstrap.py -o bigjob-bootstrap.py $ python bigjob-bootstrap.py $HOME/.bigjob/python/ $ . $HOME/.bigjob/python/bin/activate

See https://github.com/saga-project/BigJob/wiki/ for further information. The Wiki includes among many things information on how to setup BigJob on various machines, e.g. on XSEDE (Ranger, Kraken) or FutureGrid.

Configuration of Coordination Backend

BigJob supports different communication & coordination backends via a plugin mechanism. Currently supported backends:

  1. SAGA Advert Service
  2. REDIS, http://redis.io/
  3. ZMQ (ZeroMQ), http://www.zeromq.org/

### A) SAGA Advert Service

  1. Utilise coordination url parameter of BigJob to utilize the Advert Service:

    advert://advert.cct.lsu.edu:8080/ (Postgresql) advert://localhost/ (SQLite)

### Redis

1.) Install and run Redis (http://redis.io/)

Mac Homebrew:
brew install redis redis-tools hiredis

Linux: Download Redis http://redis.io/download Currently only tested with Redis 2.2.12

$ tar -xzvf redis-2.2.12.tar.gz $ cd redis-2.2.12 $ make $ make install (if root)

Start Redis (redis-server executable is located in src/ dir):

cd src ./redis-server
  1. Utilise coordination URL parameter of BigJob to connect to Redis instance:

    redis://<hostname>:6379

Standard settings are:

REDIS_SERVER=”localhost” REDIS_SERVER_PORT=6379

### C) ZMQ

1.) Install ZeroMQ and pyzmq

http://www.zeromq.org/intro:get-the-software http://zeromq.github.com/pyzmq/index.html

1.1 Install ZMQ:

$ wget http://download.zeromq.org/zeromq-2.1.7.tar.gz $ tar -xzvf zeromq-2.1.7.tar.gz $ cd zeromq-2.1.7 $ ./configure –prefix=$HOME/sw/zeromq-2.1.7 $ make install

1.2 Activate BigJob virtualenv:

. $HOME/.bigjob/python/bin/activate

1.3 Install Python bindings:

$ export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$HOME/sw/zeromq-2.1.7/lib/ $ export ZMQ_DIR=$HOME/sw/zeromq-2.1.9/

$ pip install pyzmq

  1. Utilize coordination url parameter of BigJob to utilize ZMQ:

    tcp://*

will start a local tcp server.


Packaging

Requirements:

Building PyPi package

python setup.py build

Upload to PyPi

python setup.py sdist upload

Project details


Release history Release notifications

History Node

0.64.5

History Node

0.63.4

History Node

0.63.3

History Node

0.63.2

History Node

0.63.1

History Node

0.59

History Node

0.58

History Node

0.57

History Node

0.56

History Node

0.53

History Node

0.52

History Node

0.50

History Node

0.50d

History Node

0.50c

History Node

0.50b

History Node

0.4.134

History Node

0.4.132

History Node

0.4.131

History Node

0.4.130-1-g879bec5

History Node

0.4.128

History Node

0.4.127

History Node

0.4.126

History Node

0.4.125

History Node

0.4.124

History Node

0.4.123

History Node

0.4.122

History Node

0.4.121

History Node

0.4.120

History Node

0.4.119

History Node

0.4.118

History Node

0.4.117

History Node

0.4.116

History Node

0.4.115

History Node

0.4.114

History Node

0.4.113

History Node

0.4.110

History Node

0.4.109

History Node

0.4.108

History Node

0.4.107

History Node

0.4.106

History Node

0.4.105

History Node

0.4.104

History Node

0.4.103

History Node

0.4.102

History Node

0.4.101

History Node

0.4.100

History Node

0.4.99

History Node

0.4.98

History Node

0.4.97

History Node

0.4.96

History Node

0.4.95

History Node

0.4.89

History Node

0.4.88

History Node

0.4.87

History Node

0.4.86

History Node

0.4.85

This version
History Node

0.4.84

History Node

0.4.83

History Node

0.4.82

History Node

0.4.81

History Node

0.4.80

History Node

0.4.79

History Node

0.4.78

History Node

0.4.77

History Node

0.4.76

History Node

0.4.74

History Node

0.4.73

History Node

0.4.72

History Node

0.4.71

History Node

0.4.70

History Node

0.4.64

History Node

0.4.63

History Node

0.4.62

History Node

0.4.61

History Node

0.4.60

History Node

0.4.59

History Node

0.4.58

History Node

0.4.57

History Node

0.4.56

History Node

0.4.55

History Node

0.4.53

History Node

0.4.52

History Node

0.4.51

History Node

0.4.50

History Node

0.4.49

History Node

0.4.48

Supported by

Elastic Elastic Search Pingdom Pingdom Monitoring Google Google BigQuery Sentry Sentry Error logging CloudAMQP CloudAMQP RabbitMQ AWS AWS Cloud computing Fastly Fastly CDN DigiCert DigiCert EV certificate StatusPage StatusPage Status page