Skip to main content

The ICEBERG workflow middleware

Project description

ICEBERG

The master branch generally reflects the ICEBERG release on Pypi, and is considered stable: it should work 'out of the box'. Please refer to the documentation.

The devel branch (and any other branches than master, for that matter) may not correspond to the published documentation, and specifically may have dependencies which need to be resolved manually. Please contact us with an issue if you need advice on the usage of any non-master branch.

Devel Build Status codecov

Installation:

Create a Python Virtual Environment (VirtualEnv or Conda) and activate it.

To install our package from source, you should clone the repo and execute pip install from its root folder:

git clone https://github.com/ICEBERG-project/ICEBERG-middleware.git
cd ICEBERG-middeware
pip install . --upgrade

To install from PyPi

pip install iceberg

Usage

Before using ICEBERG connections to a MongoDB server and one to a RabbitMQ server are needed. To setup a password-protected database with Docker, see these instructions here. To setup RabbitMQ with Docker see these instructions. Once the two servers are available, export the following variables with appropriate values:

export RADICAL_PILOT_DBURL=mongodb://usernane:password@mongodb_hostname:port/db_name
export RMQ_ENDPOINT=ip_value
export RMQ_PORT=port_number
export VE_SEALS=ve_path/ve_name

The ICEBERG command can be executed as follows:

iceberg [-h] [--resource RESOURCE] [--queue QUEUE] [--cpus CPUS]
               [--gpus GPUS] [--input_path INPUT_PATH]
               [--output_path OUTPUT_PATH] [--walltime WALLTIME]
               [--analysis ANALYSIS]
               {seals,penguins,4Dgeolocation,rivers,landcover} ...
               

The arguments shown are mandatory for the command to execute correctly. Executing with -h will print the command help page:


positional arguments:
  {seals,penguins,4Dgeolocation,rivers,landcover}

optional arguments:
  -h, --help            show this help message and exit

Required Arguments:
  --resource RESOURCE, -r RESOURCE
                        Where the execution will happen
  --queue QUEUE, -q QUEUE
                        The queue of the resource
  --cpus CPUS, -c CPUS  How many CPUs will be required
  --gpus GPUS, -g GPUS  How many GPUs will be required
  --input_path INPUT_PATH, -ip INPUT_PATH
                        Where the input images are
  --output_path OUTPUT_PATH, -op OUTPUT_PATH
                        Where the results should be saved
  --walltime WALLTIME, -w WALLTIME
                        The estimated execution time

Currently, the available analysis commands are:

  1. seals
  2. penguins
  3. 4DGeolocation
  4. landcover
  5. rivers

Based on the selected analysis further arguments may be required.

To see further help messages per use case execute the ICEBERG command with all the arguments and add a --help after it. It will print the arguments for the selected alanysis.

0.0.0

  • Simplifying the command line interface to support subcommands

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for iceberg, version 0.1.1
Filename, size File type Python version Upload date Hashes
Filename, size iceberg-0.1.1.tar.gz (14.0 kB) File type Source Python version None Upload date Hashes View hashes

Supported by

Elastic Elastic Search Pingdom Pingdom Monitoring Google Google BigQuery Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN DigiCert DigiCert EV certificate StatusPage StatusPage Status page