Skip to main content

Deploy workflows on laptops, servers, or HPC/HTC clusters.

Project description

PlantIT CLI

PyPI version Build Status Coverage Status

Deploy workflows on laptops, servers, or HPC/HTC clusters.

This project is in open alpha and is not yet stable.

Contents

Requirements

Installation

To install, clone the project with git clone https://github.com/Computational-Plant-Science/plantit-cli.git or use pip:

pip3 install plantit-cli

Usage

To run a workflow defined in workflow.yaml, use plantit workflow.yaml --token <PlantIT API authentication token>. The YAML schema should look something like this:

identifier: a42033c3-9ba1-4d99-83ed-3fdce08c706e
image: docker://alpine
workdir: /your/working/directory
command: echo $MESSAGE
params:
- key: message
  value: Hello, plant person!
executor:
  local:
api_url: http://plantit/apis/v1/runs/a42033c3-9ba1-4d99-83ed-3fdce08c706e/update_target_status/

Taking the elements one at a time:

  • identifier: the workflow run identifier (GUID)
  • image: the Docker or Singularity container image
  • workdir: where to execute the workflow
  • command: the command(s) to run inside the container
  • params: parameters substituted when command runs
  • executor: how to execute the pipeline (e.g., locally or on an HPC/HTC resource manager)
  • api_url: the PlantIT API endpoint to relay run status updates

Executor

The executor option specifies how to run the workflow on underlying computing resources. Currently local, pbs, and slurm executors are supported. If no executor is specified in the job definition file, the CLI will default to the local executor.

To use the PBS executor, add an executor section like the following:

executor:
  pbs:
    cores: 1
    memory: 1GB
    walltime: '00:10:00'
    processes: 1
    local_directory: "/your/scratch/directory"
    n_workers: 1

To use the SLURM executor:

executor:
  slurm:
    cores: 1
    memory: 1GB
    walltime: '00:10:00'
    processes: 1
    local_directory: "/your/scratch/directory"
    n_workers: 1
    partition: debug

Input/Output

The plantit-cli can automatically copy input files from the CyVerse Data Store onto the local (or network) file system, then push output files back to the Data Store after your workflow runs. To direct plantit-cli to pull an input file or directory, add an input section (the file or directory name will be substituted for $INPUT when the workflow's command is executed).

To configure a workflow to pull a single file from the Data Store and spawn a single container to process it, use kind: file and from: <file path>:

input:
  kind: file
  from: /iplant/home/username/directory/file

To configure a workflow to pull the contents of a directory from the Data Store and spawn a single container to process it, use kind: directory and from: <directory path>:

input:
  kind: directory
  from: /iplant/home/username/directory

To configure a workflow to pull a directory from the Data Store and spawn multiple containers to process files in parallel, use kind: file and from: <directory path>:

input:
  kind: file
  from: /iplant/home/username/directory

To configure a flow to push files matching a pattern to the Data Store after container execution (the local path will be substituted for $OUTPUT when the workflow's command is executed):

output:
  pattern: xslx
  from: directory # relative to the workflow's working directory
  to: /iplant/home/username/collection

Authenticating against the Terrain API

The CLI uses the Terrain API to query and access data in the CyVerse Data Store and expects a --cyverse_token flag.

Examples

Example configuration files can be found in examples/.

Tests

Before running tests, run scripts/bootstrap.sh. Then run:

docker-compose -f docker-compose.test.yml run cluster /bin/bash -- pytest . -s

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for plantit-cli, version 0.0.40
Filename, size File type Python version Upload date Hashes
Filename, size plantit_cli-0.0.40-py3-none-any.whl (18.8 kB) File type Wheel Python version py3 Upload date Hashes View
Filename, size plantit-cli-0.0.40.tar.gz (13.3 kB) File type Source Python version None Upload date Hashes View

Supported by

Pingdom Pingdom Monitoring Google Google Object Storage and Download Analytics Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN DigiCert DigiCert EV certificate StatusPage StatusPage Status page