Skip to main content

An opinionated Cromwell orchestration system

Project description

Oliver

Build Status PyPI License: MIT

An opinionated Cromwell orchestration manager.

Oliver Example

Getting Started

pip install stjudecloud-oliver

Please refer to the guides in the docs/ folder for more information.

Guide Name Link
Advanced Usage Link
Configuration Link
Development Link

Usage

oliver -h

The following subcommands are currently supported.

Subcommand Short Command Description
aws All subcommands related to Cromwell on AWS.
azure All subcommands related to Cromwell on Azure.
aggregate a Aggregate all results to a local or cloud folder for a run.
batches b Explore batches of jobs submitted to Cromwell.
configure Configure Oliver with default options.
config Set or get a single config value from Oliver.
inputs Find all reported outputs for a given workflow.
inspect i Describe the state of a Cromwell workflow.
kill k Kill a workflow running on a Cromwell server.
logs l Find all reported logs for a given workflow.
outputs o Find all reported outputs for a given workflow.
retry re Resubmit a workflow with the same parameters.
runtime ru Get the runtime attributes used for a specific call.
status st Report various statistics about a running Cromwell server.
submit su Submit a workflow to the Cromwell server.

Submitting Jobs

One of the novel features of oliver is the ease in which workflow parameters can be set on the command line. Typically when submitting a workflow, one must specify a workflowInputs (could be one of many), workflowOptions, and labels JSON file to Cromwell.

When you use oliver submit, you can easily specify files or individual key-value pairs to be included in the parameters above. For instance, passing inputs.json on the command line will read all key-value pairs from inputs.json and add them to the inputs dictionary. Individual key-value pairs can be passed like key=value. Values passed later are processed sequentially, meaning that later arguments overwrite any key-value pairs set by previous ones.

Additionally, argument passed on the command line can encode each the different parameter types for a Cromwell workflow:

Parameter Type Prefix Example
Input <none> key=value
Option @ @key=value
Label % %key=value

For example, consider the following command:

oliver submit workflow.wdl \
    default-inputs.json \      # loads all values in the JSON file to the inputs object.
    @default-options.json \    # loads all values in the JSON file to the options object.
    %default-labels.json \     # loads all values in the JSON file to the labels object.
    input_key=value \          # adds `input_key=value` to the inputs object (overwrites the value if `input_key` set in default-inputs.json).
    @option_key=value \        # adds `option_key=value` to the options object (overwrites the value if `option_key` set in default-options.json).
    %label_key=value \         # adds `label_key=value` to the labels object (overwrites the value if `label_key` set in default-labels.json).

Tests

Oliver provides a (currently patchy) set of tests — both unit and end-to-end. To get started with testing, you'll need to bootstrap a Docker test environment (one-time operation).

# Start development environment
docker-compose up --build -d

# Seed development environment (make sure Cromwell is live first!)
chmod +x seeds/seed.sh
seeds/seed.sh http://localhost:8000 seeds/wdl/hello.wdl

To reset your entire docker-compose environment, you can run the following:

docker-compose down

docker image rm oliver:latest
docker image rm oliver_cromwell:latest
docker image rm mysql:5.7
docker volume rm oliver_mysql_data

docker-compose up --build -d

Author

👤 St. Jude Cloud Team

🤝 Contributing

Contributions, issues and feature requests are welcome!
Feel free to check issues page. You can also take a look at the contributing guide.

📝 License

Copyright © 2020 St. Jude Cloud Team.
This project is MIT licensed.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

stjudecloud-oliver-1.0.5.tar.gz (32.5 kB view hashes)

Uploaded Source

Built Distribution

stjudecloud_oliver-1.0.5-py3-none-any.whl (42.8 kB view hashes)

Uploaded Python 3

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page