stjudecloud-oliver 1.0.1

An opinionated Cromwell orchestration system

Oliver

An opinionated Cromwell orchestration manager.

Getting Started

pip install stjudecloud-oliver

Please refer to the guides in the docs/ folder for more information.

Guide Name	Link
Advanced Usage	Link
Configuration	Link
Development	Link

Usage

oliver -h

The following subcommands are currently supported.

Subcommand	Short Command	Description
aws		All subcommands related to Cromwell on AWS.
azure		All subcommands related to Cromwell on Azure.
aggregate	a	Aggregate all results to a local or cloud folder for a run.
batches	b	Explore batches of jobs submitted to Cromwell.
configure		Configure Oliver with default options.
config		Set or get a single config value from Oliver.
inputs		Find all reported outputs for a given workflow.
inspect	i	Describe the state of a Cromwell workflow.
kill	k	Kill a workflow running on a Cromwell server.
logs	l	Find all reported logs for a given workflow.
outputs	o	Find all reported outputs for a given workflow.
retry	re	Resubmit a workflow with the same parameters.
runtime	ru	Get the runtime attributes used for a specific call.
status	st	Report various statistics about a running Cromwell server.
submit	su	Submit a workflow to the Cromwell server.

Submitting Jobs

One of the novel features of oliver is the ease in which workflow parameters can be set on the command line. Typically when submitting a workflow, one must specify a workflowInputs (could be one of many), workflowOptions, and labels JSON file to Cromwell.

When you use oliver submit, you can easily specify files or individual key-value pairs to be included in the parameters above. For instance, passing inputs.json on the command line will read all key-value pairs from inputs.json and add them to the inputs dictionary. Individual key-value pairs can be passed like key=value. Values passed later are processed sequentially, meaning that later arguments overwrite any key-value pairs set by previous ones.

Additionally, argument passed on the command line can encode each the different parameter types for a Cromwell workflow:

Parameter Type	Prefix	Example
Input	<none>	key=value
Option	@	@key=value
Label	%	%key=value

For example, consider the following command:

oliver submit workflow.wdl \
    default-inputs.json \      # loads all values in the JSON file to the inputs object.
    @default-options.json \    # loads all values in the JSON file to the options object.
    %default-labels.json \     # loads all values in the JSON file to the labels object.
    input_key=value \          # adds `input_key=value` to the inputs object (overwrites the value if `input_key` set in default-inputs.json).
    @option_key=value \        # adds `option_key=value` to the options object (overwrites the value if `option_key` set in default-options.json).
    %label_key=value \         # adds `label_key=value` to the labels object (overwrites the value if `label_key` set in default-labels.json).

Tests

Oliver provides a (currently patchy) set of tests — both unit and end-to-end. To get started with testing, you'll need to bootstrap a Docker test environment (one-time operation).

# Start development environment
docker-compose up --build -d

# Seed development environment (make sure Cromwell is live first!)
chmod +x seeds/seed.sh
seeds/seed.sh http://localhost:8000 seeds/wdl/hello.wdl

To reset your entire docker-compose environment, you can run the following:

docker-compose down

docker image rm oliver:latest
docker image rm oliver_cromwell:latest
docker image rm mysql:5.7
docker volume rm oliver_mysql_data

docker-compose up --build -d

Author

👤 St. Jude Cloud Team

• Website: https://stjude.cloud
• Twitter: @StJudeResearch
• Github: @stjudecloud

🤝 Contributing

Contributions, issues and feature requests are welcome!
Feel free to check issues page. You can also take a look at the contributing guide.

📝 License

Copyright © 2020 St. Jude Cloud Team.
This project is MIT licensed.