Runs pipelines configured in YAML
Project description
Pipeline Runner
When running, the configuration folder should have the following structure
example_configuration_folder
├── pipelines.yaml
├── variables.yaml
├── functions.py
├── templates
│ ├── script.jinja2
│ ├── singularity.jinja2
│ ├── etc.jinja2
pipelines.yaml
The root element should be a dictionary in which the keys = pipeline names, the values = an array of tasks to run. For a full example look at example/pipelines.yaml. The possible tasks are:
set_variables: dict
Directly define or override specific variables. Example:
- set_variables:
HOME: /scratch/$USER/home/
NEW_VARIABLE: Superman
load_variables: str
This will load the flat dictionary of key str. Example:
- load_variables: pipeline_one
dump_variables: filename
This will dump all of the current variables into a file, specified by filename. Example:
- dump_variables: $SUBJECT/${SESSION}.sh
# Assumming SUBJECT + SESSION have been defined in a prior step
generate_file: dict
Create a file from a Jinja2 template. This step accepts a yaml dictionary with the following key-value pairs:
- template
- The Jinja2 template file to use relative to the configuration_dir/templates/
- filepath
- The full path onto which the new file should be saved to.
- variable
- The dynamic value of filepath is converted to a static absolute path and saved to the specifed variable. The default value = "OUTPUT_FILE".
Example:
- generate_file:
template: script.jinja2
filepath: $WORKING_DIR/$SUBJECT/$SESSION_PROCESSING.sh
variable: PROCESSING_SCRIPT
# now $PROCESSING_SCRIPT can be used in a future step
# for example, a function that adds the script on the PBS queue
function: str
The function to call. Note: The functions must be defined in configuration_dir/functions.py
Example:
- function: load_scripts_onto_pbs
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.