spire-pipeline 0.6.3

Run software pipelines using YAML files

Welcome to Spire

Spire is a pipeline runner based on YAML files and Ninja. The pipeline descriptions (i.e. the list of steps to run) are stored in Jinja-templated YAML files which allow both readability and modularity. Using Ninja as a backend ensures fast and correct dependency handling.

Quickstart

Install Ninja, following official instructions.

From the source directory, use pip to ensure that dependencies are installed along with Spire:

pip install spire-pipeline

A pipeline description is a templated YAML file containing a list of steps, each step having the following fields:

• id: unique name to reference this step from other steps
• inputs: files that must be present for the step to execute
• outputs: files that will be generated by the step
• commands: a list of commands (optional for phony rules, see below)
• phony: flag indicating whether the target is phony i.e. does not produce any output (optional, defaults to false)

The id and outputs, and inputs fields are mandatory for all steps. For non-phony rules, the commands is optional. The inputs, outputs and commands may be written either as lists or single values.

The YAML content can be templated using Jinja, with extra features:

• Filters to convert to YAML (foo|yaml) or to JSON (foo|json)
• Function to perform globbing in the directory where the build will be executed (glob("*.c"))
• The basename and dirname functions from the Python module os.path
• The current build directory (build_directory)
• The absolute path to the directory containing the current pipeline (pipeline_directory)
• References to the outputs or inputs of the current step or of another step, specified by its identifier ({{ inputs("other_step_id") }}, {{ outputs("other_step_id") }}). These references are always list.
• Functions to load data from a file in CSV, JSON or YAML format (respectively load_csv, load_json, load_yaml).

The following example demonstrates most of those features. The first step looks for .c files and compile them to .o; the second step links them together to build an executable.

steps:
  - id: compile
    inputs: {{ glob("*.c") }}
    outputs: {{ inputs("compile")|replace(".c", ".o") }}
    commands: gcc -c {{ inputs("compile")|join(" ") }}

  - id: link
    inputs: {{ outputs("compile") }}
    outputs: foo
    commands: 
      - gcc -o {{ outputs("link")[0] }} {{ inputs("link")|join(" ") }}
      - strip {{ outputs("link")[0] }}

If you use Atom and the atom-jinja2 package, save it as .yml.j2 for better syntax highlighting.

The runner takes two sets of options: everything before a litteral -- is passed to the parser, everything after is passed to the build engine. The parser expects the file name containing the pipeline description (mandatory), and a list of variable definitions that will be fed to Jinja, in the form key=value.

The following willl parse build.yml.j2 with the variable root containing the current working directory, and will then build the target foo.

spire build.yml.j2 root=$(pwd) -- foo

Design

The input data is a text file containing a YAML-templated description of the pipeline steps. Three entities will process this data in order to run the pipeline:

• Parser: run the template engine (e.g. globbing, root directory, number of slots for parallel jobs, etc.) and handle cross-step references
• Generator: convert the rendered description to build-engine specific file
• Build engine: run the build itself with user-specified options

Under this design, multiple build engines and their corresponding generators may be implemented; the build engine of choice is Ninja, due to its ability to handle multiple outputs per step and to its parallel build features.