Run software pipelines using YAML files
Project description
Welcome to Spire
Spire is a pipeline runner based on YAML files and Ninja. The pipeline descriptions (i.e. the list of steps to run) are stored in Jinja-templated YAML files which allow both readability and modularity. Using Ninja as a backend ensures fast and correct dependency handling.
Quickstart
Install Ninja, following official instructions.
From the source directory, use pip
to ensure that dependencies are installed along with Spire:
pip install .
A pipeline description is a templated YAML file containing a list of steps
, each step having the following fields:
id
: unique name to reference this step from other stepsinputs
: files that must be present for the step to executeoutputs
: files that will be generated by the stepcommands
: a list of commands (optional forphony
rules, see below)phony
: flag indicating whether the target is phony i.e. does not produce any output (optional, defaults tofalse
)
The id
and outputs
, and inputs
fields are mandatory for all steps. For non-phony
rules, the commands
is optional. The inputs
, outputs
and commands
may be written either as lists or single values.
The YAML content can be templated using Jinja, with extra features:
- Filters to convert to YAML (
foo|yaml
) or to JSON (foo|json
) - Function to perform globbing in the directory where the build will be executed (
glob("*.c")
) - The
basename
anddirname
functions from the Python moduleos.path
- The current build directory (
build_directory
) - The absolute path to the directory containing the current pipeline (
pipeline_directory
) - References to the outputs or inputs of the current step or of another step, specified by its identifier (
{{ inputs("other_step_id") }}
,{{ outputs("other_step_id") }}
). These references are always list. - Functions to load data from a file in CSV, JSON or YAML format (respectively
load_csv
,load_json
,load_yaml
).
The following example demonstrates most of those features. The first step looks for .c
files and compile them to .o
; the second step links them together to build an executable.
steps:
- id: compile
inputs: {{ glob("*.c") }}
outputs: {{ inputs("compile")|replace(".c", ".o") }}
commands: gcc -c {{ inputs("compile")|join(" ") }}
- id: link
inputs: {{ outputs("compile") }}
outputs: foo
commands:
- gcc -o {{ outputs("link")[0] }} {{ inputs("link")|join(" ") }}
- strip {{ outputs("link")[0] }}
If you use Atom and the atom-jinja2 package, save it as .yml.j2
for better syntax highlighting.
The runner takes two sets of options: everything before a litteral --
is passed to the parser, everything after is passed to the build engine. The parser expects the file name containing the pipeline description (mandatory), and a list of variable definitions that will be fed to Jinja, in the form key=value
.
The following willl parse build.yml.j2
with the variable root
containing the current working directory, and will then build the target foo
.
spire build.yml.j2 root=$(pwd) -- foo
Design
The input data is a text file containing a YAML-templated description of the pipeline steps. Three entities will process this data in order to run the pipeline:
- Parser: run the template engine (e.g. globbing, root directory, number of slots for parallel jobs, etc.) and handle cross-step references
- Generator: convert the rendered description to build-engine specific file
- Build engine: run the build itself with user-specified options
Under this design, multiple build engines and their corresponding generators may be implemented; the build engine of choice is Ninja, due to its ability to handle multiple outputs per step and to its parallel build features.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distributions
Hashes for spire_pipeline-0.6.0-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | aad9b29d9d2ae8c6dea8286418b9410fb674045e499278bd8a02eea3585bb215 |
|
MD5 | 36d1efb0b45f880f6f239338c1f758aa |
|
BLAKE2b-256 | c15be20e993f05d581d8d63a8a7ccef3e42174f4f5b28a291a6b9b52d39fcf9b |
Hashes for spire_pipeline-0.6.0-py2-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 1dbbcd01b2175f68d3a50fbcf9f17c0f903a23dc5a792c1d6765d37df935358d |
|
MD5 | e5f9270d34520dc25d92a5ad7678f011 |
|
BLAKE2b-256 | e89d6aa452a7e5bb03eb7fd57971001ef0caa80c87dd67257e9a71e24e5efb3b |