Run software pipelines using YAML files
Project description
# Welcome to Spire
Spire is a pipeline runner based on [YAML](https://en.wikipedia.org/wiki/YAML) files and [Ninja](https://ninja-build.org/). The pipeline descriptions (i.e. the list of steps to run) are stored in [Jinja](http://jinja.pocoo.org/)-templated YAML files which allow both readability and modularity. Using Ninja as a backend ensures fast and correct dependency handling.
## Quickstart
Install Ninja, following [official instructions](https://github.com/ninja-build/ninja/wiki/Pre-built-Ninja-packages).
From the source directory, use `pip` to ensure that dependencies are installed along with Spire:
```
pip install .
```
A pipeline description is a templated YAML file containing a list of `steps`, each step having the following fields:
* `id`: unique name to reference this step from other steps
* `inputs`: files that must be present for the step to execute
* `outputs`: files that will be generated by the step
* `commands`: a list of commands (optional for `phony` rules, see below)
* `phony`: flag indicating whether the target is phony i.e. does not produce any output (optional, defaults to `false`)
The `id` and `outputs`, and `inputs` fields are mandatory for all steps. For non-`phony` rules, the `commands` is optional. The `inputs`, `outputs` and `commands` may be written either as lists or single values.
The YAML content can be templated using Jinja, with extra features:
* Filters to convert to YAML (`foo|yaml`) or to JSON (`foo|json`)
* Function to perform globbing in the directory where the build will be executed (`glob("*.c")`)
* The `basename` and `dirname` functions from the Python module `os.path`
* The current build directory (`build_directory`)
* The absolute path to the directory containing the current pipeline (`pipeline_directory`)
* References to the outputs or inputs of the current step or of another step, specified by its identifier (`{{ inputs("other_step_id") }}`, `{{ outputs("other_step_id") }}`). These references are always list.
The following example demonstrates most of those features. The first step looks for `.c` files and compile them to `.o`; the second step links them together to build an executable.
```yaml
steps:
- id: compile
inputs: {{ glob("*.c") }}
outputs: {{ inputs("compile")|replace(".c", ".o") }}
commands: gcc -c {{ inputs("compile")|join(" ") }}
- id: link
inputs: {{ outputs("compile") }}
outputs: foo
commands:
- gcc -o {{ outputs("link")[0] }} {{ inputs("link")|join(" ") }}
- strip {{ outputs("link")[0] }}
```
If you use Atom and the [atom-jinja2](https://atom.io/packages/atom-jinja2) package, save it as `.yml.j2` for better syntax highlighting.
The runner takes two sets of options: everything before a litteral `--` is passed to the parser, everything after is passed to the build engine. The parser expects the file name containing the pipeline description (mandatory), and a list of variable definitions that will be fed to Jinja, in the form `key=value`.
The following willl parse `build.yml.j2` with the variable `root` containing the current working directory, and will then build the target `foo`.
```bash
spire build.yml.j2 root=$(pwd) -- foo
```
## Design
The input data is a text file containing a YAML-templated description of the pipeline steps. Three entities will process this data in order to run the pipeline:
* Parser: run the template engine (e.g. globbing, root directory, number of slots for parallel jobs, etc.) and handle cross-step references
* Generator: convert the rendered description to build-engine specific file
* Build engine: run the build itself with user-specified options
Under this design, multiple build engines and their corresponding generators may be implemented; the build engine of choice is Ninja, due to its ability to handle multiple outputs per step and to its parallel build features.
Spire is a pipeline runner based on [YAML](https://en.wikipedia.org/wiki/YAML) files and [Ninja](https://ninja-build.org/). The pipeline descriptions (i.e. the list of steps to run) are stored in [Jinja](http://jinja.pocoo.org/)-templated YAML files which allow both readability and modularity. Using Ninja as a backend ensures fast and correct dependency handling.
## Quickstart
Install Ninja, following [official instructions](https://github.com/ninja-build/ninja/wiki/Pre-built-Ninja-packages).
From the source directory, use `pip` to ensure that dependencies are installed along with Spire:
```
pip install .
```
A pipeline description is a templated YAML file containing a list of `steps`, each step having the following fields:
* `id`: unique name to reference this step from other steps
* `inputs`: files that must be present for the step to execute
* `outputs`: files that will be generated by the step
* `commands`: a list of commands (optional for `phony` rules, see below)
* `phony`: flag indicating whether the target is phony i.e. does not produce any output (optional, defaults to `false`)
The `id` and `outputs`, and `inputs` fields are mandatory for all steps. For non-`phony` rules, the `commands` is optional. The `inputs`, `outputs` and `commands` may be written either as lists or single values.
The YAML content can be templated using Jinja, with extra features:
* Filters to convert to YAML (`foo|yaml`) or to JSON (`foo|json`)
* Function to perform globbing in the directory where the build will be executed (`glob("*.c")`)
* The `basename` and `dirname` functions from the Python module `os.path`
* The current build directory (`build_directory`)
* The absolute path to the directory containing the current pipeline (`pipeline_directory`)
* References to the outputs or inputs of the current step or of another step, specified by its identifier (`{{ inputs("other_step_id") }}`, `{{ outputs("other_step_id") }}`). These references are always list.
The following example demonstrates most of those features. The first step looks for `.c` files and compile them to `.o`; the second step links them together to build an executable.
```yaml
steps:
- id: compile
inputs: {{ glob("*.c") }}
outputs: {{ inputs("compile")|replace(".c", ".o") }}
commands: gcc -c {{ inputs("compile")|join(" ") }}
- id: link
inputs: {{ outputs("compile") }}
outputs: foo
commands:
- gcc -o {{ outputs("link")[0] }} {{ inputs("link")|join(" ") }}
- strip {{ outputs("link")[0] }}
```
If you use Atom and the [atom-jinja2](https://atom.io/packages/atom-jinja2) package, save it as `.yml.j2` for better syntax highlighting.
The runner takes two sets of options: everything before a litteral `--` is passed to the parser, everything after is passed to the build engine. The parser expects the file name containing the pipeline description (mandatory), and a list of variable definitions that will be fed to Jinja, in the form `key=value`.
The following willl parse `build.yml.j2` with the variable `root` containing the current working directory, and will then build the target `foo`.
```bash
spire build.yml.j2 root=$(pwd) -- foo
```
## Design
The input data is a text file containing a YAML-templated description of the pipeline steps. Three entities will process this data in order to run the pipeline:
* Parser: run the template engine (e.g. globbing, root directory, number of slots for parallel jobs, etc.) and handle cross-step references
* Generator: convert the rendered description to build-engine specific file
* Build engine: run the build itself with user-specified options
Under this design, multiple build engines and their corresponding generators may be implemented; the build engine of choice is Ninja, due to its ability to handle multiple outputs per step and to its parallel build features.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
spire-pipeline-0.5.2.tar.gz
(7.8 kB
view hashes)
Built Distributions
Close
Hashes for spire_pipeline-0.5.2-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 7c087db4c6727d8290948511021929a2f6d0438cc2aaaf6616582fcf52bb231e |
|
MD5 | b44e5f2cb4d3f804876fc3f843643a05 |
|
BLAKE2b-256 | e525ca75ff6ef55411c43d2e453bf4af78d9d2d30d0b44a5ef09a57d31a957cc |
Close
Hashes for spire_pipeline-0.5.2-py2-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | eb9a8fac8ffe38667d4245bfa804e841425e8a72574605fbe6e1b321ad34f27f |
|
MD5 | 3bf4258aec9dc3d22baff3d4a53f0ed3 |
|
BLAKE2b-256 | 09fa8e016a8cc6d790213906e58b130cdd7a7b1315682c41a84b186700ebbd1f |