Run software pipelines using doit
Project description
Spire
Spire is a thin wrapper around doit. It eases the declaration of tasks through:
- Class-based task declarations
- Built-in factories for repetitive tasks
- Optional pruning of the task graph when some dependencies are missing
Moreover, tasks will be rerun whenever their actions are modified.
Task declaration
Spire tasks can be classes: this syntax facilitates the reusability of dependencies and targets in the list of actions.
import spire
class BuildHouse(spire.Task):
file_dep = ["brick", "mortar"]
targets = ["house", "dog_house"]
actions = [["build"]+file_dep+targets]
This task file can then be run with the usual doit
command:
$ doit run -f build_house.py -d /home/somebody/vacant_lot
. BuildHouse
For simple tasks (single target or single action), it is not mandatory to use lists. In this case, the singular form of the member name must be used (i.e. targets becomes target and actions becomes action).
import spire
class BuildShed(spire.Task):
file_dep = "wood"
target = "shed"
action = ["build", file_dep, target]
Spire tasks are cleanable by default: using the previous examples, calling doit clean -f ... -d ...
will remove the targets.
Repetitive tasks
For repetitive tasks, Spire provides the TaskFactory
class. Classes derived from TaskFactory
need to set the following members for each object:
- The task name, through the constructor of
TaskFactory
file_dep
,targets
andactions
import spire
class BuildHouse(spire.TaskFactory):
def __init__(self, material):
spire.TaskFactory.__init__(self, "Build{}House".format(material))
self.file_dep = [material]
self.targets = ["{}_house".format(material)]
self.actions = [["build", material]]
houses = [BuildHouse(material) for material in ["Straw", "Sticks", "Bricks"]]
Pruning the task graph
Tasks with missing dependencies may be skipped instead of being executed and failing. For this, missing dependencies must be specified as None
entries in file_dep
, and the function spire.prune()
must be called. The task graph will be pruned starting at the current task, ensuring that no error will occur on account of these missing targets.
In the following example, if either brick
or mortar
is missing, neither BuildHouse
nor BuildDogHouse
will be executed:
BuildHouse
will be skipped sincefile_dep
contains entries which areNone
andspire.prune()
was calledBuildDogHouse
will be skipped since one of its parent has been skipped.
On the other hand, if brick
and mortar
are present but doggie_basket
is missing, BuildHouse
will be successfully executed but BuildDogHouse
will fail as none of its file_dep
equal None
.
import os
import spire
class BuildHouse(spire.Task):
file_dep = [x if os.path.isfile(x) else None for x in ["brick", "mortar"]]
target = "house"
action = ["build"] + file_dep + [target]
class BuildDogHouse(spire.Task):
file_dep = [BuildHouse.target, "doggie_basket"]
target = "dog_house"
action = ["build"] + file_dep + [target]
spire.prune()
Graphical representation of the task graph
A graphical representation of the task graph, in the Graphviz format, can be generated by calling the spire
module:
$ python3 -m spire graph tasks.py tasks.dot
A simplified representation, omitting the targets and dependencies nodes, can be generated py passing the option --tasks-only
. Any other option will be passed directly to doit, e.g. to specify command-line variables.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for spire_pipeline-1.2.1-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 0dea1fa0002b144a2ce1ff0e843a00cecdd4ecad3ceb69ad9ce1501c2508e0c5 |
|
MD5 | 1e0b08fa9be1e587c6b37a6090b8f7ac |
|
BLAKE2b-256 | bd13ed1581fcda2e1424eaaed09c80bbb303d48cb9b51debc600e27a155eb802 |