Skip to main content

Run software pipelines using doit

Project description

Spire

Spire is a thin wrapper around doit. It eases the declaration of tasks through:

  • Class-based task declarations
  • Built-in factories for repetitive tasks
  • Optional pruning of the task graph when some dependencies are missing

Moreover, tasks will be rerun whenever their actions are modified.

Task declaration

Spire tasks can be classes: this syntax facilitates the reusability of dependencies and targets in the list of actions.

import spire

class BuildHouse(spire.Task):
    file_dep = ["brick", "mortar"]
    targets = ["house", "dog_house"]
    actions = [["build"]+file_dep+targets]

This task file can then be run with the usual doit command:

$ doit run -f build_house.py -d /home/somebody/vacant_lot
. BuildHouse

For simple tasks (single target or single action), it is not mandatory to use lists. In this case, the singular form of the member name must be used (i.e. targets becomes target and actions becomes action).

import spire

class BuildShed(spire.Task):
    file_dep = "wood"
    target = "shed"
    action = ["build", file_dep, target]

Spire tasks are cleanable by default: using the previous examples, calling doit clean -f ... -d ... will remove the targets.

Repetitive tasks

For repetitive tasks, Spire provides the TaskFactory class. Classes derived from TaskFactory need to set the following members for each object:

  • The task name, through the constructor of TaskFactory
  • file_dep, targets and actions
import spire

class BuildHouse(spire.TaskFactory):
    def __init__(self, material):
        spire.TaskFactory.__init__(self, "Build{}House".format(material))
        self.file_dep = [material]
        self.targets = ["{}_house".format(material)]
        self.actions = [["build", material]]

houses = [BuildHouse(material) for material in ["Straw", "Sticks", "Bricks"]]

Pruning the task graph

Tasks with missing dependencies may be skipped instead of being executed and failing. For this, missing dependencies must be specified as None entries in file_dep, and the function spire.prune() must be called. The task graph will be pruned starting at the current task, ensuring that no error will occur on account of these missing targets.

In the following example, if either brick or mortar is missing, neither BuildHouse nor BuildDogHouse will be executed:

  • BuildHouse will be skipped since file_dep contains entries which are None and spire.prune() was called
  • BuildDogHouse will be skipped since one of its parent has been skipped.

On the other hand, if brick and mortar are present but doggie_basket is missing, BuildHouse will be successfully executed but BuildDogHouse will fail as none of its file_dep equal None.

import os
import spire

class BuildHouse(spire.Task):
    file_dep = [x if os.path.isfile(x) else None for x in ["brick", "mortar"]]
    target = "house"
    action = ["build"] + file_dep  + [target]

class BuildDogHouse(spire.Task):
    file_dep = [BuildHouse.target, "doggie_basket"]
    target = "dog_house"
    action = ["build"] + file_dep  + [target]

spire.prune()

Graphical representation of the task graph

A graphical representation of the task graph, in the Graphviz format, can be generated by calling the spire module:

$ python3 -m spire graph tasks.py tasks.dot

A simplified representation, omitting the targets and dependencies nodes, can be generated py passing the option --tasks-only. Any other option will be passed directly to doit, e.g. to specify command-line variables.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

spire-pipeline-1.2.1.tar.gz (16.2 kB view details)

Uploaded Source

Built Distribution

spire_pipeline-1.2.1-py3-none-any.whl (18.8 kB view details)

Uploaded Python 3

File details

Details for the file spire-pipeline-1.2.1.tar.gz.

File metadata

  • Download URL: spire-pipeline-1.2.1.tar.gz
  • Upload date:
  • Size: 16.2 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.4.1 importlib_metadata/3.7.3 pkginfo/1.7.0 requests/2.25.1 requests-toolbelt/0.9.1 tqdm/4.59.0 CPython/3.9.2

File hashes

Hashes for spire-pipeline-1.2.1.tar.gz
Algorithm Hash digest
SHA256 37b81969f434f5a6ffa3123e5d95317f94b8cda91df08ab91b78c2ae37d761f3
MD5 1533f6a7b08360233faf7c7c35d62b88
BLAKE2b-256 1a087306ff889d483c34f2bc0264516e68bee03024a0794774eff795af59a74d

See more details on using hashes here.

File details

Details for the file spire_pipeline-1.2.1-py3-none-any.whl.

File metadata

  • Download URL: spire_pipeline-1.2.1-py3-none-any.whl
  • Upload date:
  • Size: 18.8 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.4.1 importlib_metadata/3.7.3 pkginfo/1.7.0 requests/2.25.1 requests-toolbelt/0.9.1 tqdm/4.59.0 CPython/3.9.2

File hashes

Hashes for spire_pipeline-1.2.1-py3-none-any.whl
Algorithm Hash digest
SHA256 0dea1fa0002b144a2ce1ff0e843a00cecdd4ecad3ceb69ad9ce1501c2508e0c5
MD5 1e0b08fa9be1e587c6b37a6090b8f7ac
BLAKE2b-256 bd13ed1581fcda2e1424eaaed09c80bbb303d48cb9b51debc600e27a155eb802

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page