stepmaker 0.1.0

Step Maker Step-parsing Framework

The ansible system automation tool uses, as its primary primitive, a list of steps to execute, expressed in YAML list syntax. Each step is described as a dictionary, with one key indicating the actual action to take, along with some additional keys that describe metadata about the step (such as a description) or modifiers for the step (such as conditional expressions). This package provides a framework for building applications that use similar step descriptions.

Steps

As mentioned above, steps consist of metadata, modifiers, and an action, all expressed through keys on the step dictionary. The stepmaker package provides the abstract superclasses Step, Modifier, and Action that can be extended to provide application-specific step structure.

The Step class is the main class for stepmaker. Implementors must subclass Step and provide an implementation for the validate() method, as well as setting the namespace_actions and namespace_modifiers class variables. The metadata_keys class variable can be used to identify particular keys as metadata. The Step class provides a parse_list() class method for parsing a list of dictionaries as step descriptions, using actions and modifiers discovered in the entrypoint groups declared using namespace_actions and namespace_modifiers. Invoking the step is as simple as calling the Step object with an application-specific context.

The Action class is an abstract superclass for step actions. Implementors must subclass Action and implement its validate() and __call__() methods. The Action subclass performs the actual work of the step. Note that actions are classed as either “eager” or “lazy”, controlled by the eager class variable, with the default being lazy. Eager actions can be used to allow for including other files or other libraries of step actions during parsing by Step.parse_list().

The Modifier class is an abstract superclass for step modifiers. A step modifier is able to modify how the action is performed; everything from temporary mutations of the execution context to skipping the step, or even executing the action multiple times (the Step.evaluate() method can facilitate this). Implementors must implement its validate() method, and then may implement the pre_call() and/or post_call() hook methods to perform the necessary work. Implementors may also set the restriction class variable to restrict which actions a modifier can be used with; the before and after class variables provide control over the order with which modifiers are applied; and the required and prohibited class variables can control which other modifiers are required or prohibited on a given step.

For full details on defining steps, see the documentation on the Step, Action, and Modifier classes.

Utilities

A number of utilities are also made available to assist with the creation of a step-driven application. For instance, the validate() methods of the Step, Action, and Modifier classes could be implemented using the jsonschema package; the jsonschema_validator() context manager can be used with jsonschema.validate() to translate schema validation errors into more helpful StepError exceptions, which include the “address” of a step configuration error. The Environment class is a special dictionary-like object containing system environment variables, but also includes methods for registering “special” translators for environment variables (e.g., the “PATH” environment variable could be translated into a Python list-like object using the SpecialList translator), opening files relative to a working directory associated with the Environment object, and even executing shell commands. Finally, the RedactedObject and RedactedDict classes proxy to other objects, but are additionally capable of masking certain attributes or dictionary keys; this could be used on output routines to ensure that sensitive data such as passwords is not exposed to the console.

Modifiers

Modifiers can inhibit the further processing of a step by raising the AbortStep exception from their pre_call() hook method. Modifiers can also specify a result to be returned to the step’s caller by passing that result to AbortStep. If no result is passed, the result will be the special singleton skipped.

Note that post_call() processing of the modifier still occurs; raising AbortStep prevents the processing of modifiers after the one that raised it, but the post_call() method of the raising modifier, along with the ones called before, are still called with the result proposed in the AbortStep.

Addresses and the Validator Methods

The StepAddress class is used to express the location of a configuration item, and is used during parsing by Step.parse() and Step.parse_list() to raise helpful errors that indicate the location of a configuration problem. These addresses are also passed on to actions and modifiers, and can be used by the validate() methods to raise appropriate StepError exceptions. Additionally, if using the jsonschema package for validation, the jsonschema_validator() context manager can be used to translate schema validation errors raised by the package into StepError exceptions that include the address. It can be used like so:

with jsonschema_validator(addr):
    jsonschema.validate(config, schema)

(Note that jsonschema is not a dependency of stepmaker. The jsonschema_validator() function uses duck-typing to avoid needing to install jsonschema alongside stepmaker.)

Redacted Objects and Dictionaries

Some data may be sensitive: an application developer may wish to inhibit the display of that data to the console. This data may be a set of variables associated with the execution context, or it may even be environment variables that may contain such things as passwords. To ensure that such information cannot be accidentally displayed or used, an implementor may choose to proxy an object or dictionary using the RedactedObject and RedactedDict classes. These classes proxy attribute and, in the case of RedactedDict, item accesses back to an underlying object, but can return instances of Redacted for certain attributes or items. By default, these classes return a singleton redacted instance of Redacted, which has a default string representation of “<redacted>”.

The attributes and items to redact are controlled by sets of attribute names or item keys. This implements a black-list policy, where only certain attributes or items are redacted; to implement a white-list policy, where all attributes or items are redacted except for specified exceptions, wrap a set in the Inverter class; this will invert the sense of membership tests.

It should be noted that the sets of attributes and items passed to RedactedObject and RedactedDict (and Inverter) are saved directly, and can be updated by processes outside of the classes.

Environment

Step-driven applications often need at least one step capable of executing shell commands on the system, and also often need to be able to manipulate environment variables and open files. The stepmaker package provides an Environment class which provides all of this functionality in a single object. The class is a dictionary containing the environment variables for execution of system commands (note that this is distinct from the current contents of os.environ, though the Environment class constructor uses the current contents of os.environ as the default environment); the class also keeps track of a current working directory (which is also distinct from the process’s current working directory). Finally, special interpreters can be associated with environment variables, enabling, for instance, list-like access to the “PATH” environment variable; a full collection of special interpreters is included, and described below.

There are two ways to invoke a shell command using an Environment instance. The first is to call popen() with a string or list describing the command and its options, and a set of keyword arguments suitable for passing to subprocess.Popen. This will return a subprocess.Popen instance, which may then be manipulated using the methods provided by that class. The second way to invoke a shell command is to call the Environment; the __call__() method is similar to the subprocess.run() function provided in Python 3 versions of subprocess, and will return a stepmaker.CompletedProcess object with the command’s return code, along with captured standard output and standard error (to capture these streams, pass subprocess.PIPE or stepmaker.PIPE to the stdout and/or stderr keyword arguments to __call__()). Additionally, if the input keyword argument is provided, it will be sent to the command’s standard input; and if the check keyword argument is set to True, a stepmaker.ProcessError exception will be raised if the command’s return code is non-zero. This will, of course, wait for process execution to complete before continuing.

In addition to stepmaker.PIPE, the stepmaker package also copies subprocess.STDOUT for convenience. This allows the use of the Environment command execution facilities without having to separately import subprocess.

The Environment class tracks a working directory, which can be changed by setting the cwd property. Commands are, by default, executed with there working directory set to the value of cwd. It is also possible to locate a file relative to the cwd, using the filename() method; and the file may even be opened (using the open() built-in) with the open() method.

Specials

Specials are environment variable interpreters attached to an Environment instance. They can be registered at construction time, by passing keyword arguments of the form VARIABLE=factory (e.g., PATH=SpecialList) to the constructor, or they can be registered after the fact by calling the register() method of the Environment. (Specials may also be unregistered by calling register() without a factory function.) Several specials are provided, such as the SpecialList for list-like environment variables, such as “PATH”; SpecialSet, for set-like environment variables (distinguished from list-like environment variables in that ordering is not important); SpecialDict, for dictionary-like environment variables containing “key=value” pairs; or SpecialOrderedDict, which is distinguished from SpecialDict by the fact that it maintains the original key order. The Special abstract base class can be used for constructing other specials.

It should be noted that the SpecialList, SpecialSet, SpecialDict, and SpecialOrderedDict classes all contain a with_sep() class method that can be used to construct a factory function using alternate separators. If the default separators are not suitable for a given application, then, instead of passing the class as the factory function, pass the result of calling the class’s with_sep() class method with appropriate arguments.

It should also be noted that Environment never deletes an instance of a special unless a new special factory is registered (or the special is deregistered). This means that the value can be kept outside of the environment. In particular, it is possible to use a SpecialSet with a RedactedDict class wrapping the Environment, so that environment variables to be redacted can be listed in a particular environment variable.