replacement 0.4.4

A tool to execute yaml templates and output text

Replacement

Replacement is a python utility that parses a yaml template and outputs text.

Installing

pip3 install replacement

or, if you use nix:

nix-env --install -A 'nixpkgs.python3Packages.replacement'

Introduction

NOTE: the examples given here are further documented in the tests.

A template is a YAML file containing a replacement object.

A replacement object contains a list of blocks.

1. basic template

A template:

---
# simplistic template example
# tests/hello.yaml
replacement:
  - text: text
    input:
      hello world
...

Execute a template using replacement:

$ replacement -t tests/hello.yaml
hello world

Blocks always begin with a directive, which specifies:

• the data type the block will yield, such as text or a dict
• the input data type, such as file or yaml

In the block above, text: text is the directive; it specifies the block takes text input and yields text output.

2. reading from a file

File hello.out:

hello world

---
# read from file
# tests/file.yaml
replacement:
  - text: file
    input:
      hello.out
...

$ replacement -t tests/file.yaml
hello world

Notice that:

• The directive in the above block is text: file; this means: "yield text, input from a file".
• The input key is used for the filename to be read; in the first example input was used to store the literal text input.
• File paths are relative to the path of the template.

intermission: schema

A schema of all the replacement directives is contained in schema.yaml.

Here is a snippet showing just the directive and input portions of a block:

---
##
# format of a block
##
block:
  - yield: inputType  # this is a directive
    input: inputVal

##
# definition of schema elements
##
schema:
  # 'yield' : what to output (how to format output)
  yield:
    - text  # text: a list of newline-separated strings
    - dict  # a dictionary of key-value pairs
    - meta  # output nothing: set a metadata (substitution dictionary) variable

  # 'input' : where to source data
  inputType:
    # we can input everything we output (see 'yield' above)
    - text
    - dict
    - meta  # a key-value pair retrieved from substitutions dictionary
    # we can also input from other sources
    - file  # open a file on disk
    - eval  # python3 eval() statement
    - func  # import and then call call a function
    - exec  # subprocess execution (usually bash)

  inputVal: input_specific
...

3. metadata and substitution

The meta directive specifies that the output of a block should be inserted into the "substitutions dictionary" (aka: meta).

• this implies input must be a valid dictionary
• replacement will parse YAML or JSON (which is a subset of YAML)

The proc keyword specifies that block output should be processed (string substitution). Valid proc directives are:

• format :: str.format()
• substitute :: string.Template().substitute()
• safe_substitute :: string.Template().safe_substitute()

---
# metadata
# tests/metadata.yaml
replacement:
  # parse 'input' and insert the resulting dictionary into 'meta'
  - meta: dict
    # metadata may be given as an object
    input:
      version: 1.1
      tag: my_awesome_tag
  - meta: text
    # metadata may be given as text which can be parsed as valid YAML
    input: |
      ---
      message: hello world
      ...
  - meta: text
    # metadata may also be given as JSON
    input: |
      { "hi": 5 }
  # use 'proc' to specify that 'str.format(**meta)' should be run on output
  - text: text
    proc: format
    input: |
      v{version} tag "{tag}"
  - text: text
    proc: substitute
    input: |
      message $message
  - text: text
    proc: safe_substitute
    input: |
      hi $hi
      this value may not exist - $nonexistent
...

$ replacement -t tests/metadata.yaml
v1.1 tag "my_awesome_tag"
message hello world
hi 5
this value may not exist - $nonexistent

Metadata can also be read from a file.

File a.json:

{
"hello": "world",
"hi": 5,
"list": [ 1, 2, 3, 4 ]
}

---
# metadata
# tests/meta_file.yaml
replacement:
  # parse file, inserting result into metadata dictionary
  - meta: file
    input:
      a.json
      # string substitution with 'proc'
  - text: text
    proc: format
    input: |
      hello {hello}
      hi {hi}
      list {list}
...

$ replacement -t tests/meta_file.yaml
hello world
hi 5
list [1, 2, 3, 4]

3a. value -> dictionary using key

---
# insert contents of file as metadata
# tests/file_as_meta.yaml
replacement:
  - text: dict
    input:
      # produces {"hello": "hi"}
      - dict: text
        key: hello
        input: |
          hi
      - dict: file
        key: contents
        input: hello.out

      - meta: file
        key: data
        input: prep.out
      - dict: dict
        prep: substitute
        input:
          also: $data

      - meta: file
        key: largefile
        input: recurse.out

      - dict: dict
        prep: format
        input:
          hello: world
          hi: |
            {largefile}
...

---
hello: hi
contents: hello world
also: hello world
hi: |-
  recursed inline
  v1.1 tag "my_awesome_tag"
  message hello world
  hi 5
  this value may not exist - I exist I promise!
...

4. nesting

1. Blocks are executed in sequence, and may nest.
   • nested blocks may extend/alter meta, which will be seen by later blocks or child blocks, but not parent blocks.

---
# nesting and dictionaries
# tests/nesting.yaml
replacement:
  # parse 'input' and insert the resulting dictionary into 'meta'
  - meta: dict
    input:
      version: 1.1
  # same thing, but parse *text* input, instead of a dictionary
  - meta: text
    input: |  # note the '|'
      ---
      tag: my_awesome_tag
      ...
  # use 'proc' to specify that 'str.format(**meta)' should be run on output
  - text: text
    proc: format
    input: |
      v{version} tag "{tag}"
  - text: text
    input:
      # metadata additions/changes seen by later blocks and any children,
      # but seen outside this list
      - meta: dict
        input:
          version: 1.0
      - text: text
        proc: format
        input: |
          #v{version} tag "{tag}" (version clobbered in inner scope)
      - text: file
        input:
          hello.out  # contains 'hello world'
  - text: text
    proc: format
    input: |
      outer still v{version}
...

$ replacement -t tests/nesting.yaml
v1.1 tag "my_awesome_tag"
#v1.0 tag "my_awesome_tag" (version clobbered in inner scope)
hello world
outer still v1.1

5. preprocessing

Substitution can also be performed on the values of the block itself before it is parsed. The keyword prep is used, with the same semantics as proc (above):

---
# preprocessing
# tests/prep.yaml
replacement:
  - meta: dict
    input:
      filename: hello.out
  # preprocessing will substitute {filename} before evaluating 'file' input
  - text: file
    prep: format
    input: |
      {filename}
...

$ replacement -t tests/prep.yaml
hello world

6. access to python eval

---
# use of eval
# tests/an_eval.yaml
replacement:
  # 'eval' returning a dictionary that can me appended to 'meta'
  - meta: eval
    input: |
      {"hello": 5 + 1}
  - text: text
    prep: format
    input: |
      hello {hello}
  # eval returning a scalar value
  - text: eval
    prep: format
    input: |
      {hello}**3
...

$ replacement -t tests/an_eval.yaml
hello 6
216

7. python exec

Sometimes it is advantageous to manipulate the runtime environment.

In the below example, this is being used to import a module which will be needed by a subsequency call to eval.

---
# use of eval with an exec statement to execute an import call
# tests/eval_exec.yaml
replacement:
  - meta: exec
    input: |
      global IPv4Network
      from ipaddress import IPv4Network
  - meta: eval
    input: |
      {'gateway': str([h for h in IPv4Network('192.168.1.0/24').hosts()][-1])}
  - text: text
    prep: format
    input: |
      my gateway is {gateway}
...

$ replacement -t tests/eval_exec.yaml
my gateway is 192.168.1.254

NOTE: python exec is very powerful, please avoid running replacement as a privileged user.

8. imports and function execution

The func input directive can be used to find and call an external function (see demo.py for reference):

---
# function execution
# tests/func.yaml
replacement:
  # run a function returning a dictionary, and merge that into 'meta'
  - meta: func
    args:
      existing: {'original': 'thesis'}
    # NOTE that this is sensitive to PYTHONPATH
    input: |
      ret_a_dict ./demo.py
  - text: text
    prep: format
    input: |
      original {original}
      secret {secret}

  # run a function returning a dictionary and export return as JSON
  - text: func
    options:
      - json  # emit JSON rather than YAML (the default)
    args:
      existing: {'original': 'thesis'}
    input: |
      ret_a_dict ./demo.py

  # run a function returning an IOStream
  - text: func
    args: {}
    input: |
      ret_a_stream ./demo.py
    # previous use of non-portable BRAINDEAD python dot.notation
    # ... will break if 'replacement' (the script) is MOVED vis-a-vis demo.py
    # (of OF COURSE the CWD is blithely ignored ... why would ANYONE use it LOL)
    # TLDR: use the '[symbol] [path]' notation just above.
    # The _original_ call (broken) was:
    # tests.demo.ret_a_stream

  # A function returning a list of lines of text
  # (using the proper, cleaner import strategy)
  - text: func
    args:
      an_arg: ["question"]
    input: |
      ret_a_list ./demo.py

  # a static function inside a class
  - text: func
    input: |
      aClass.invented_list ./demo.py

$ replacement -t tests/func.yaml
original thesis
secret 42
{"secret": 42, "original": "thesis"}
1. hello
2. world
42
meaning
question
hello
from
staticmethod

9. Recursion

Or, how I learned to stop worrying and have templates import other templates.

---
# recursion (nested 'replacement' templates)
# tests/recurse.yaml
replacement:
  # parse a 'replacement' template inline
  - replacement: text
    input: |
      ---
      replacement:
        - text: text
          input: |
            recursed inline
      ...
  - meta: dict
    input:
      nonexistent: "I exist I promise!"
  # parse a replacement template from a file
  # NOTE *relative* path)
  # NOTE our 'meta' dictionary is propagated to child replacement being parsed
  - replacement: file
    input: metadata.yaml
...

$ replacement -t tests/recurse.yaml
recursed inline
v1.1 tag "my_awesome_tag"
message hello world
hi 5
this value may not exist - I exist I promise!

Development

The recommended development environment is:

1. Python venv with nix:

    nix-shell --pure  # activates a venv
   

   alternately:

    python -m venv venv
    venv/bin/activate
    alias pip="python -m pip"
   

2. Install requirements:

    pip install -r requirements.txt
    pip install -e .
   

3. Test and build:

    pytest
    ./create_package.sh
   

NOTES

1. This documentation is itself built as a template, so that the content of .yaml tests and .out results is kept in sync with the tests directory. See README.template.yaml and gen_readme.sh.

Project TODO

Project TODO list

1. subprocess execution and output capture

2. accept template from STDIN, not just a template file

3. Packaging:

   • proper test runner (perhaps tox?)
   • add test runner to all the builds

4. Dependency output command: runs all preprocessing and outputs a list of file dependencies. For use e.g. in Makefiles.

5. Express the schema formally and write validation code for it.