revo 0.2.3

Referenced Values in Objects.

Revo – Referenced Values in Objects

Revo is a non-intrusive variable substitution solution for config files.

Revo is listed on PyPI.

Revo casts a Python built-in dict or list into a mutable tree, where the value of each tree node is either a string or a number, while any node can reference any other node in the tree with GNU make style $(var) variable syntax.

Although a tiny (under 200 lines of code) library, revo provides some level of programmability to plain old Python objects and thus great flexibility to many applications, especially configuration processing. No matter what format (JSON, YAML, TOML, etc.) your config file is, as long as it maps to Python built-in types cleanly, revo provides Makefile-style variable substitution to it.

Design ideas

Revo treats a Python built-in object as a tree of data, like an XML doc. The top-level object must be dict or list. Values on tree nodes must be either str, bool, or built-in number (int or float).

In order to reference any node in an object tree, we need to design a path mechanism. Yes, the idea is like XPath for Python objects, but much simpler.

In fact, we can almost simply borrow the design of UNIX path: slash-separated string, with only a small adaptation: path segments in integer literals are treated as numeric indexes for list.

An exmple would be nice, isn't it?

# This is legal Python code, making an object from literals.
# You can also construct it from a YAML or JSON file.
conf = {
  "project": {
    "name": "revo",
    "version": "0.1.0",
    "rules": {
      "Homepage": "https://github.com/sunyj/$(project/name)"
    }
  },
  "classifiers": [
    "Programming Language :: Python :: 3",
    "License :: OSI Approved :: MIT License",
    "Operating System :: OS Independent"
  ],
  "tag": "$(classifiers/0)", # reference a list element with integer index
  "v1": "project",
  "v2": "version",
  "v3": "$($(v1)/$(v2))" # support variables in variable names
}

# Resolve variable references with revo
from revo import Revo
obj = Revo(conf).resolve()

print(obj['project/rules/Homepage'])  # https://github.com/sunyj/revo
print(obj['tag'])                     # Programming Language :: Python :: 3
print(obj['v3'])                      # 0.1.0

Typical use cases

Almost everyone who works long enough with config files appreciates the great value of a robust and flexible variable substitution mechanism. It helps to reduce tedious config-processing logic by moving them to config files.

Revo was designed to be a non-intrusive solution for that purpose. It resolves on Python objects, not the files storing those objects, so it's transparent to the config file format.

Design details

The core mechanism is the path to reach any node in the object tree. With a unique path for every node, an object can be melted down to a set of flat dict of key-value mappings. The definition of class Revo reflects that.

class Revo(MutableMapping):
    ... ...

As the path of every node is unique within an object, it may serve as the name for the corresponding node's value. Voilà! variable, as in the context of "variable substitution", can now be properly defined.

Concept	Implementation
variable	tree node
variable name	node path
variable value	node value

Variables Make Makefiles Simpler, equally true for config files. We simply use the variable reference syntax of GNU make.

Variable overrides

Another common need for config files is overriding values from command line or other input sources. Revo supports that with its override method:

from revo import Revo
override_specs = ['date=20200110', 'conf/path=/another/path/with=/in/it']

# construct with overrides
conf_obj = load_my_json_config(...)
conf = Revo(conf_obj, override_specs)

# or in a separate call
conf = Revo(load_my_yaml_config(...))
conf.override(override_specs).resolve()

Override spec parsing rules:

• First = is used as the name-value separator. Subsequent = characters are all put into the value string.
• Revo always tries parsing the value string as a Python literal, and falls back to string if that fails.

Fault tolerance

Keyword-only boolean argument mercy controls if revo raises exceptions on errors during the resolution process. Typical errors are:

• Unknown variable.
• Illegal variable syntax.
• Self-reference or circular reference.

It defaults to False. When mercy=True, unresolved or partially-resolved values are left in the object.

Definition merging

Top-level overrides may contain variables that are not in the object under resolve, for example:

import revo
conf = revo.Revo({'name': 'hello $(date)'}, ['date=20220101'])
conf['name'] # 'hello 20220101'
conf['date'] # what do you expect?

Such top-level overrides are called definitions. Definitions may stay in the object after the resolution. Keyword-only boolean argument absorb controls this, it defaults to False.

Definition extending

Overrides may also contain variables with new leaf-node values, for example:

import revo
obj = {'data': {'name': 'foo'}}
conf = revo.Revo(obj, overrides=['data/func=bar'])

# good, as 'func' is a leaf node
obj['data']['func'] # 'bar'

# not good if extend is turned off
conf = revo.Revo(obj, overrides=['data/func=bar'], extend=False)

# error, as new is NOT a leaf node
conf = revo.Revo(obj, overrides=['new/func=bar'])

Keyword-only boolean argument extend controls this, it defaults to True.

Type retaining

Keyword-only boolean argument retain controls if revo tries to keep variable value type in substitutions.

It defaults to True. When retain=False, values resolved are always str.

import revo
conf = revo.Revo({'name': 'n$(val)', 'x': '$(val)', 'val': 10}, retain=True)
conf.resolve()

type(conf['name'])  # <class 'str'>
type(conf['x'])     # <class 'int'>

Limitations

Variable substitutions are resolved bottom-up, which means revo copies and iterates over all values in loops until no more incremental substitution happens. This algorithm is easy to understand and implement, but it's pretty slow. For average config files with typically dozens or hundreds of entries, performance should not be a concern on modern computers. However, if you plan to use it on very large config files, make performance tests before you invest further.