Manage context-less text with [[SIGILS]].
Project description
Inscribed or painted symbols considered to have magical power.
A sigil is a universal token embedded in text, used as placeholder for out-of-context values. When resolved, the actual value is determined from a configurable thread-local context and interpolated into the text with proper formatting, extracted as a mapping, sanitized, etc.
This is different from conventional string interpolation, in which you have to provide the map of values in advance. Instead, the same sigil can be embedded in different contexts (SQL queries, HTML or email templates, configuration files, etc.) and be resolved only when needed.
A sigil can also hold metadata in it’s filter structure, such as the type of value, the name of the field and model, or remote resource where the data should be fetched from, what functions should be applied to the found value, etc.
Sigils includes all you need to manipulate text containing sigils. It works with Django OOTB, but it can be used in any Python project. Sigils is thread-safe and fast. It is also very easy to extend.
Sigils can be anywhere.
Installation
Install and update from PyPI:
pip install -U sigils
Structure of Sigils
A typical sigil has one of the following forms in text:
[[USERNAME]]
[[SETTING.BASE_DIR]]
[[MODEL='natural-key'.FIELD]]
[[MODEL="natural-key".FIELD]]
[[ENV=TIER.HOST]]
[[MODEL.USER=[USERNAME].STATUS]]
Each sigil is a linked list of nodes separated by a dot. Nodes can be natural or parametrized. Natural means, use the natural context to figure out the value. If parametrized, they can take an argument using the equals sign.
The argument can be a number, a quoted string, or another sigil. When the argument is a sigil, only one brace pair is needed.
[[NODE1='ARG1'.NODE2=[ARG2].NODE3=10.ETC]]
Nesting can be done to any depth. If quotes are avoided around the inner sigils, data types will be preserved until the final interpolation. If quotes are used, the result is always a string.
Sigils should always be uppercase. This is not enforced, but may cause problems if you use lowercase sigils in your templates. Args are case-sensitive.
Whitespace
Whitespace inside the sigil is ignored, so you can use it to make the sigils more readable. The following are all equivalent:
[[MODEL='natural-key' .FIELD]]
[[MODEL = 'natural-key' . FIELD]]
[[ MODEL="natural-key" .FIELD.SUBFIELD ]]
Whitespace inside quoted strings is preserved, so you can use it to separate words in the argument:
[[MODEL='natural key'.FIELD]]
[[MODEL='natural-key'.FIELD]]
[[MODEL='natural_key'.FIELD]]
If you want to use quotes inside a quoted string, you can escape them or mix the quotes.
[[MODEL='natural "key"'.FIELD]]
[[MODEL='natural \'key\''.FIELD]]
Piping Values and Functions
The found value can be piped into a function using the . operator. This also allows piping the result of one sigil into another. For example:
[[MODEL='natural-key'.FIELD.UPPER]]
[[USER.NAME.UPPER.TRIM]]
[[USER='arthexis@gmail.com'.DOMAIN.SLUG]]
[[USER=[USERNAME.TRIM].DOMAIN.SLUG]]
The function after . can be a built-in function, one found in the current context, or a method of the value (for example, if it was a model instance before being serialized for interpolation). If no function is found, the sigil is not resolved, even if a value was found. The function must accept one or two arguments. The first argument is the value to be piped into the function’s first parameter. The second argument is the argument of the current node. For example:
[[MODEL='natural-key'.NAMES.FORMAT='Hello, {0}!']]
[[SESSION.USER.ACCOUNT_ID.ZFILL=8]]
[[DOM='#someid'.ELEM.STYLE='color']]
If the function fails, the sigil is not resolved. By default, no exceptions are raised and not extraneous information is logged.
Currently, the following built-in functions are available everywhere:
SYS: access system variables
NUM: convert to number
UPPER: convert to uppercase
LOWER: convert to lowercase
TRIM: remove leading and trailing whitespace
STRIP: idem
SLUG: convert to slug
ZFILL: zero-fill the string to the given length
F: format the found value using the given format string
STYLE: convert a CSS style string to a dict
JOIN: join the list with the given separator
SPLIT: split the string with the given separator
OR: return the found value, or the argument if None
AND: return the found value, or None if the argument is None
NOT: negate the found value
BOOL: convert to boolean
INT: convert to integer
FLOAT: convert to float
LIST: convert to list
DICT: convert to dict
TUPLE: convert to tuple
SET: convert to set
JSON: convert to JSON
B64: convert to base64
B64D: convert from base64
URL: convert to URL (percent-encoding)
URLD: convert from URL (percent-decoding)
LEN: return the length of the found value
REV: reverse the found value
SORT: sort the found value
ITEM: return an item of the found value explicitly by index or key
KEY: idem
ATTR: return an attribute of the found value explicitly by name
ANY: return True if any item in the found value is True
ALL: return True if all items in the found value are True
NONE: return True if all items in the found value are False
SUM: return the sum of the found value
MIN: return the minimum of the found value
MAX: return the maximum of the found value
AVG: return the average of the found value
ABS: return the absolute value of the found value
ROUND: return the rounded value of the found value
CEIL: return the ceiling value of the found value
FLOOR: return the floor value of the found value
TRUNC: return the truncated value of the found value
MOD: return the modulo of the found value
FDIV: return the floor division of the found value
DIV: return the division of the found value
ADD: return the sum of the found value and the argument
SUB: return the difference of the found value and the argument
MUL: return the product of the found value and the argument
DIV: return the quotient of the found value and the argument
EQ: return True if the found value is equal to the argument
NE: return True if the found value is not equal to the argument
LT: return True if the found value is less than the argument
LE: return True if the found value is less than or equal to the argument
GT: return True if the found value is greater than the argument
GE: return True if the found value is greater than or equal to the argument
IN: return True if the found value is in the argument
CONTAINS: idem but backwards
FIRST: return the first item of the found value
LAST: return the last item of the found value
HEAD: return the first N items of the found value
TAIL: return the last N items of the found value
TYPE: return the type of the found value
FLAT: flatten the found value
UNIQ: return the unique items of the found value
ZIP: zip the found value with the argument
SIG: treat the found value as a sigil (recursive interpolation)
WORD: return the Nth word of the found value
The SYS root function can be used to access system variables and special functions. The sub-functions available may change depending on the context, the current environment, user privileges and the installed packages.
Currently these are available in all contexts:
ENV: access environment variables
ARGS: access all command-line arguments as a list
OPTS: idem
NOW: return the current datetime
TODAY: return the current date
TIME: return the current time
UUID: return a new UUID
RNG: return a random number
PI: return the value of pi
PID: return the current process ID
PYTHON: return the path to the python executable
PY_VER: return the version of the python interpreter
SIG_VER: return the version of the sigils package
OS: return the operating system name
ARCH: return the operating system architecture
HOST: return the hostname
IP: return the IP address
USER: return the username
HOME: return the home directory
PWD: return the current working directory
CWD: as above
TMP: return the path to the temporary directory
Special and Reserved Characters
The following characters are reserved and cannot be used inside sigils, except as specified in this document:
[[ and ]]: delimiters
.: node separator or function call
' and ": string delimiters
=: argument or natural key separator
\: escape character
( and ): reserved for future use
Quotes can be used interchangeably, but they must be balanced.
Available Tools
Sigils comes with a number of tools that can be used to manipulate and interpolate sigils. You can load them individually, or all at once:
from sigils import *
# or
from sigils import context, spool, splice, execute, vanish
Context
The context function is a context manager that can be used to manage the context for the other functions. It can be used to set the context for a single function call, or for a block of code.
from sigils import context, execute
with context(
USERNAME="arthexis",
SETTING={"BASE_DIR": "/home/arth/webapp"},
):
result = execute("print('[[USERNAME]]')")
assert result == "arthexis"
You can also pass a filename to the context function, which will try to guess the format and load the context from the file. The file can be in JSON, YAML, TOML or INI format.
from sigils import context, execute
with context("context.json"):
result = execute("print('[[USERNAME]]')")
assert result == "arthexis"
Spool
The spool function iterates over all sigils in a string, yielding each one in the same order they appear in the string, without resolving them.
from sigils import spool
sql = "select * from users where username = [[USER]]"
assert list(spool(sql)) == ["[[USER]]"]
Spool is a fast way to check if a string contains sigils without hitting the ORM or the network. For example:
from sigils import spool
if sigils := set(spool(text)):
# do something with sigils
else:
# do something else
Splice & Resolve
The splice function will replace any sigils found in the string with the actual values from the context. Returns the interpolated string.
from sigils import splice, context
with context(
USERNAME="arthexis",
SETTING={"BASE_DIR": "/home/arth/webapp"},
):
result = splice("[[USERNAME]]: [[SETTINGS.BASE_DIR]].")
assert result == "arthexis: /home/arth/webapp"
All keys in the context mapping should be strings (behavior is undefined if not) and will be automatically converted to uppercase.
Values can be anything: a string, a number, a list, a dict, or an ORM instance, anything that can self-serialize to text with the str function works.
class Model:
owner = "arthexis"
with context(
MODEL: Model, # [[MODEL.OWNER]]
UPPER: lambda x: x.upper(), # [[UPPER='text']]
):
assert splice("[[MODEL.OWNER.UPPER]]") == "ARTHEXIS"
If the value cannot or should not self-serialize, you can pass a function to use as the serializer. The function will be called with the value as-is.
class Model:
owner = "arthexis"
def serializer(model):
return f"owner: {model.owner}"
with context(
MODEL: Model, # [[MODEL.OWNER]]
):
assert splice("[[MODEL]]", seralizer=serializer) == "owner: arthexis"
You can pass additional context to splice directly:
assert splice("[[NAME.UPPER]]", **{"NAME": "arth"}) == "ARTH"
By default, the splice function will recurse into the found values, interpolating any sigils found in them. This can be disabled by setting the recursion parameter to 0. Default recursion is 6.
from sigils import splice, context
with context(
USERNAME="arthexis",
DIR="/home/[[USERNAME]]",
SETTING={"BASE_DIR": "[[DIR]]/webapp"},
):
result = splice("[[USERNAME]]: [[SETTINGS.BASE_DIR]]", recursion=1)
assert result == "arthexis: /home/[[USERNAME]]"
The function resolve is an alias for splice that never recurses.
Vanish & Unvanish
The vanish function doesn’t resolve sigils, instead it replaces them with another pattern of text and extracts all the sigils that were replaced to a map. This can be used for debugging, logging, async processing, or to sanitize user input that might contain sigils.
from sigils import vanish
text, sigils = vanish("select * from users where username = [[USER]]", "?")
assert text == "select * from users where username = ?"
assert sigils == ["[[USER]]"]
The unvanish function does the opposite of vanish, replacing the vanishing pattern with the sigils.
from sigils import vanish, unvanish
text, sigils = vanish("select * from users where username = [[USER]]", "?")
assert text == "select * from users where username = ?"
assert sigils == ["[[USER]]"]
assert unvanish(text, sigils) == "select * from users where username = [[USER]]"
Execute
The execute function is similar to resolve, but executes the found text as a python block (not an expression). This is useful for interpolating code:
from sigils import execute, context
with context(
USERNAME="arthexis",
SETTING={"BASE_DIR": "/home/arth/webapp"},
):
result = execute("print('[[USERNAME]]')")
assert result == "arthexis"
result = execute("print([[SETTING.BASE_DIR]])")
assert result == "/home/arth/webapp"
Sigils will only be resolved within strings inside the code unless the unsafe flag is set to True. For example:
from sigils import execute, context
with context(
USERNAME="'arthexis'",
):
result = execute("print([[USERNAME]])", unsafe=True)
assert result == "arthexis"
Async & Multiprocessing
All sigils are resolved asynchronously and in-parallel, so you can use them in loops, conditionals, and other control structures. For example:
from sigils import execute, splice, context
with context("context.json"):
if result := execute("if '[[USERNAME]]' == 'arthexis': print('[[GROUP]]')"):
assert splice("[[USERNAME]]") == "arthexis"
assert result == "Administrators"
This can also make it more efficient to resolve documents with many sigils, instead of resolving each one individually.
Django Integration
There are several ways to integrate sigils with Django:
Use the context decorator to set the context for a view.
Use the resolve template tag to resolve sigils in templates.
Context Decorator
The context decorator can be used to set the context for a view. For example, to set the context for a view in views.py:
from sigils import context
@context(
USERNAME="arthexis",
SETTING={"BASE_DIR": "/home/arth/webapp"},
)
def my_view(request):
...
You can create a simple tag to resolve sigils in templates. Create <your_app>/templatetags/sigils.py with the following code:
import sigils
from django import templates
register = template.Library()
@register.simple_tag
def resolve(text):
return sigils.resolve(text)
In app.py add the following to register a model in the global context (rename MyModel to the name of your model class):
import sigils
from django.apps import AppConfig
class MyAppConfig(AppConfig):
def ready():
from .models import MyModel
def my_model_lookup(parent, slug):
if not parent:
return MyModel.objects.filter(slug=slug)
return parent.my_models.get(slug=slug)
sigils.set_context("MY_MODEL", my_model_lookup)
You can change the callable param to make your model searchable with a different argument or manager, here the primary key is used.
Then you can use something like this in your template:
{% load sigils %}
{% sigil '[[SOME_MODEL=[USER].SOME_FIELD]]' %}
Command Line Interface
The sigils command line tool can be used to resolve sigils in arguments or files, with a given context. Example usage:
$ sigils --help
Usage: python -m sigils [OPTIONS] [TEXT]...
Resolve sigils found in the given text.
Options:
-e, --on-error [raise|remove|default|ignore]
What to do when a sigil cannot be resolved.
Default: ignore.
-d, --default TEXT Default value for ignored sigils.
-v, --verbose Increase verbosity.
-i, --interactive Enter interactive mode.
-f, --file TEXT Read text from given file.
-c, --context TEXT Context to use for resolving sigils.
--help Show this message and exit.
$ sigils "Hello, [[USERNAME]]!"
# arthexis/sigils
Project Dependencies
Feature Requests & Bug Reports
All feature requests and bug reports are welcome. Please open an issue on GitHub Issues.
Issues must use one of the approved templates. If you don’t know which one to use, use the “Bug Report” template.
Special Thanks
My wife, Katia Larissa Jasso García, for the name “sigils”.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.