custom-literals 0.1.3

A module implementing custom literal suffixes using pure Python

custom-literals

A module implementing custom literal suffixes using pure Python. custom-literals mimics C++'s user-defined literals (UDLs) by defining literal suffixes that can be accessed as attributes of literal values, such as numeric constants, string literals and more.

(c) RocketRace 2022-present. See LICENSE file for more.

Examples

See the examples/ directory for more.

Function decorator syntax:

from custom_literals import literal
from datetime import timedelta

@literal(float, int, name="s")
def seconds(self):
    return timedelta(seconds=self)

@literal(float, int, name="m")
def minutes(self):
    return timedelta(seconds=60 * self)

print(30 .s + 0.5.m) # 0:01:00

Class decorator syntax:

from custom_literals import literals
from datetime import timedelta

@literals(float, int)
class Duration:
    def s(self):
        return timedelta(seconds=self)
    def m(self):
        return timedelta(seconds=60 * self)

print(30 .s + 0.5.m) # 0:01:00

Removing a custom literal:

from custom_literals import literal, unliteral

@literal(str)
def u(self):
    return self.upper()

print("hello".u) # "HELLO"

unliteral(str, "u")
assert not hasattr("hello", "u")

Context manager syntax (automatically removes literals afterwards):

from custom_literals import literally
from datetime import timedelta

with literally(float, int, 
    s=lambda x: timedelta(seconds=x), 
    m=lambda x: timedelta(seconds=60 * x)
):
    print(30 .s + 0.5.m) # 0:01:00

Features

Currently, three methods of defining custom literals are supported: The function decorator syntax @literal, the class decorator syntax @literals, and the context manager syntax with literally. (The latter will automatically unhook the literal suffixes when the context is exited.) To remove a custom literal, use unliteral.

Custom literals are defined for literal values of the following types:

Type	Example	Notes
int	(42).x	The Python parser interprets 42.x as a float literal followed by an identifier. To avoid this, use (42).x or 42 .x instead.
float	3.14.x
complex	1j.x
bool	True.x	Since bool is a subclass of int, int hooks may influence bool as well.
str	"hello".x	F-strings (f"{a}".x) are also supported.
bytes	b"hello".x
None	None.x
Ellipsis	....x	Yes, this is valid syntax.
tuple	(1, 2, 3).x	Generator expressions ((x for x in ...)) are not tuple literals and thus won't be affected by literal suffixes.
list	[1, 2, 3].x	List comprehensions ([x for x in ...]) may not function properly.
set	{1, 2, 3}.x	Set comprehensions ({x for x in ...}) may not function properly.
dict	{"a": 1, "b": 2}.x	Dict comprehensions ({x: y for x, y in ...}) may not function properly.

In addition, custom literals can be defined to be strict, that is, only allow the given literal suffix to be invoked on constant, literal values. This means that the following code will raise a TypeError:

@literal(str, name="u", strict=True)
def utf_8(self):
    return self.encode("utf-8")

my_string = "hello"
print(my_string.u) 
# TypeError: the strict custom literal `u` of `str` objects can only be invoked on literal values

By default, custom literals are not strict. This is because determining whether a suffix was invoked on a literal value relies on bytecode analysis, which is a feature of the CPython interpreter, and is not guaranteed to be forwards compatible. It can be enabled by passing strict=True to the @literal, @literals or literally functions.

Caveats

Stability

This library relies almost entirely on implementation-specific behavior of the CPython interpreter. It is not guaranteed to work on all platforms, or on all versions of Python. It has been tested on common platforms (windows, ubuntu, macos) using python 3.7 through to 3.10, but while changes that would break the library are quite unlikely, they are not impossible either.

That being said, custom-literals does its absolute best to guarantee maximum stability of the library, even in light of possible breaking changes in CPython internals. The code base is well tested. In the future, the library may also expose multiple different backends for the actual implementation of builtin type patching. As of now, the only valid backend is forbiddenfruit, which uses the forbiddenfruit library.

Feature	Stability
Hooking with the forbiddenfruit backend	Quite stable, but may be affected by future releases. Relies on the ctypes module.
Strict mode using the strict=True kwarg	Quite stable, but not forwards compatible. Relies on stack frame analysis and opcode checks to detect non-literal access.

Type safety

The library code, including the public API, is fully typed. Registering and unregistering hooks is type-safe, and static analysis tools should have nothing to complain about.

However, accessing custom literal suffixes is impossible to type-check. This is because all major static analysis tools available for python right now (understandably) assume that builtins types are immutable. That is, the attributes and methods builtin types cannot be dynamically modified. This goes against the core idea of the library, which is to patch custom attributes on builtin types.

Therefore, if you are using linters, type checkers or other static analysis tools, you will likely encounter many warnings and errors. If your tool allows it, you should disable these warnings (ideally on a per-diagnostic, scoped basis) if you want to use this library without false positives.

FAQ

Should I use this in production?

Emphatically, no! But I won't stop you.

Nooooooo (runs away from computer)

I kind of disagree: yessss (dances in front of computer)

Why?

Python's operator overloading allows for flexible design of domain-specific languages. However, Python pales in comparison to C++ in this aspect. In particular, User-Defined Literals (UDLs) are a powerful feature of C++ missing in Python. This library is designed to emulate UDLs in Python, with syntactic sugar comparable to C++ in elegance.

But really, why?

Because it's posssible.

How? (please keep it short)

custom-literals works by patching builtin types with custom objects satisfying the descriptor protocol, similar to the builtin property decorator. The patching is done trough a "backend", which is an interface implementing functions to mutate the __dict__ of builtin types. If strict=True mode is enabled, the descriptor will also traverse stack frames backwards to the invokation site of the literal suffix, and check the most recently executed bytecode opcode to ensure that the literal suffix was invoked on a literal value.

How? (I love detail)

Builtin types in CPython are implemented in C, and include checks to prevent mutation at runtime. This is why the following lines will each raise a TypeError:

int.x = 42 # TypeError: cannot set 'x' attribute of immutable type 'int'
setattr(int, "x", 42) # TypeError: cannot set 'x' attribute of immutable type 'int'
int.__dict__["x"] = 42 # TypeError: 'mappingproxy' object does not support item assignment

However, these checks can be subverted in a number of ways. One method is to use CPython's APIs directly to bypass the checks. For the sake of stability, custom-literals calls the curse() and reverse() functions of the forbiddenfruit library to implement these bypasses. Internally, forbiddenfruit uses the ctypes module to access the C API and use the ctypes.pythonapi.PyType_Modified() function to signal that a builtin type has been modified. Other backends may also be available in the future, but are not implemented at the moment. (As an example, there is currently a bug in CPython that allows mappingproxy objects to be mutated without using ctypes. This was deemed too fragile to be included in the library.)

Python's @property decorator implements the descriptor protocol. This is a protocol that allows for custom code to be executed when accessing specific attributes of a type. For example, the following code will print 42:

class MyClass:
    @property
    def x(self):
        print(42)

MyClass().x

custom-literals patches builtin types with objects implementing the same protocol, allowing for user-defined & library-defiend code to be executed when invoking a literal suffix on a builtin type. It cannot however use @property directly, as elaborated below.

The descriptor protocol is very flexible, used as the the backbone of bound methods, class methods, and static methods and more. It is defined by the presence of one of the following methods*:

class SomeDescriptor:
    # <instance>.<attribute>
    def __get__(self, instance, owner) -> value: ...
    # <instance>.<attribute> = <value>
    def __set__(self, instance, value) -> None: ...
    # del <instance>.<attribute>
    def __delete__(self, instance) -> None: ...

*and optionally __set_name__

The descriptor methods can be invoked from an instance (some_instance.x) or from a class (SomeClass.x). Importantly for us, the __get__ method is called with different arguments depending on whether the descriptor is accessed from an instance or a class:

class MyDesciptor:
    def __get__(self, instance, owner) -> value:
        print(f"Instance: {instance}")
        print(f"Owner: {owner}")

class MyClass:
    x = MyDesciptor()

MyClass().x 
# Instance: <__main__.MyClass object at 0x110e3a170> 
# Owner: <class '__main__.MyClass'>
MyClass.x 
# Instance: None 
# Owner: <class '__main__.MyClass'>

This is used to differentiate between the two cases. @property's implementation simply returns the descriptor instance if instance is None, which is a fair test for whether the descriptor is accessed from a class or an instance.

Keen-eyed readers may notice however that this is not a perfect test. What if MyClass is somehow type(None)? In this case, the two cases will be indistinguishable. In normal code, this is not a problem, as type(None) is a builtin type, and thus cannot be mutated. In custom-literals, however, this breaks custom literals that are defined on type(None).

This can thankfully be mitigated thanks to the concept of a data descriptor. A data descriptor is a descriptor that defines __set__ or __delete__. When Python tries to resolve attribute access on an instance, it will first check whether its type has a data descriptor for the attribute, overriding any descriptors defined on the instance itself. For example, suppose the following example using a metaclass (a class inheriting from type):

class DataDescriptor:
    def __get__(self, instance, owner):
        print("The data descriptor was called!")
        print(f"Instance: {instance}")
    
    # Simply the presence of the method is enough
    # to convert this into a data descriptor
    def __set__(self, instance, value):
        raise AttributeError

class NormalDescriptor:
    def __get__(self, instance, owner):
        print("The normal descriptor was called!")
        print(f"Instance: {instance}")

class MyMeta(type):
    x = DataDescriptor()

class MyClass(metaclass=MyMeta):
    x = NormalDescriptor()

MyClass.x 
# The data descriptor was called!
# Instance: <class '__main__.MyClass'>
MyClass().x 
# The normal descriptor was called!
# Instance: <__main__.MyClass object at 0x10f468ee0>s

This example shows that it is possible to ensure that a descriptor is always called on an instance with instance set to an instance of the class. In the case of custom-literals, this is achieved by patching a data descriptor (any data descriptor) on type when type(None) is also being patched. This removes the ambiguity of whether the descriptor is called on an instance or a class. Yay!

Finally, custom-literals also provides a mechanism for optionally detecting when a custom literal suffix is invoked on a constant and literal type. (This is invoked when the strict argument is set to True.) This is achieved by attaching extra code to the __get__ method of the custom literal descriptor. The code performs bytecode analysis at the invocation site of the custom literal suffix.

The CPython interpreter uses stack frames to implement function calls. When a function is called, a new frame is created and pushed to the stack, and when the function returns, the frame is popped off the stack. Importantly, these frame objects can be accessed directly from Python:

import inspect

def foo():
    local_variable = 123
    bar()

def bar():
    # Alternatively, use `sys._getframe()`
    frame = inspect.currentframe()
    # The `f_back` attribute of a frame object
    # points to the frame that called it
    previous_frame = frame.f_back
    # Frame objects have information about the
    # invocation context of the frame, including
    # e.g. local variables
    previous_locals = previous_frame.f_locals
    print(previous_locals['local_variable']) # 123

The f_code attribute of a frame object contains information about the bytecode of the currently executed code. CPython being an interpreter, this bytecode corresponds roughly to the source code of the function. For example, see the disassembly of the following:

import dis

def add(a, b):
    c = a + b
    return int(c)

dis.dis(add) # Outputs:
# 4           0 LOAD_FAST                0 (a)
#             2 LOAD_FAST                1 (b)
#             4 BINARY_ADD
#             6 STORE_FAST               2 (c)
# 
# 5           8 LOAD_GLOBAL              0 (int)
#            10 LOAD_FAST                2 (c)
#            12 CALL_FUNCTION            1
#            14 RETURN_VALUE

• First, the two arguments a and b are pushed onto the stack.
• The arguments are popped from the stack and used as the operands for +. The result is pushed onto the stack.
• The top of the stack is popped and stored in a local variable c.
• The int function is fetched from the global namespace and pushed to the stack.
• The local variable c is pushed to the stack.
• The int function is called with one argument, and the return value of int is pushed to the stack.
• The result is popped from the stack and returned.

In the case of custom literals, the opcodes we are concerned about are the following:

• LOAD_CONST, used to load a constant (including most literal values) to the stack
• BUILD_TUPLE/BUILD_LIST/BUILD_SET/BUILD_MAP, used to push tuple/list/set/dict literals to the stack
• FORMAT_VALUE, used to push a formatted f-string literal (f"{a} {b} {c}") to the stack
• LIST_TO_TUPLE/LIST_EXTEND/SET_UPDATE/DICT_UPDATE, sometimes used in list/set/dict literals, for example when using the star unpack syntax ([a, b, c, *x])

(Do keep in mind that opcodes are not necessarily forwards compatible. Python 3.11 could release a dozen new opcodes tomorrow that need to be accounted for by the library! This is why custom-literals does not perform bytecode analysis by default.)

If strict mode is enabled, the library will traverse up through the stack frames, inspect the bytecode, check the most recently executed opcode (available in frame.f_lasti), and check if it is one of the opcodes listed above. If the opcode is not in the allowed list, an error is raised, which is why the following code raises an error:

@literal(str, strict=True)
def xyz(self):
    return 123

abc = "abc"
abc.xyz 
# TypeError: the strict custom literal `xyz` of `str` objects can only be invoked on literal values

Putting all of these features together, custom-literals is able to do the seemingly impossible - define custom literal suffixes on builtin types that can only be invoked on literal values!

Making this project has been a fascinating deep dive into some of the internals of CPython, and I hope it has been equally interesting to you, the reader.

Could this ever be type safe?

I doubt it. The assumptions made by static analysis tools are incredibly useful, and this is such an edge case it makes no sense for them to assume builtin literal types can have dynamically set attributes. In addition, there isn't a good way to signal to your type checker that an immutable type is going to be endowed with new attributes!

License

(c) RocketRace 2022-present. This library is under the Mozilla Public License 2.0. See the LICENSE file for more details.

Contributing

Patches, bug reports, feature requests and pull requests are welcome.

Links

• GitHub repository
• PyPI