yet-more-functools 1.0.0

Additional functional tools for python.

SPDX-FileCopyrightText: © 2025 Roger Wilson

SPDX-License-Identifier: MIT

Yet More Functools

Summary

1. Reap takes a function, its positional and keyword arguments, and returns a 2-tuple: (the result of a function call, and collections of any values that sow is called on during the execution of that function). Reap may also be used to set up callbacks for values sow is called on.
2. Catch takes a function, its positional and keyword arguments, and returns either the result of a function call or a value throw is called on during the execution of that function.

---

Sow and Reap

These are two functions that work together to allow values to be returned from arbitrarily deep in the call hierarchy without modifying the return signatures of intermediate functions. This can be done either: by using reap to collect and return the values saved into collections by sow or by using reap to register callback functions that are called when sow is called (collections collect the values sow is called on, callbacks are called with the values sow is called on).

Sow has a simple call signature. It takes one argument x and one keyword argument tag which defaults to None. It pushes the value x to the destination(s) identified by tag which is a hashable identifier or a sequence of them. Sow returns the value it is called with.

sow(x, [tag=None])

Reap takes a callable function f as its first argument; followed by f's arguments, keyword-arguments and finally a keyword-argument tag which defaults to None. Tag can be a hashable identifier, a 2-tuple: hashable identifier, destination) or a sequence of these. Reap sets up destinations (collections or callbacks) for the values that sow is called on, anywhere in the call hierarchy below f. (See Notes below for more information on tag). Reap returns a 2-tuple: (the value returned by f, a dict of tag:collection or tag:callback), where the collections will contain the values sow was called on.

reap(f, *args, **kwargs, [tag=None])

Reap also has a context-manager equivalent called Reaper.

---

Example 1 (See tests/sow_reap/example1.py) - Basic use

Imagine we are providing the function f to be called within another (library) function g. This f might be a function to set up a connection to an external service, or a function to be numerically optimized (see example_newton_raphson.py) or otherwise used by g (or still deeper calls made by g). We may need to know if g is calling f at all, with what parameters, details of f's internal state during execution, or how many times it is called; but we cannot modify g. However, within f we can use sow to save a record of those values. We can recover these values by wrapping the call to g with a call to reap.

def f(*args, **kwargs):
    sow((args, kwargs))
    # ... do some more stuff...
    return_value = "return value from f" 
    sow(return_value)
    return return_value

The function g we cannot change and possibly cannot easily see inside. Unknown to us, it calls f twice with two different values.

def g(f):
    # Do unknown stuff...
    a_value_g_needs = f("Unknown value 1")
    # ...more unknown stuff...
    another_value_g_needs = f("Unknown value 2")
    # ...do yet more  unknown stuff.
    return "Return value of g"

Calling reap on g returns a 2-tuple: (the return value of g, a list of the "reaped" values from sow). Reap takes g as its first argument and then takes the arguments with which to invoke g.

print(reap(g, f))

This produces a 2-tuple: (the value returned by g, the dictionary returned by reap):

("Return value of g", {None: [(("Unknown value 1",), {}), "return value from f", (("Unknown value 2",), {}), "return value from f"]})

The reaped values are returned as a dictionary with the key None. This is because no tag has been specified in the calls to sow and None is the default tag for both reap and sow.

Callback alternative (See tests/sow_reap/example1_callback.py)

Instead of returning the values sow is called on as lists; reap may be used to set up callbacks that are invoked on each value sow is called on.

This may be achieved simply by passing the callback function into reap, as a 2-tuple with the relevant tag.

def callback(x):
    print(f"Callback called with '{x}'.")

print(reap(g, f, tag=(None, callback)))

This will result in the callback running everytime sow is called, and finally reap returning a 2-tuple: (the return value of g, the dict from reap containing the callback).

Callback called with "(("Unknown value 1",), {})".
Callback called with "return value from f".
Callback called with "(("Unknown value 2",), {})".
Callback called with "return value from f".
("Return value of g", {None: <function callback at 0x0000023CC6564AE0>})

---

Example 2 (See tests/sow_reap/example2.py) - as a context-manager

As an alternative to reap, reaper can be used in the guise of a context-manager...

with reaper() as r:
    print(g(f))
print(r())

This will produce:

Return value of g
{None: [(("Unknown value 1",), {}), "return value from f", (("Unknown value 2",), {}), "return value from f"]}

Callback alternative (See tests/sow_reap/example2_callback.py)

def callback(x):
    print(f"Callback called with '{x}\".")

with reaper(tag=(None, callback)) as r:
    print(g(f))
print(r())

This will produce:

Callback called with "(("Unknown value 1",), {})".
Callback called with "return value from f".
Callback called with "(("Unknown value 2",), {})".
Callback called with "return value from f".
Return value of g
{None: <function callback at 0x000001F7B0D54AE0>}

Note: Reap itself now returns the tag:callback function dictionary.

---

Example 3 (See tests/sow_reap/example3.py) - with named tags

We can improve the separation of values in example 1 by using tags to set up different reaps for different values and sow to these different tags. Rather than everything that is passed to sow coming out in one big list, items will then be collected as multiple lists.

We can modify f so that the first sow uses a different tag to the second sow. A tag can be any hashable value. Without a tag specified a default tag of None is used.

def f(*args, **kwargs):
    sow((args, kwargs), tag="input")
    # ... do some more stuff...
    return_value = "return value from f" 
    sow(return_value, tag="return")
    return return_value

This will produce:

r() -> {input: [(("Unknown value 1",), {}), "return value from f", (("Unknown value 2",), {}), "return value from f"]}

If sow is called with a specific tag for which there is no corresponding outer reap a 2-tuple: (tag, value) will be pushed onto the tag=None collection (or callback) and a warning issued.

Callback alternative (See tests/sow_reap/example3_callback.py)

Setting up two callbacks, one for each tag.

def callback_input(x):
    print(f"Input callback called with '{x}'.")

def callback_return(x):
    print(f"Return callback called with '{x}'.")

print(reap(g, f, tag=[("input", callback_input), ("return", callback_return)]))

This will produce:

Input callback called with "(("Unknown value 1",), {})".
Return callback called with "return value from f".
Input callback called with "(("Unknown value 2",), {})".
Return callback called with "return value from f".
("Return value of g", {"input": <function callback_input at 0x0000027EE6B94AE0>, "return": <function callback_return at 0x0000027EE6B956C0>})

---

Notes

Sow can be disabled from reap.

If we need to turn off the collection or callbacks of values by sow for a specific tag (including the default tag of None) we can do this by passing None as the collection, collection-constructor or callback in reap.

print(reap(g, f, tag=[("input", callback_input), ("return", None)]))

This will produce:

Input callback called with "(("Unknown value 1",), {})".
Input callback called with "(("Unknown value 2",), {})".
("Return value of g", {"input": <function callback_input at 0x000002AB45DD4AE0>, "return": None})

The return sow has been disabled.

Tags

Most simply, tags are just hashable identifiers of destinations: collections that sow should append values to, or callbacks sow should invoke.

From the point of view of sow, the tag is always just the id of a destination for the values it is called on, that has already been set up by a call to a reap higher up the call hierarchy. This destination may be a collection, in which case the value is appended, or a callback function, in which case the callback is invoked on the value. If the destination is None then the call to sow does nothing. Tag may also be a sequence, in which case sow sends the value to all the destinations identified by that sequence.

Tags as far as reap is concerned can be more complicated and can take the form of:

1. A hashable id (corresponding to one in an inner sow).
2. A 2-tuple: (a hashable id, a destination).
3. A sequence (not a tuple) of hashable ids or 2-tuples: (a hashable id, a destination) or a mix of both.

A destination may be:

1. A zero-argument constructor for a collection with an append method.
2. An explicit collection with an "append" method (possible but probably not required).
3. A callback function (which requires one positional argument as reap will initially try to call it with no arguments to determine if it is a collection constructor).
4. None, which will effectively disable any corresponding sow(s).

Within reap, if no destination is given, then the destination becomes the constructor "list" which then builds a list instance for each tag to hold the values sow is called on and return them via reap.

The default tag of reap is None which becomes (None, list) and so builds a list instance to collect values sow is called on with that tag. This is also the default tag of sow.

As mentioned above, if a sow is encountered with a tag that has not been set up by an enclosing reap, those values AND their tag, as a 2-tuple: (tag, value), will be pushed to the destination for the tag None if this exists and a warning emitted. If it does not, a different warning will be emitted.

Nesting

If an outer reap calls a function that somewhere deeper in the call hierarchy calls an inner reap using the same tag, those values sown below the inner reap will be sent to destinations (collections or callbacks) set up by the inner reap on that tag. Only those values sown above that level will be sent to destinations belonging to the outer reap. This only applies if the tags are the same; but as the default tag is None, this can occur.

Internally reap maintains a FIFO stack of destinations for each tag. In most cases there would only be one. If there are more the one from the most deeply nested reap is used. As the call hierarchy is wound up, and each reap returns, reap removes the most recent destination for its tags from the stack and returns it. The stacks of destinations are held in a contextvars.ContextVar object and so should be threadsafe and async safe.

---

Throw and Catch

Throw and catch are also a pair of functions that work together to return values from deep in the call hierarchy. However, unlike sow and reap, when they do so, they immediately interrupt execution flow. As soon as throw throws a value the nearest encapsulating catch, for the same tag, will return.

Throw has a simple call signature. It takes one argument x and one keyword argument tag which defaults to None. It throws the value x to the catch identified by tag, a hashable identifier or a sequence of them. Throw interrupts execution flow at the point it is called. Internally catch sets up a conditional try...except block and throw conditionally raises a specific type of exception.

throw(x, [tag=None])

Catch takes a callable function f as its first argument followed by f's arguments and keyword-arguments and finally a keyword-argument tag which defaults to None. Tag can be a hashable identifier, a 2-tuple: (hashable identifier, boolean) or a sequence of these. The boolean in the tuple can be used to turn a throw below this call to catch on or off, for the given tag. Catch catches values thrown by throw anywhere in the call hierarchy below f. Catch returns a 2-tuple: (the relevant tag, the value thrown by throw) or the value returned by f if no throws occur or none are relevant or active.

catch(f, *args, **kwargs, [tag=None])

As with sow and reap, if a throw uses a tag for which there is no enclosing catch using that same tag but there is an enclosing catch using the default tag of None a 2-tuple (tag, value) will be received by that enclosing catch and a warning will be issued. If there is no enclosing catch, for the correct tag or None, then a different warning will be issued.

Catch has a context-manager equivalent called Catcher.

Example 1 (See tests/throw_catch/example1.py)

def f(x):
    throw(x)
    return x + 1

The call f(1) will return 2 because there is no encapsulating catch so the call to throw does nothing.

The call catch(f, 1) will return (None, 1) as within the encapsulating call to catch the throw becomes active and will interrupt the execution of f. The default tag is None.

We can control this from the catch using the tag keyword. The call catch(f, 1, tag=(None, False)) will again return 2 as this has disabled throw inside this catch using tag=None. The call catch(f, 1, tag=(None, True)) will return (None, 1).

Example 2 (See tests/throw_catch/example2.py) - with named tags

def f(x):
    # Will try to **throw** on either of these two tags.
    throw(x, tag=["tag0", "tag1"])
    return x + 1

The call f(1) or the call catch(f, 1) will return 2. As in the first case, there is no encapsulating catch and in the second case catch will be listening for throws on the two specific tags given, not on the default tag of None.

The calls catch(f, 1, tag="tag0"), catch(f, 1, tag=["tag0", "tag1"]) and catch(f, 1, tag=["tag1", "tag0"]) all return (tag0, 1) because throw is trying to use tag0 first.

The call catch(f, 1, tag=["tag1"]) will return (tag1, 1) because throw is also trying to use tag1 and in this case catch is not listening for tag0.

The call catch(f, 1, tag=["tag2"]) will return 2 because throw isn't trying to use tag2 and so throw will again do nothing and f will return normally.

Example 3 (See tests/throw_catch/example3.py) - as a context-manager

Catcher allows function calls to be wrapped in a context-manager "with" block that listens for calls to throw (on tags passed into catcher). Catcher emits a callable (ct in the examples below) to be called after the managed block exits to pick up any value thrown during the execution of the block. If the block executes normally and nothing is thrown inside then the result of calling ct will be None. If anything is thrown during execution then execution of the block will be interrupted and the result of calling ct will be the thrown value.

def f(x):
    # Will try to throw on either of these two tags.
    throw(x, tag=["tag0", "tag1"])
    return x + 1


with catcher() as ct:
    print(f"Returns {f(1)}, the return value from f, as throw is not using the default tag=None.")
print(f"And of course ct() = {ct()} as nothing was caught because nothing was thrown.")

print()
with catcher(tag="tag0") as ct:
    print(f"This {f(1)} does not get printed because the throw fires.")
print(f"Returns ct() = {ct()} as catch is listening for one of the tags throw is using.")

print()
with catcher(tag=["tag0", "tag1"]) as ct:
    print(f"This {f(1)} does not get printed because the throw fires.")
print(f"Returns ct() = {ct()} as throw and catcher are using the same tags, the first tag will be used.")

print()
with catcher(tag=["tag1", "tag0"]) as ct:
    print(f"This {f(1)} does not get printed because the throw fires.")
print(f"Returns ct() = {ct()} still the first tag (in the order throw got them) will be used.")

print()
with catcher(tag=["tag1"]) as ct:
    print(f"This {f(1)} does not get printed because the throw fires.")
print(f"Returns ct() = {ct()} second tag as this is the only tag catch is listening for.")

print()
with catcher(tag=["tag2"]) as ct:
    print(f"Returns {f(1)}, the return value from f, as throw is not using tag2.")
print(f"Returns ct() = {ct()} as now catch isn't listening for any tag throw is using.")

This will produce:

Returns 2, the return value from f, as throw is not using the default tag=None.
And of course ct() = None as nothing was caught because nothing was thrown.

Returns ct() = ('tag0', 1) as catch is listening for one of the tags throw is using.

Returns ct() = ('tag0', 1) as throw and catcher are using the same tags, the first tag will be used.

Returns ct() = ('tag0', 1) still the first tag (in the order throw got them) will be used.

Returns ct() = ('tag1', 1) second tag as this is the only tag catch is listening for.

Returns 2, the return value from f, as throw is not using tag2.
Returns ct() = None as now catch isn't listening for any tag throw is using.