yet-more-functools 2.0.1

Additional functional tools for python.

Yet More Functools

Summary

1. sow and reap. reap takes a function, its positional and keyword arguments, and returns a 2-tuple: (the result of a function call, and collections of any values that sow is called on during the execution of that function). reap may also be used to set up callbacks for values sow is called on.
2. throw and catch. catch takes a function, its positional and keyword arguments, and returns either the result of a function call or a 2-tuple: (tag, value) that throw is called on during the execution of that function.
3. stash and grab. stash takes a dict of tags and values, a function, its positional and keyword arguments and returns the result of calling that function. During the evaluation of that function the values stashed may be retrieved by a call to grab with the appropriate tag. In some sense this function pair performs the opposite function to sow and reap.

---

sow and reap

These are two functions that work together to allow values to be returned from arbitrarily deep in the call hierarchy without modifying the return signatures of intermediate functions. This can be done either: by using reap to collect and return the values saved into collections by sow or by using reap to register callback functions that are called when sow is called (collections collect the values sow is called on, callbacks are called with the values sow is called on).

sow has a simple call signature. It takes one argument x and one keyword argument tag which defaults to None. It pushes the value x to the destination(s) identified by tag which is a hashable identifier or a sequence of them. sow returns the value it is called with.

sow(x, [tag=None]) -> x

reap takes a callable function f as its first argument; followed by f's arguments, keyword-arguments and finally a keyword-argument tag which defaults to None. Tag can be a hashable identifier, a dict:{identifier: destination, ...} or a sequence of these. reap sets up destinations (collections, collection constructors or callbacks) for the values that sow is called on, anywhere in the call hierarchy below f. (See below for more information on tag). reap returns a 2-tuple: (the value returned by f, a dict of tag:collection or tag:callback), where the collections will contain the values sow was called on.

result, reaped = reap(f, *args, **kwargs, [tag=None])

reap also has a context-manager equivalent called reaper.

---

Tags

Most simply, tags are just hashable identifiers of destinations: collections that sow should append values to, or callbacks sow should invoke.

From the point of view of sow, the tag is always just the id of a destination for the values it is called on, that has already been set up by a call to a reap higher up the call hierarchy. This destination may be a collection, in which case the value is appended, or a callback function, in which case the callback is invoked on the value. If the destination is None then the call to sow does nothing. Tag may also be a sequence, in which case sow sends the value to all the destinations identified by that sequence.

Tags as far as reap is concerned can be more complicated and can take the form of:

1. A hashable id (corresponding to one in an inner sow).
2. A dict: {id: destination, ...}.
3. A sequence (not a tuple) of hashable ids or dicts or a mix of both.

A destination may be:

1. A zero-argument constructor for a collection with an append method.
2. An explicit collection with an "append" method.
3. A callback function (which requires one positional argument as reap will initially try to call it with no arguments to determine if it is a collection constructor).
4. None, which will effectively disable any corresponding sow(s).

The default tag of reap is None, this is also the default tag of sow. Within reap, if no destination is given for a tag, then the destination becomes the constructor "list" which then builds a list instance for each tag to hold the values sow is called on and return them via reap.

As mentioned above, if a sow is encountered with a tag that has not been set up by an enclosing reap, those values AND their tag, as a 2-tuple: (tag, value), will be pushed to the destination for the tag None if this exists and a warning emitted. If it does not, a different warning will be emitted.

Nesting

If an outer reap calls a function that somewhere deeper in the call hierarchy calls an inner reap using the same tag, those values sown below the inner reap will be sent to destinations (collections or callbacks) set up by the inner reap on that tag. Only those values sown above that level will be sent to destinations belonging to the outer reap. This only applies if the tags are the same; but as the default tag is None, this can occur.

Internally reap maintains a FIFO stack of destinations for each tag. In most cases there would only be one. If there are more the one from the most deeply nested reap is used. As the call hierarchy is wound up, and each reap returns, reap removes the most recent destination for its tags from the stack and returns it. The stacks of destinations are held in a contextvars.ContextVar object and so should be threadsafe and async safe.

---

Example 1 (See tests/sow_reap/example1.py) - Basic use

Imagine we are providing the function f to be called within another (library) function g. This f might be a function to set up a connection to an external service, or a function to be numerically optimized (see example_newton_raphson.py) or otherwise used by g (or still deeper calls made by g). We may need to know if **g ** is calling f at all, with what parameters, details of f's internal state during execution, or how many times it is called; but we cannot modify g. However, within f we can use sow to save a record of those values. We can recover these values by wrapping the call to g with a call to reap.

def f(*args, **kwargs):
    sow((args, kwargs))
    # ... do some more stuff...
    return_value = "return value from f" 
    sow(return_value)
    return return_value

The function g we cannot change and possibly cannot easily see inside. Unknown to us, it calls f twice with two different values.

def g(f):
    # Do unknown stuff...
    a_value_g_needs = f("Unknown value 1")
    # ...more unknown stuff...
    another_value_g_needs = f("Unknown value 2")
    # ...do yet more  unknown stuff.
    return "Return value of g"

Calling reap on g returns a 2-tuple: (the return value of g, a list of the "reaped" values from sow). reap takes g as its first argument and then takes the arguments with which to invoke g.

print(reap(g, f))

This produces a 2-tuple: (the value returned by g, the dictionary returned by reap):

("Return value of g", {None: [(("Unknown value 1",), {}), "return value from f", (("Unknown value 2",), {}), "return value from f"]})

The reaped values are returned as a dictionary with the key None. This is because no tag has been specified in the calls to sow and None is the default tag for both reap and sow.

Callback alternative (See tests/sow_reap/example1_callback.py)

Instead of returning the values sow is called on as lists; reap may be used to set up callbacks that are invoked on each value sow is called on.

This may be achieved simply by passing the callback function into reap, as a 2-tuple with the relevant tag.

def callback(x):
    print(f"Callback called with '{x}'.")

print(reap(g, f, tag={None: callback}))

This will result in the callback running everytime sow is called, and finally reap returning a 2-tuple: (the return value of g, the dict from reap containing the callback).

Callback called with "(("Unknown value 1",), {})".
Callback called with "return value from f".
Callback called with "(("Unknown value 2",), {})".
Callback called with "return value from f".
("Return value of g", {None: <function callback at 0x0000023CC6564AE0>})

sow can be disabled from reap/reaper.

If we need to turn off the collection or callbacks of values by sow for a specific tag (including the default tag of None) we can do this by passing None as the collection, collection-constructor or callback in reap or reaper.

print(reap(g, f, tag={"input":callback_input, "return": None}))

This will produce:

Input callback called with "(("Unknown value 1",), {})".
Input callback called with "(("Unknown value 2",), {})".
("Return value of g", {"input": <function callback_input at 0x000002AB45DD4AE0>, "return": None})

The return sow has been disabled.

---

Example 2 (See tests/sow_reap/example2.py) - as a context-manager

As an alternative to reap, reaper can be used in the guise of a context-manager...

with reaper() as r:
    print(g(f))
print(r())

This will produce:

Return value of g
{None: [(("Unknown value 1",), {}), "return value from f", (("Unknown value 2",), {}), "return value from f"]}

Callback alternative (See tests/sow_reap/example2_callback.py)

def callback(x):
    print(f"Callback called with '{x}\".")

with reaper(tag={None: callback}) as r:
    print(g(f))
print(r())

This will produce:

Callback called with "(("Unknown value 1",), {})".
Callback called with "return value from f".
Callback called with "(("Unknown value 2",), {})".
Callback called with "return value from f".
Return value of g
{None: <function callback at 0x000001F7B0D54AE0>}

Note: reap itself now returns the tag:callback function dictionary.

---

Example 3 (See tests/sow_reap/example3.py) - with named tags

We can improve the separation of values in example 1 by using tags to set up different reaps for different values and sow to these different tags. Rather than everything that is passed to sow coming out in one big list, items will then be collected as multiple lists.

We can modify f so that the first sow uses a different tag to the second sow. A tag can be any hashable value. Without a tag specified a default tag of None is used.

def f(*args, **kwargs):
    sow((args, kwargs), tag="input")
    # ... do some more stuff...
    return_value = "return value from f" 
    sow(return_value, tag="return")
    return return_value

This will produce:

r() -> {input: [(("Unknown value 1",), {}), "return value from f", (("Unknown value 2",), {}), "return value from f"]}

If sow is called with a specific tag for which there is no corresponding outer reap a 2-tuple: (tag, value) will be pushed onto the tag=None collection (or callback) and a warning issued.

Callback alternative (See tests/sow_reap/example3_callback.py)

Setting up two callbacks, one for each tag.

def callback_input(x):
    print(f"Input callback called with '{x}'.")

def callback_return(x):
    print(f"Return callback called with '{x}'.")

print(reap(g, f, tag={"input": callback_input, "return": callback_return}))

This will produce:

Input callback called with "(("Unknown value 1",), {})".
Return callback called with "return value from f".
Input callback called with "(("Unknown value 2",), {})".
Return callback called with "return value from f".
("Return value of g", {"input": <function callback_input at 0x0000027EE6B94AE0>, "return": <function callback_return at 0x0000027EE6B956C0>})

---

throw and catch

throw and catch are also a pair of functions that work together to return values from deep in the call hierarchy. However, unlike sow and reap, when they do so, they immediately interrupt execution flow. As soon as throw throws a value the nearest encapsulating catch, for the same tag, will return.

throw has a simple call signature. It takes one argument x and one keyword argument tag which defaults to None. It throws the value x to the catch identified by tag, a hashable identifier or a sequence of them. throw interrupts execution flow at the point it is called. Internally catch sets up a conditional try...except block and throw conditionally raises a specific type of exception which carries the value throw was called on. If throw has been disabled by an enclosing call to catch then throw will simply return the value it was called on.

throw(x, [tag=None]) -> x

catch takes a callable function f as its first argument followed by f's arguments and keyword-arguments and finally a keyword-argument tag which defaults to None. Tag can be a hashable identifier, a dict of {id: boolean, ...} or a sequence of these. The boolean in the tuple can be used to turn a throw below this call to catch on or off, for the given tag. catch catches values thrown by throw anywhere in the call hierarchy below f. catch returns a 2-tuple: (the relevant tag, the value thrown by throw) or the value returned by f if no throws occur or none are relevant or active.

tag, value = catch(f, *args, **kwargs, [tag=None])

As with sow and reap, if a throw uses a tag for which there is no enclosing catch using that same tag but there is an enclosing catch using the default tag of None a 2-tuple (tag, value) will be received by that enclosing catch and a warning will be issued. If there is no enclosing catch, for the correct tag or None, then a different warning will be issued.

catch has a context-manager equivalent called catcher.

---

Example 1 (See tests/throw_catch/example1.py)

Basic operation using the default tag of None.

def f(x):
    return throw(x) + 1
    
# Example A.
# The result is 2 because there is no surrounding catch so throw does nothing but return x, f returns normally.
# NOTE: Throw emits a warning as it is expecting to be called within a catch/catcher.
print(f"Example A -> {f(1)}")
flush_and_pause()

# Example B.
# The result is (None, 1`) because there is a surrounding catch so throw interrupts f.
print(f"Example B -> {catch(f, 1)}")
flush_and_pause()

# Example C.
# This exactly the same as B but with the tag and default setting.
# The result is (None, 1`) because there is a surrounding catch so throw interrupts f.
print(f"Example C -> {catch(f, 1, tag={None: True})}")
flush_and_pause()

# Example D.
# The result is 2 because although there is a surrounding catch we've turned
# the throw off for that tag (None) so throw does nothing, f returns normally.
# There is no warning because throw happens inside a catch/catcher.
print(f"Example D -> {catch(f, 1, tag={None: False})}")
flush_and_pause()

# Example E.
# The result is (None, 1) because there is a surrounding catch we've turned
# on throw for that tag (None) so throw interrupts f.
print(f"Example E -> {catch(f, 1, tag={None: True})}")
flush_and_pause()

produces...

Example A -> 2
..\yetmorefunctools\src\yet_more_functools\throw_catch\thow_catch.py:79: UserWarning: No tag None found as there is no outer catch for this tag OR tag=None.
  warnings.warn(f'No tag {tag} found as there is no outer catch for this tag OR tag=None.',


Example B -> (None, 1)


Example C -> (None, 1)


Example D -> 2


Example E -> (None, 1)

The call f(1) will return 2 because there is no encapsulating catch so the call to throw does nothing (throw simply returns the value it was called with but emits a warning about no encapsulating catch/catcher).

The call catch(f, 1) will return (None, 1) as within the encapsulating call to catch the throw becomes active and will interrupt the execution of f. The default tag is None.

We can control this from the catch using the tag keyword. The call catch(f, 1, tag={None: False}) will again return 2 as this has disabled throw inside this catch using tag=None. The call catch(f, 1, tag={None: True}) will return (None, 1).

Example 2 (See tests/throw_catch/example2.py) - with named tags

More advanced behaviour using different tags and the special behaviour of the None tag catching values thrown on other tags if they are not specifically being caught and catch/catcher is using the default tag=None.

def f(x):
    # Will try to **throw** on either of these two tags.
    return throw(x, tag=["tag0", "tag1"]) + 1
    
# Example A
# Returns (None, ("tag0", 1)), NOT the return value from f, as throw is not using the default tag=None.
# It's caught by the default tag of None (as no tag is specified in catch) which "hoovers up" anything
# thrown on another tag - with a warning.
print(f"Example A ->  {catch(f, 1)}")
flush_and_pause()

# Example A2
# Returns 2, the return value from f, as throw is not using the default tag=None.
# Disable None catching these other tags.
print(f"Example A2 ->  {catch(f, 1, tag={None: False})}")
flush_and_pause()

# Example B
# Returns ("tag0", 1) as catch is listening for one of the tags throw is using.
print(f"Example B -> {catch(f, 1, tag="tag0")}")
flush_and_pause()

# Example C
# Returns ("tag0", 1) as throw and catch are using the same tags, the first tag will be used.
print(f"Example C -> {catch(f, 1, tag=["tag0", "tag1"])}")
flush_and_pause()

# Example D
# Returns ("tag0", 1) still the first tag (in the order throw tried them) will be used."
print(f"Example D -> {catch(f, 1, tag=["tag1", "tag0"])}")
flush_and_pause()

# Example E
# "Returns ("tag1", 1) second tag as this is the only tag catch is listening for.
# Issues a warning because throw tries tag0 first and there is no catch for it.
print(f"Example E -> {catch(f, 1, tag=["tag1"])}")
flush_and_pause()

# Example F
# Returns , the return value from f, as now catch isn't listening for any tag throw is using
# and issues a warning about this in the process.
print(f"Example F -> {catch(f, 1, tag=["tag2"])}")
flush_and_pause()

produces...

Example A ->  (None, ('tag0', 1))
...\yetmorefunctools\src\yet_more_functools\throw_catch\thow_catch.py:74: UserWarning: No tag tag0 found as there is no outer catch for this specific tag.  Values will be sent to the tag=None catch which does exist as tuples of (tag, value).
  warnings.warn(


Example A2 ->  2


Example B -> ('tag0', 1)


Example C -> ('tag0', 1)


Example D -> ('tag0', 1)


Example E -> ('tag1', 1)
...\yetmorefunctools\src\yet_more_functools\throw_catch\thow_catch.py:79: UserWarning: No tag tag0 found as there is no outer catch for this tag OR tag=None.
  warnings.warn(f'No tag {tag} found as there is no outer catch for this tag OR tag=None.',


...\yetmorefunctools\src\yet_more_functools\throw_catch\thow_catch.py:79: UserWarning: No tag tag1 found as there is no outer catch for this tag OR tag=None.
  warnings.warn(f'No tag {tag} found as there is no outer catch for this tag OR tag=None.',
Example F -> 2

The call f(1) or the call catch(f, 1) will return 2. As in the first case, there is no encapsulating catch and in the second case catch will be listening for throws on the two specific tags given, not on the default tag of None.

The calls catch(f, 1, tag="tag0"), catch(f, 1, tag=["tag0", "tag1"]) and catch(f, 1, tag=["tag1", "tag0"]) all return (tag0, 1) because throw is trying to use tag0 first.

The call catch(f, 1, tag=["tag1"]) will return (tag1, 1) because throw is also trying to use tag1 and in this case catch is not listening for tag0.

The call catch(f, 1, tag=["tag2"]) will return 2 because throw isn't trying to use tag2 and so throw will again do nothing and f will return normally.

Example 3 (See tests/throw_catch/example3.py) - as a context-manager

catcher allows function calls to be wrapped in a context-manager "with" block that listens for calls to throw (on tags passed into catcher). catcher emits a callable (ct in the examples below) to be called after the managed block exits to pick up any value thrown during the execution of the block. If the block executes normally and nothing is thrown inside then the result of calling ct will be None. If anything is thrown during execution then execution of the block will be interrupted and the result of calling ct() will be a 2-tuple: (tag, thrown value).

def f(x):
    # Will try to throw on either of these two tags.
    return throw(x, tag=["tag0", "tag1"]) + 1
    
# Example A
# Returns (None, ("tag0", 1)) as throw is not using the default tag=None.
# And there is a warning because catch isn't using the tags throw is.
with catcher() as ct:
    print(f"Example A -> {f(1)}")  # Doesn't happen.
print(f"Example A -> ct() = {ct()}")
flush_and_pause()

# Example B
# Returns ("tag0", 1) as catch is listening for one of the tags throw is using.
with catcher(tag="tag0") as ct:
    print(f"Example B -> {f(1)}")  # Doesn't happen.
print(f"Example B -> ct() = {ct()} ")
flush_and_pause()

# Example C
# Returns ("tag0", 1) as throw and catcher are using the same tags, the first tag will be used.
with catcher(tag=["tag0", "tag1"]) as ct:
    print(f"Example C -> {f(1)}")  # Doesn't happen.
print(f"Example C -> ct() = {ct()}")
flush_and_pause()

# Example D
# Returns ("tag0", 1) still the first tag (in the order throw got them) will be used.
# Catcher is listening for both tag1 and tag0 BUT throw uses tag0 first and first wins.
with catcher(tag=["tag1", "tag0"]) as ct:
    print(f"Example D -> {f(1)}")  # Doesn't happen.
print(f"Example D -> ct() = {ct()}")
flush_and_pause()

# Example E
# Returns ("tag1", 1) second tag as this is the only tag catch is listening for.
# Catcher is only listening for tag1 and throw tries tag0 first (no catcher) and then tag1.
# There is a warning because there is no catcher for tag0.
with catcher(tag=["tag1"]) as ct:
    print(f"Example E -> {f(1)}")  # Doesn't happen.
print(f"Example E -> ct() = {ct()}")
flush_and_pause()

# Example F
# The function f actually returns because throw doesn't do anything.
# So ct() which contains the thrown value is None.
with catcher(tag=["tag2"]) as ct:
    print(f"Example F -> {f(1)}")  # This DOES happen because there is no catcher for tag0 or tag1.
print(f"Example F -> ct() = {ct()}") # So this is None.

produces...

Example A -> ct() = (None, ('tag0', 1))
...\yetmorefunctools\src\yet_more_functools\throw_catch\thow_catch.py:74: UserWarning: No tag tag0 found as there is no outer catch for this specific tag.  Values will be sent to the tag=None catch which does exist as tuples of (tag, value).
  warnings.warn(


Example B -> ct() = ('tag0', 1) 


Example C -> ct() = ('tag0', 1)


Example D -> ct() = ('tag0', 1)


Example E -> ct() = ('tag1', 1)
...\yetmorefunctools\src\yet_more_functools\throw_catch\thow_catch.py:79: UserWarning: No tag tag0 found as there is no outer catch for this tag OR tag=None.
  warnings.warn(f'No tag {tag} found as there is no outer catch for this tag OR tag=None.',


Example F -> 2
Example F -> ct() = None
...\yetmorefunctools\src\yet_more_functools\throw_catch\thow_catch.py:79: UserWarning: No tag tag1 found as there is no outer catch for this tag OR tag=None.
  warnings.warn(f'No tag {tag} found as there is no outer catch for this tag OR tag=None.',

---

stash and grab

These are two functions that work together to allow values to be passed deep into the call hierarchy without modifying the call signatures of intermediate functions. stash takes in a dict, a function f and its positional and keyword arguments. It makes the values of that dict available to any code executed within the call to f via calls to grab using the appropriate tags.

grab has a simple call signature. It takes one argument tag which defaults to None. It returns the value set up for that tag by an enclosing call to stash.

v = grab([tag=None])

stash takes a dict, setting up the tags and values for the call to f as well as f's arguments and returns the result of the call to f. It makes the values in the dict stashed available during the evaluation of the function f.

r = stash(stashed, f, *args, **kwargs)

stash also has a context-manager equivalent called stasher.

Example 1 (See tests/stash_grab/example1.py)

def f(x):
    return x+grab()

or functionally identical but more explicit:-

def f(x):
    return x+grab(tag=None)

Then running the following.

result = stash({None: 2}, f, 1)

The value result will be 3 which is the sum of 1 from the input and 2 from the stashed value of 2 (under the default tag of None for grab).

Example 2 (See tests/stash_grab/example2.py) - as a context-manager

def f(x):
    return x+grab()

or functionally identical but more explicit:-

def f(x):
    return x+grab(tag=None)

Then running the following.

with stasher({None: 2}):
    result = f(1)

The value result will be 3 which is the sum of 1 from the input and 2 from the stashed value of 2 (under the default tag of None for grab).

Example 3 (See tests/stash_grab/example3.py) - nesting

def f(x):
    with stasher({"Nested": 5}):
        temp = g(x)
    return temp * grab(tag="Global") + grab(tag="Nested")


def g(x):
    return x * grab(tag="Global") + grab(tag="Nested")


with stasher({"Global": 2, "Nested": 3}):
    result = f(1)

This will produce a result of 17. The call to stasher inside f means that the value associated with the tag Nested supersedes the one set outside, so for the duration of the call to g it becomes 5 not 3. Outside of this inner context managed block it reverts to the value set up by the outer context managed block of 3.

The same effect can be produced using stash but this is perhaps less clear.

def f(x):
    return stash({"Nested": 5}, g, x) * grab(tag="Global") + grab(tag="Nested")


result = stash({"Global": 2, "Nested": 3}, f, 1)