Skip to main content

meshed

Object composition. In particular: Link functions up into callable objects (e.g. pipelines, DAGs, etc.)

To install: pip install meshed

Documentation

Note: The initial focus of meshed was on DAGs, a versatile and probably most known kind of composition of functions, but meshed aims at capturing much more than that.

For AI agents

meshed ships tooling for coding agents. If you are one, start here.

The documentation, machine-readable: llms.txt indexes every page; meshed.md is the whole documentation in one file; every page has a .md twin; objects.inv maps symbols to URLs.

If you identify as a dinosaur, the rest of this README is written for you, starting at Quick Start.

Quick Start

from meshed import DAG

def this(a, b=1):
    return a + b
def that(x, b=1):
    return x * b
def combine(this, that):
    return (this, that)

dag = DAG((this, that, combine))
print(dag.synopsis_string())
x,b -> that_ -> that
a,b -> this_ -> this
this,that -> combine_ -> combine

But what does it do?

It's a callable, with a signature:

from inspect import signature
signature(dag)
<Signature (x, a, b=1)>

And when you call it, it executes the dag from the root values you give it and returns the leaf output values.

dag(1, 2, 3)  # (a+b,x*b) == (2+3,1*3) == (5, 3)
(5, 3)
dag(1, 2)  # (a+b,x*b) == (2+1,1*1) == (3, 1)
(3, 1)

You can see (and save image, or ascii art) the dag:

dag.dot_digraph()

You can extend a dag

dag2 = DAG([*dag, lambda this, a: this + a])
dag2.dot_digraph()

You can get a sub-dag by specifying desired input(s) and outputs.

dag2[['that', 'this'], 'combine'].dot_digraph()

Note on flexibility

The above DAG was created straight from the functions, using only the names of the functions and their parameters to define how to hook the network up.

But if you didn't write those functions specifically for that purpose, or you want to use someone else's functions, one would need to specify the relation between parameters, inputs and outputs.

For that purpose, functions can be adapted using the class FuncNode. The class allows you to essentially rename each of the parameters and also specify which output should be used as an argument for any other functions.

Let us consider the example below.

def f(a, b):
    return a + b

def g(a_plus_b, d):
    return a_plus_b * d

Say we want the output of f to become the value of the parameter a_plus_b. We can do that by assigning the string 'a_plus_b' to the out parameter of a FuncNode representing the function f:

f_node = FuncNode(func=f, out="a_plus_b")

We can now create a dag using our f_node instead of f:

dag = DAG((f_node, g))

Our dag behaves as wanted:

dag(a=1, b=2, d=3)
9

Now say we would also like for the value given to b to be also given to d. We can achieve that by binding d to b in the bind parameter of a FuncNode representing g:

g_node = FuncNode(func=g, bind={"d": "b"})

The dag created with f_node and g_node has only two parameters, namely a and b:

dag = DAG((f_node, g_node))
dag(a=1, b=2)
6

Sub-DAGs

dag[input_nodes:output_nodes] is the sub-dag made of intersection of all descendants of input_nodes (inclusive) and ancestors of output_nodes (inclusive), where additionally, when a func node is contained, it takes with it the input and output nodes it needs.

from meshed import DAG

def f(a): ...
def g(f): ...
def h(g): ...
def i(h): ...
dag = DAG([f, g, h, i])

dag.dot_digraph()
image

Get a subdag from g_ (indicates the function here) to the end of dag

subdag = dag['g_',:]
subdag.dot_digraph()
image

From the beginning to h_

dag[:, 'h_'].dot_digraph()
image

From g_ to h_ (both inclusive)

dag['g_', 'h_'].dot_digraph()
image

Above we used function (node names) to specify what we wanted, but we can also use names of input/output var-nodes. Do note the difference though. The nodes you specify to get a sub-dag are INCLUSIVE, but when you specify function nodes, you also get the input and output nodes of these functions.

The dag['g_', 'h_'] give us a sub-dag starting at f (the input node), but when we ask dag['g', 'h_'] instead, g being the output node of function node g_, we only get g -> h_ -> h:

dag['g', 'h'].dot_digraph()
image

If we wanted to include f we'd have to specify it:

dag['f', 'h'].dot_digraph()
image

Those were for simple pipelines, but let's now look at a more complex dag.

Note the definition: dag[input_nodes:output_nodes] is the sub-dag made of intersection of all descendants of input_nodes (inclusive) and ancestors of output_nodes (inclusive), where additionally, when a func node is contained, it takes with it the input and output nodes it needs.

We'll let the following examples self-comment:

from meshed import DAG


def f(u, v): ...

def g(f): ...

def h(f, w): ...

def i(g, h): ...

def j(h, x): ...

def k(i): ...

def l(i, j): ...

dag = DAG([f, g, h, i, j, k, l])

dag.dot_digraph()
image
dag[['u', 'f'], 'h'].dot_digraph()
image
dag['u', 'h'].dot_digraph()
image
dag[['u', 'f'], ['h', 'g']].dot_digraph()
image
dag[['x', 'g'], 'k'].dot_digraph()
image
dag[['x', 'g'], ['l', 'k']].dot_digraph()
image

Examples

A train/test ML pipeline

Consider a simple train/test ML pipeline that looks like this.

image

With this, we might decide we want to give the user control over how to do train_test_split and learner, so we offer this interface to the user:

image

With that, the user can just bring its own train_test_split and learner functions, and as long as it satisfied the expected (and even better; declared and validatable) protocol, things will work.

In some situations we'd like to fix some of how train_test_split and learner work, allowing the user to control only some aspects of them. This function would look like this:

image

And inside, it does:

image

meshed allows us to easily manipulate such functional structures to adapt them to our needs.

itools module

Tools that enable operations on graphs where graphs are represented by an adjacency Mapping.

Again.

Graphs: You know them. Networks. Nodes and edges, and the ecosystem descriptive or transformative functions surrounding these. Few languages have builtin support for the graph data structure, but all have their libraries to compensate.

The one you're looking at focuses on the representation of a graph as Mapping encoding its adjacency list. That is, a dictionary-like interface that specifies the graph by specifying for each node what nodes it's adjacent to:

assert graph[source_node] == set_of_nodes_that_source_node_has_edges_to

We emphasize that there is no specific graph instance that you need to squeeze your graph into to be able to use the functions of meshed. Suffices that your graph's structure is expressed by that dict-like interface -- which grown-ups call Mapping (see the collections.abc or typing standard libs for more information).

You'll find a lot of Mappings around pythons. And if the object you want to work with doesn't have that interface, you can easily create one using one of the many tools of py2store meant exactly for that purpose.

Examples

>>> from meshed.itools import edges, nodes, isolated_nodes
>>> graph = dict(a='c', b='ce', c='abde', d='c', e=['c', 'b'], f={})
>>> sorted(edges(graph))
[('a', 'c'), ('b', 'c'), ('b', 'e'), ('c', 'a'), ('c', 'b'), ('c', 'd'), ('c', 'e'), ('d', 'c'), ('e', 'b'), ('e', 'c')]
>>> sorted(nodes(graph))
['a', 'b', 'c', 'd', 'e', 'f']
>>> set(isolated_nodes(graph))
{'f'}
>>>
>>> from meshed.itools import edge_reversed_graph
>>> g = dict(a='c', b='cd', c='abd', e='')
>>> assert edge_reversed_graph(g) == {'c': ['a', 'b'], 'd': ['b', 'c'], 'a': ['c'], 'b': ['c'], 'e': []}
>>> reverse_g_with_sets = edge_reversed_graph(g, set, set.add)
>>> assert reverse_g_with_sets == {'c': {'a', 'b'}, 'd': {'b', 'c'}, 'a': {'c'}, 'b': {'c'}, 'e': set([])}

Release files for meshed 0.1.168

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for meshed 0.1.168
File Size Uploaded
meshed-0.1.168.tar.gz 136.0 kB Details

Built distribution (wheel)

Table of built distributions (wheels) for meshed 0.1.168
File Interpreter ABI Platform
meshed-0.1.168-py3-none-any.whl Python 3 none any Details

Total release size: 282.8 kB

Release files / meshed-0.1.168.tar.gz

Download URL meshed-0.1.168.tar.gz
Size 136.0 kB
Tags Source
SHA-256 checksum
How to use checksums
f1b3951066e141d3626b53899ffed30c6e069884b03e05963194f10e2832f931
BLAKE2b-256 checksum
How to use checksums
9d5657290652659c81dde093a57b6c6504cb473608ef88b2600f6134b4da9a6c
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.10.21

Release files / meshed-0.1.168-py3-none-any.whl

Download URL meshed-0.1.168-py3-none-any.whl
Size 146.9 kB
Tags Python 3
SHA-256 checksum
How to use checksums
b2ef1296aca4cc89f53d4bec97eb3daf30e265672a7642e2feda7d2742651cfa
BLAKE2b-256 checksum
How to use checksums
f91698b2e4213c6870f7f07dda954dd517fcff60ec2c696ec11b94a5b732dfa6
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.10.21

Release history Release notifications | RSS feed

This release

0.1.168 This release

2 release files

0.1.144

1 release file

0.1.143

1 release file

0.1.142

1 release file

0.1.141

1 release file

0.1.140

1 release file

0.1.139

1 release file

0.1.138

1 release file

0.1.137

1 release file

0.1.136

1 release file

0.1.135

1 release file

0.1.126

1 release file

0.1.125

1 release file

0.1.124

1 release file

0.1.123

1 release file

0.1.117

1 release file

0.1.116

1 release file

0.1.115

1 release file

0.1.114

1 release file

0.1.113

1 release file

0.1.112

1 release file

0.1.111

1 release file

0.1.110

1 release file

0.1.109

1 release file

0.1.108

1 release file

0.1.107

1 release file

0.1.106

1 release file

0.1.104

1 release file

0.1.103

1 release file

0.1.102

1 release file

0.1.99

1 release file

0.1.98

1 release file

0.1.97

1 release file

0.1.96

1 release file

0.1.95

1 release file

0.1.94

1 release file

0.1.93

1 release file

0.1.92

1 release file

0.1.91

1 release file

0.1.90

1 release file

0.1.89

1 release file

0.1.88

1 release file

0.1.87

1 release file

0.1.86

1 release file

0.1.85

1 release file

0.1.84

1 release file

0.1.83

1 release file

0.1.82

1 release file

0.1.81

1 release file

0.1.80

1 release file

0.1.79

1 release file

0.1.78

1 release file

0.1.77

1 release file

0.1.76

1 release file

0.1.75

1 release file

0.1.74

1 release file

0.1.73

1 release file

0.1.72

1 release file

0.1.71

1 release file

0.1.70

1 release file

0.1.69

1 release file

0.1.68

1 release file

0.1.67

1 release file

0.1.66

1 release file

0.1.65

1 release file

0.1.64

1 release file

0.1.63

1 release file

0.1.62

1 release file

0.1.61

1 release file

0.1.60

1 release file

0.1.59

1 release file

0.1.58

1 release file

0.1.57

1 release file

0.1.56

1 release file

0.1.55

1 release file

0.1.54

1 release file

0.1.53

1 release file

0.1.52

1 release file

0.1.51

1 release file

0.1.50

1 release file

0.1.49

1 release file

0.1.48

1 release file

0.1.47

1 release file

0.1.46

1 release file

0.1.45

1 release file

0.1.44

1 release file

0.1.43

1 release file

0.1.42

1 release file

0.1.41

1 release file

0.1.40

1 release file

0.1.39

1 release file

0.1.38

1 release file

0.1.37

1 release file

0.1.36

1 release file

0.1.35

1 release file

0.1.34

1 release file

0.1.33

1 release file

0.1.32

1 release file

0.1.31

1 release file

0.1.30

1 release file

0.1.29

1 release file

0.1.28

1 release file

0.1.27

1 release file

0.1.26

1 release file

0.1.25

1 release file

0.1.24

1 release file

0.1.23

1 release file

0.1.22

1 release file

0.1.21

1 release file

0.1.20

1 release file

0.1.19

1 release file

0.1.18

1 release file

0.1.17

1 release file

0.1.16

1 release file

0.1.15

1 release file

0.1.14

1 release file

0.1.13

1 release file

0.1.12

2 release files

0.1.9

2 release files

0.1.8

2 release files

0.1.7

2 release files

0.1.6

1 release file

0.1.5

2 release files

0.1.4

2 release files

0.1.3

2 release files

0.1.2

2 release files

0.1.1

1 release file

0.1.0

1 release file

0.0.11

2 release files

0.0.9

2 release files

0.0.8

2 release files

0.0.7

2 release files

0.0.6

1 release file

0.0.5

1 release file

0.0.2

2 release files

0.0.1

2 release files

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page