memory-graph 0.1.20

Draw a graph of your data to see the structure of its references.

Graph your Memory

Does your Python code have a bug, is it behaving differently from what you expect? The problem could be a misunderstanding of the Python data model, and the first step to the solution could be drawing your data as a graph using memory_graph.show( your_data ), an example:

import memory_graph

data = [ (1, 2), [3, 4], {5:'five', 6:'six'} ]
memory_graph.show( data, block=True )

This shows a graph with the starting point of your 'data' drawn with thick lines, the program blocks until the ENTER key is pressed.

Alternatively render the graph to an output file of your choosing using for example:

memory_graph.render( data, "my_graph.png")

Installation

Install memory_graph using pip:

pip install memory-graph

Additionally Graphviz needs to be installed.

Python Data Model

The Python Data Model makes a distiction between mutable and immutable types:

• mutable: bool, int, float, complex, str, tuple, bytes, frozenset
• immutable: list, dict, set, user-defined classes, all other types

immutable type

In the code below variable a and b both reference the same int value 10. An int is an immutable type and therefore when we change variable a its value can not be mutated, and thus a copy is made and a and b reference a different value afterwards.

import memory_graph
memory_graph.rewrite_to_node.reduce_reference_children.remove("int") # shows references to 'int'

a = 10
b = a
memory_graph.render(locals(), 'immutable1.png')
a += 1
memory_graph.render(locals(), 'immutable2.png')
print(f'a: {a} b: {b}')

mutable type

With mutable types the result is different. In the code below variable a and b both reference the same list value [4, 3, 2]. A list is a mutable type and therefore when we change variable a its value can be mutated in place and a and b both reference the same new value afterwards.

import memory_graph

a = [4, 3, 2]
b = a
memory_graph.render(locals(), 'mutable1.png')
a.append(1)
memory_graph.render(locals(), 'mutable2.png')

Python makes this distiction between mutable and immutable types because a value of a mutable type generally could be large and therefore it would be slow to make a copy each time you change it. On the other hand, a value of an immutable type generally is small and therefore fast to copy.

copying

Python offers three different "copy" options that we will demonstrate using a nested list (list of lists):

import memory_graph
import copy

a = [ [1, 2], ['a', 'b'] ] # a nested list

# three different ways to make a "copy" of 'a':
c1 = a
c2 = copy.copy(a) # equivalent:  a.copy() a[:]
c3 = copy.deepcopy(a)

memory_graph.render(locals(), 'copies.png')

• c1 is an assignment, all the data is shared.
• c2 is a shallow copy, only the data referenced by the first reference is copied and the underlying data is shared
• c3 is a deep copy, all the data is copied

custom copy method

For a class you can write your own custom copy() method in case neither of these three "copy" options does what you want. For example the copy() method of My_Class in the code below copies its numbers but shares it letters between different objects.

import memory_graph
import copy

class My_Class:

    def __init__(self):
        self.numbers = [1, 2]
        self.letters = ['a', 'b']

    def copy(self): # custom copy method copies the numbers but shares the letters
        c = copy.copy(self)
        c.numbers = copy.copy(self.numbers)
        return c

a = My_Class()
b = a.copy()

memory_graph.render(locals(), 'copy_method.png')

Graph all Local Variables

Often it is useful to graph all the local variables using:

memory_graph.show( locals(), block=True )

So much so that function d() is available as alias for easier debugging. Additionally it logs all locals by printing them which allows for comparing them over time. For example:

from memory_graph import d

my_squares = []
my_squares_ref = my_squares
for i in range(5):
    my_squares.append(i**2)
    d()                                    # 'd' for debug, logs and graphs all local variables and blocks
my_squares_copy = my_squares.copy()
d(block=False)                             # debug without blocking
d(log=False,block=False)                   # debug without logging and blocking

import memory_graph
memory_graph.log_file=open("log.txt","w")  # now log to file instead of screen (sys.stdout)
d(graph=False)                             # debug without showing the graph

Which in the end results in:

my_squares: [0, 1, 4, 9, 16]
my_squares_ref: [0, 1, 4, 9, 16]
i: 4
my_squares_copy: [0, 1, 4, 9, 16]

Notice that in the graph it is clear that my_squares and my_squares_ref share their data while my_squares_copy has its own copy. This can not be observed in the log and shows the benefit of the graph.

Alternatively debug by setting this expression as a 'watch' in a debugger tool and open the output file:

memory_graph.render( locals(), "my_debug_graph.pdf" )

Larger Example

This larger example shows objects that share a class (static) variable and also shows we can handle recursive references.

my_list = [10, 20, 10]

class My_Class:
    my_class_var = 20 # class variable: shared by different objects
    
    def __init__(self):
        self.var1 = "foo"
        self.var2 = "bar"
        self.var3 = 20

obj1 = My_Class()
obj2 = My_Class()

data=[my_list, my_list, obj1, obj2]

my_list.append(data) # recursive reference

import memory_graph
memory_graph.show( locals() )

Config

Different aspects of memory_graph can be configured.

Config Visualization, graphviz_nodes

Configure how the nodes of the graph are visualized with:

• memory_graph.graphviz_nodes.linear_layout_vertical : bool
  • if False, linear node layout is horizontal
• memory_graph.graphviz_nodes.linear_any_ref_layout_vertical : bool
  • if False, linear node layout is horizontal if any of its elements is a refence
• memory_graph.graphviz_nodes.linear_all_ref_layout_vertical : bool
  • if False, linear node layout is horizontal if all elements are reference
• memory_graph.graphviz_nodes.key_value_layout_vertical : bool
  • if False, key_value node layout is horizontal
• memory_graph.graphviz_nodes.key_value_any_ref_layout_vertical : bool
  • if False, key_value node layout is horizontal if any of its elements is a refence
• memory_graph.graphviz_nodes.key_value_all_ref_layout_vertical : bool
  • if False, key_value node layout is horizontal if all elements are reference
• memory_graph.graphviz_nodes.padding : int
  • the padding in nodes
• memory_graph.graphviz_nodes.padding : int
  • the spacing in nodes
• memory_graph.graphviz_nodes.join_references_count : int
  • minimum number of reference we join together
• memory_graph.graphviz_nodes.join_circle_size : string
  • size of the join circle
• memory_graph.graphviz_nodes.join_circle_minlen : string
  • extra space for references above a join circle
• memory_graph.graphviz_nodes.max_string_length : int
  • maximum string length where the string is cut off
• memory_graph.graphviz_nodes.category_to_color_map : dict
  • mapping van type/caterogries to node colors
• memory_graph.graphviz_nodes.uncategorized_color : dict
  • color for unkown types/categories
• memory_graph.graphviz_nodes.graph_attr : dict
  • allows to set various graphviz graph attributes
• memory_graph.graphviz_nodes.node_attr : dict
  • allows to set various graphviz node attributes
• memory_graph.graphviz_nodes.edge_attr : dict
  • allows to set various graphviz edges attributes

See for color names: graphviz colors

To configure more about the visualization use:

digraph = memory_graph.create_graph( locals() )

and see the graphviz api to render it in many different ways.

Config Graph Structure, rewrite_to_node

Configure the structure of the nodes in the graph with:

• memory_graph.rewrite_to_node.reduce_reference_parents : set
  • the node types/categories for which we remove the reference to children
• memory_graph.rewrite_to_node.reduce_reference_children : bool
  • the node types/categories for which we remove the reference from parents

Config Node Creation, rewrite

Configure what nodes are created based on reading the given data structure:

• memory_graph.rewrite.ignore_types : dict
  • all types that we ignore, these will not be in the graph
• memory_graph.rewrite.singular_types : set
  • all types rewritten to node as singular values (bool, int, float, ...)
• memory_graph.rewrite.linear_types : set
  • all types rewritten to node as linear values (tuple, list, set, ...)
• memory_graph.rewrite.dict_types : set
  • all types rewritten to node as dictionary values (dict, mappingproxy)
• memory_graph.rewrite.dict_ignore_dunder_keys : bool
  • determines if we ignore dunder keys ('__example') in dict_types
• memory_graph.rewrite.custom_accessor_functions : dict
  • custom accessor functions to define how to read various data types

Config Examples

With configuration:

memory_graph.graphviz_nodes.linear_layout_vertical = False           # draw lists,tuples,sets,... horizontally
memory_graph.graphviz_nodes.category_to_color_map['list'] = 'yellow' # change color of 'list' type
memory_graph.graphviz_nodes.spacing=15                               # more spacing in each node
memory_graph.graphviz_nodes.graph_attr['ranksep']='1.2'              # more vertical separation
memory_graph.graphviz_nodes.graph_attr['nodesep']='1.2'              # more horizontal separation
memory_graph.rewrite_to_node.reduce_reference_children.remove("int") # draw references to 'int' type

the last example looks like:

Custom Accessor Functions

For any type a custom accessor function can be introduced. For example Panda DataFrames and Series are not visualized correctly by default. This can be fixed by adding custom accessor functions:

import pandas as pd

data = {'Name':['Tom', 'Anna', 'Steve', 'Lisa'],
        'Age':[28,34,29,42],
        'Length':[1.70,1.66,1.82,1.73] }
df = pd.DataFrame(data)

import memory_graph
memory_graph.rewrite.custom_accessor_functions[pd.DataFrame] = lambda d: list(d.items())
memory_graph.rewrite.custom_accessor_functions[pd.Series] = lambda d: list(d.items())
memory_graph.rewrite_to_node.reduce_reference_parents.add("DataFrame")
memory_graph.rewrite_to_node.reduce_reference_parents.add("Series")
memory_graph.graphviz_nodes.category_to_color_map['Series'] = 'lightskyblue'
memory_graph.show( locals() )

which results in:

Troubleshooting

• When graph edges overlap it can be hard to distinguish them. Using an interactive graphviz viewer, such as xdot, on a '*.gv' output file will help.

Author

Bas Terwijn

Inspiration

Inspired by PythonTutor.