idict 0.211018.1

Lazy dict with universally unique identifier for values

idict

A lazy dict with universally unique deterministic identifiers.

Latest release | Current code | API documentation

Overview

An idict is a dict with str keys.

See also

• laziness+identity (ldict)
• laziness+identity+persistence (cdict)

We consider that every value is generated by a process, starting from an empty idict. The process is a sequence of transformation steps done through the operator >>, which symbolizes the ordering of the steps. There are two types of steps:

• value insertion - represented by dict-like objects
• function application - represented by ordinary Python functions

Functions, idicts, and values have a deterministic UUID (called hosh - operable hash). Identifiers (hoshes) for idicts and values are predictable through the magic available here. An idict is completely defined by its key-value pairs so that it can be converted from/to a built-in dict.

Creating an idict is not different from creating an ordinary dict. Optionally it can be created through the >> operator used after empty or Ø (usually AltGr+Shift+o in most keyboards). The resulting idict always contains two extra entries id and ids:

Function application is done in the same way. The parameter names define the input fields, while the keys in the returned dict define the output fields:

After evaluated, the value will not be calculated again:

Functions can accept parameters:

Installation

...as a standalone lib

# Set up a virtualenv. 
python3 -m venv venv
source venv/bin/activate

# Install from PyPI...
pip install --upgrade pip
pip install -U idict

# ...or, install from updated source code.
pip install git+https://github.com/davips/idict

...from source

git clone https://github.com/davips/idict
cd idict
poetry install

Examples

<>

Identity example

from idict import idict

a = idict(x=3)
print(a)
"""
{
    "x": 3,
    "id": "WB_e55a47230d67db81bcc1aecde8f1b950282cd",
    "ids": {
        "x": "WB_e55a47230d67db81bcc1aecde8f1b950282cd"
    }
}
"""

b = idict(y=5)
print(b)
"""
{
    "y": 5,
    "id": "0U_e2a86ff72e226d5365aea336044f7b4270977",
    "ids": {
        "y": "0U_e2a86ff72e226d5365aea336044f7b4270977"
    }
}
"""

print(a >> b)
"""
{
    "x": 3,
    "y": 5,
    "id": "Xt_a63010fa2b5b4c671270fbe8ec313568a8b35",
    "ids": {
        "x": "WB_e55a47230d67db81bcc1aecde8f1b950282cd",
        "y": "0U_e2a86ff72e226d5365aea336044f7b4270977"
    }
}
"""

Merging two idicts

from idict import idict

a = idict(x=3)
print(a)
"""
{
    "x": 3,
    "id": "WB_e55a47230d67db81bcc1aecde8f1b950282cd",
    "ids": {
        "x": "WB_e55a47230d67db81bcc1aecde8f1b950282cd"
    }
}
"""

b = idict(y=5)
print(b)
"""
{
    "y": 5,
    "id": "0U_e2a86ff72e226d5365aea336044f7b4270977",
    "ids": {
        "y": "0U_e2a86ff72e226d5365aea336044f7b4270977"
    }
}
"""

print(a >> b)
"""
{
    "x": 3,
    "y": 5,
    "id": "Xt_a63010fa2b5b4c671270fbe8ec313568a8b35",
    "ids": {
        "x": "WB_e55a47230d67db81bcc1aecde8f1b950282cd",
        "y": "0U_e2a86ff72e226d5365aea336044f7b4270977"
    }
}
"""

Lazily applying functions to idict

from idict import idict

a = idict(x=3)
print(a)
"""
{
    "x": 3,
    "id": "WB_e55a47230d67db81bcc1aecde8f1b950282cd",
    "ids": {
        "x": "WB_e55a47230d67db81bcc1aecde8f1b950282cd"
    }
}
"""

a = a >> idict(y=5) >> {"z": 7} >> (lambda x, y, z: {"r": x ** y // z})
print(a)
"""
{
    "r": "→(x y z)",
    "x": 3,
    "y": 5,
    "z": 7,
    "id": "H8DftZZ4nH6d67WSvYYxh-KsdBqp9MQBdvkLxU2o",
    "ids": {
        "r": "n57RGOgdv03kK4IqBkIf6oFrvgAp9MQBdvkLxU2o",
        "x": "WB_e55a47230d67db81bcc1aecde8f1b950282cd",
        "y": "0U_e2a86ff72e226d5365aea336044f7b4270977",
        "z": "nX_da0e3a184cdeb1caf8778e34d26f5fd4cc8c8"
    }
}
"""

print(a.r)
"""
34
"""

print(a)
"""
{
    "r": 34,
    "x": 3,
    "y": 5,
    "z": 7,
    "id": "H8DftZZ4nH6d67WSvYYxh-KsdBqp9MQBdvkLxU2o",
    "ids": {
        "r": "n57RGOgdv03kK4IqBkIf6oFrvgAp9MQBdvkLxU2o",
        "x": "WB_e55a47230d67db81bcc1aecde8f1b950282cd",
        "y": "0U_e2a86ff72e226d5365aea336044f7b4270977",
        "z": "nX_da0e3a184cdeb1caf8778e34d26f5fd4cc8c8"
    }
}
"""

Parameterized functions and sampling

from random import Random

from idict import Ø, let


# A function provide input fields and, optionally, parameters.
# For instance:
# 'a' is sampled from an arithmetic progression
# 'b' is sampled from a geometric progression
# Here, the syntax for default parameter values is borrowed with a new meaning.
def fun(x, y, a=[-100, -99, -98, ..., 100], b=[0.0001, 0.001, 0.01, ..., 100000000]):
    return {"z": a * x + b * y}


def simplefun(x, y):
    return {"z": x * y}


# Creating an empty ldict. Alternatively: d = ldict().
d = Ø >> {}
d.show(colored=False)
"""
{
    "id": "0000000000000000000000000000000000000000",
    "ids": {}
}
"""

# Putting some values. Alternatively: d = ldict(x=5, y=7).
d["x"] = 5
d["y"] = 7
d.show(colored=False)
"""
{
    "x": 5,
    "y": 7,
    "id": "mP_2d615fd34f97ac906e162c6fc6aedadc4d140",
    "ids": {
        "x": ".T_f0bb8da3062cc75365ae0446044f7b3270977",
        "y": "mX_dc5a686049ceb1caf8778e34d26f5fd4cc8c8"
    }
}
"""

# Parameter values are uniformly sampled.
d1 = d >> simplefun
d1.show(colored=False)
print(d1.z)
"""
{
    "z": "→(x y)",
    "x": 5,
    "y": 7,
    "id": "ZAasLu0lIEqhJyS1s8ML8WGeTnradBnjS7VNt6Mg",
    "ids": {
        "z": "iE6rHiYYwfwOBqa4Luh4XCd-myeadBnjS7VNt6Mg",
        "x": ".T_f0bb8da3062cc75365ae0446044f7b3270977",
        "y": "mX_dc5a686049ceb1caf8778e34d26f5fd4cc8c8"
    }
}
35
"""

d2 = d >> simplefun
d2.show(colored=False)
print(d2.z)
"""
{
    "z": "→(x y)",
    "x": 5,
    "y": 7,
    "id": "ZAasLu0lIEqhJyS1s8ML8WGeTnradBnjS7VNt6Mg",
    "ids": {
        "z": "iE6rHiYYwfwOBqa4Luh4XCd-myeadBnjS7VNt6Mg",
        "x": ".T_f0bb8da3062cc75365ae0446044f7b3270977",
        "y": "mX_dc5a686049ceb1caf8778e34d26f5fd4cc8c8"
    }
}
35
"""

# Parameter values can also be manually set.
e = d >> let(fun, a=5, b=10)
print(e.z)
"""
95
"""

# Not all parameters need to be set.
e = d >> let(simplefun, a=5)
print(e.z)
"""
35
"""

# Each run will be a different sample for the missing parameters.
e = e >> let(simplefun, a=5)
print(e.z)
"""
35
"""

# We can define the initial state of the random sampler.
# It will be in effect from its location place onwards in the expression.
e = d >> Random(0) >> let(fun, a=5)
print(e.z)
"""
725.0
"""

# All runs will yield the same result,
# if starting from the same random number generator seed.
e = e >> Random(0) >> let(fun, a=[555, 777])
print("Let 'a' be a list:", e.z)
"""
Let 'a' be a list: 700003885.0
"""

# Reproducible different runs are achievable by using a single random number generator.
e = e >> Random(0) >> let(fun, a=[5, 25, 125, ..., 10000])
print("Let 'a' be a geometric progression:", e.z)
"""
Let 'a' be a geometric progression: 700003125.0
"""

rnd = Random(0)
e = d >> rnd >> let(fun, a=5)
print(e.z)
e = d >> rnd >> let(fun, a=5)  # Alternative syntax.
print(e.z)
"""
725.0
700000025.0
"""

Composition of sets of functions

from random import Random

from idict import Ø


# A multistep process can be defined without applying its functions


def g(x, y, a=[1, 2, 3, ..., 10], b=[0.00001, 0.0001, 0.001, ..., 100000]):
    return {"z": a * x + b * y}


def h(z, c=[1, 2, 3]):
    return {"z": c * z}


# In the 'idict' framework 'data is function',
# so the alias Ø represents the 'empty data object' and the 'reflexive function' at the same time.
# In other words: 'inserting nothing' has the same effect as 'doing nothing'.
fun = Ø >> g >> h  # 'empty' or 'Ø' enable the cartesian product of the subsequent sets of functions within the expression.
print(fun)
"""
«<function g at 0x7fbd9bebc8b0> × <function h at 0x7fbd9c1c3820>»
"""

# Before a function is applied to a dict-like, the function free parameters remain unsampled.
# The result is an ordered set of composite functions.
d = {"x": 5, "y": 7} >> (Random(0) >> fun)
print(d)
"""
{
    "z": "→(c z→(a b x y))",
    "x": 5,
    "y": 7,
    "id": "fxUC9sbaX2rNuWEutGTJHWKMV5Af0h9G8FLRPWeq",
    "ids": {
        "z": "o5r8PbsxYejqtbjdN0p22yhwpgDf0h9G8FLRPWeq",
        "x": ".T_f0bb8da3062cc75365ae0446044f7b3270977",
        "y": "mX_dc5a686049ceb1caf8778e34d26f5fd4cc8c8"
    }
}
"""

print(d.z)
"""
105.0
"""

d = {"x": 5, "y": 7} >> (Random(0) >> fun)
print(d.z)
"""
105.0
"""

# Reproducible different runs by passing a stateful random number generator.
rnd = Random(0)
e = d >> rnd >> fun
print(e.z)
"""
105.0
"""

e = d >> rnd >> fun
print(e.z)
"""
14050.0
"""

# Repeating the same results.
rnd = Random(0)
e = d >> rnd >> fun
print(e.z)
"""
105.0
"""

e = d >> rnd >> fun
print(e.z)
"""
14050.0
"""

Concept

An idict is like a common Python dict, with extra functionality and lazy. It is a mapping between string keys, called fields, and any serializable (pickable) object. Each idict has two extra entries: id (identifier) and ids (value identifiers).

A custom 40-digit unique identifier (see GaROUPa) can be provided as an attribute for each function. Value objects can have custom identifiers as well, if provided whithin the entry ids.

Otherwise, identifiers for functions and values will be calculated through blake3 hashing of their content. For functions, the bytecode is used as content. For this reason, such functions should be simple, with minimal external dependencies or with their import statements inside the function body. This decreases the odds of using two functions with identical local code (and, therefore, identical identifiers) performing different calculations.

Grants

This work was supported by Fapesp under supervision of Prof. André C. P. L. F. de Carvalho at CEPID-CeMEAI (Grants 2013/07375-0 – 2019/01735-0) until 2021-03-31.