ldict 2.210908.5

Lazy dict with universally unique identifier for values

ldict

Uniquely identified lazy dict.

Latest version as a package

Current code

Overview

We consider that every data is generated by a process, starting from empty. The process is a sequence of transformation steps that can be of two types: value insertion and function application. Value insertion is done using dict-like objects as shown below. The operator >> concatenate the steps chronologically.

Function application is done in the same way. The parameter names define the input fields, while the keys in the returned dict define the output fields:

Similarly, for anonymous functions:

Finally, the result is only evaluated at request:

Installation

...as a standalone lib

# Set up a virtualenv. 
python3 -m venv venv
source venv/bin/activate

# Install from PyPI...
pip install --upgrade pip
pip install -U ldict

# ...or, install from updated source code.
pip install git+https://github.com/davips/ldict

...from source

git clone https://github.com/davips/ldict
cd ldict
poetry install

Examples

Merging two ldicts

from ldict import ldict

a = ldict(x=3)
print(a)
"""
{
    "id": "kr_4aee5c3bcac2c478be9901d57fd1ef8a9d002",
    "ids": "kr_4aee5c3bcac2c478be9901d57fd1ef8a9d002",
    "x": 3
}
"""

b = ldict(y=5)
print(b)
"""
{
    "id": "Uz_0af6d78f77734fad67e6de7cdba3ea368aae4",
    "ids": "Uz_0af6d78f77734fad67e6de7cdba3ea368aae4",
    "y": 5
}
"""

print(a >> b)
"""
{
    "id": "c._2b0434ca422114262680df425b85cac028be6",
    "ids": "kr_4aee5c3bcac2c478be9901d57fd1ef8a9d002 Uz_0af6d78f77734fad67e6de7cdba3ea368aae4",
    "x": 3,
    "y": 5
}
"""

Lazily applying functions to ldict

from ldict import ldict

a = ldict(x=3)
print(a)
"""
{
    "id": "kr_4aee5c3bcac2c478be9901d57fd1ef8a9d002",
    "ids": "kr_4aee5c3bcac2c478be9901d57fd1ef8a9d002",
    "x": 3
}
"""

a = a >> ldict(y=5) >> {"z": 7} >> (lambda x, y, z: {"r": x ** y // z})
print(a)
"""
{
    "id": "8jopGVdtSEyCk1NSKcrEF-Lfv8up9MQBdvkLxU2o",
    "ids": "J3tsy4vUXPELySBicaAy-h-UK7Dp9MQBdvkLxU2o... +2 ...Ss_7dff0a161ba7462725cac7dcee71b67669f69",
    "r": "→(x y z)",
    "x": 3,
    "y": 5,
    "z": 7
}
"""

print(a.r)
"""
34
"""

print(a)
"""
{
    "id": "8jopGVdtSEyCk1NSKcrEF-Lfv8up9MQBdvkLxU2o",
    "ids": "J3tsy4vUXPELySBicaAy-h-UK7Dp9MQBdvkLxU2o... +2 ...Ss_7dff0a161ba7462725cac7dcee71b67669f69",
    "r": 34,
    "x": 3,
    "y": 5,
    "z": 7
}
"""

Concept

A ldict is like a common Python dict, with extra funtionality and lazy. It is a mapping between string keys, called fields, and any serializable object. The ldict id (identifier) and the field ids are also part of the mapping.

The user can provide a unique identifier (hosh) for each function or value object. Otherwise, they will be calculated through blake3 hashing of the content of data or bytecode of function. For this reason, such functions should be simple, i.e., with minimal external dependencies, to avoid the unfortunate situation where two functions with identical local code actually perform different calculations through calls to external code that implement different algorithms with the same name.

Grants

This work was partially supported by Fapesp under supervision of Prof. André C. P. L. F. de Carvalho at CEPID-CeMEAI (Grants 2013/07375-0 – 2019/01735-0).