hdict { A unique data structure }

Website | Latest Release | Current Code | API Documentation

Please see the website for more detailed usage information.

Overview

Shortly: A data structure based on a novel identification paradigm useful for frictionless computing, experiments, distributed data, among others.

Formally: Hosh-based cacheable lazy dict with predictable/deterministic universally (probabilistically guaranteed) unique identifiers.

Context

• hosh: identification engine at the core of hdict
• previous projects: hoshmap, idict, ldict, garoupa

The concept

hdict is like an ordinary dict with str keys. Each entry, called field, and the hdict itself, are identified by 40-digit hashes (see hosh). A hdict object (say d) provides two extra entries: _id (hdict identifier) and _ids (field identifiers), accessible through square brackets or through the shortcuts d.id and d.ids.

hdict creation

from hdict import hdict

# From named arguments.
d = hdict(x=5, y=7, z=10)

# From a dict object.
d = hdict({"x": 5, "y": 7, "z": 10})

# From an empty 'hdict' object.
d = hdict() >> {"x": 5} >> {"y": 7, "z": 10}

# All three options have the same result.
d.show(colored=False)
"""
{
    x: 5,
    y: 7,
    z: 10,
    _id: "BN-3Q3Z.2Q.9nsbIYnOI75HT7xhgjvF6wErwBPTn",
    _ids: {
        x: "ecvgo-CBPi7wRWIxNzuo1HgHQCbdvR058xi6zmr2",
        y: "eJCW9jGsdZTD6-AD9opKwjPIOWZ4R.T0CG2kdyzf",
        z: "u-Yykj2nDtKaUMGzfqScX5Y14qC7eqJrO7lXrJ1m"
    }
}
"""

from hosh import setup

# For better integration within the documentation, we change the color theme.
setup(dark_theme=False)

d.show(colored=False)

"""
{
    x: 5,
    y: 7,
    z: 10,
    _id: "BN-3Q3Z.2Q.9nsbIYnOI75HT7xhgjvF6wErwBPTn",
    _ids: {
        x: "ecvgo-CBPi7wRWIxNzuo1HgHQCbdvR058xi6zmr2",
        y: "eJCW9jGsdZTD6-AD9opKwjPIOWZ4R.T0CG2kdyzf",
        z: "u-Yykj2nDtKaUMGzfqScX5Y14qC7eqJrO7lXrJ1m"
    }
}
"""

A field contains a value or a function application. A field pointing to an application is only evaluated on demand, i.e., lazily.

Please refer to our website for more examples.

Installation

...as a standalone lib

# Set up a virtualenv. 
python3 -m venv venv
source venv/bin/activate

# Install from PyPI...
pip install --upgrade pip
pip install -U hdict

# use the flag 'full' for extra functionality (recommended if working with persistence or some special values like DataFrames)
pip install -U hdict[full]

# ...or, install from updated source code.
pip install git+https://github.com/davips/hdict

...from source

git clone https://github.com/davips/hdict
cd hdict
poetry install

# use the flag 'full' for extra functionality (recommended if working with persistence or some special values like DataFrames)  
poetry install -E full

Examples

Please refer to our website for examples.

Development

Licensing

The initial license choosen is GPL. Please contact the developer for other licensing needs.

Versioning

The version is a has a meaningful calendar component (minor=yymmdd), but it is still compatible with semantic versioning. For instance, the version 0.230215.1 means major=0, minor=230215, micro/patch=1. Notes:

• While major=0, some compatibility breaking changes may occur.
• From major=1 onwards, compatibility breaking changes increment it, and update the minor version to reflect the release date.
• New (non breaking) features update only the minor version to reflect the release date.
• Bug fixes (including breaking ones) increment only the micro version.

Contributing

Collaboration

We have ongoing research applying this tool to machine learning and clinical academic experiments. Although, the scope of application is broad as it encompasses software development in general. Feel free to contact us if you are interested in the project/concept or have a suggestion/interesting problem to be solved.

Donation

Currently there are no established forms of donation. Expenses:

• Programming hours
• Support
• Custom features
• Domain name maintenance yearly costs

Acknowledgement

This work was based on a previous research supported by Fapesp under supervision of Prof. André C. P. L. F. de Carvalho at CEPID-CeMEAI (Grants 2013/07375-0 – 2019/01735-0) until 2021-03-31.

We would like to thank the providers of free tools that make this project feasible:

• github for hosting the code repository freely
• Oracle for hosting the website freely
• Developers of Mint, XFCE, and GNU/Linux for the operating system
• JetBrains for Intellij IDEA Community Edition
• Developers of Python packages listed in our project.toml