Henge

Henge is a Python package for building data storage and retrieval interfaces for arbitrary data. Henge is based on the idea of decomposable recursive unique identifiers (DRUIDs), which are hash-based unique identifiers for data derived from the data itself. For arbitrary data with any structure, Henge can mint unique DRUIDs to identify data, store the data in a key-value database of your choice, and provide lookup functions to retrieve the data in its original structure using its DRUID identifier.

Henge was intended as a building block for sequence collections, but is generic enough to use for any data type that needs content-derived identifiers with database lookup capability.

Install

pip install henge

Quick Start

Create a Henge object by providing a database and a data schema. The database can be a Python dict or backed by persistent storage. Data schemas are JSON-schema descriptions of data types, and can be hierarchical.

import henge

schemas = ["path/to/json_schema.yaml"]
h = henge.Henge(database={}, schemas=schemas)

Insert items into the henge. Upon insert, henge returns the DRUID (digest/checksum/unique identifier) for your object:

druid = h.insert({"name": "Pat", "age": 38}, item_type="person")

Retrieve the original object using the DRUID:

h.retrieve(druid)
# {'age': '38', 'name': 'Pat'}

Tutorial

For a comprehensive walkthrough covering basic types, arrays, nested objects, and advanced features, see the tutorial notebook.

What are DRUIDs?

DRUIDs are a special type of unique identifiers with two powerful properties:

• Decomposable: Identifiers in henge automatically retrieve structured data (tuples, arrays, objects). The structure is defined by a JSON schema, so henge can be used as a back-end for arbitrary data types.

• Recursive: Individual elements retrieved by henge can be tagged as recursive, meaning these attributes contain their own DRUIDs. Henge can recurse through these, allowing you to mint unique identifiers for arbitrary nested data structures.

A DRUID is ultimately the result of a digest operation (such as md5 or sha256) on some data. Because DRUIDs are computed deterministically from the item, they represent globally unique identifiers. If you insert the same item repeatedly, it will produce the same DRUID -- this is true across henges as long as they share a data schema.

Persisting Data

In-memory (default)

Use a Python dict as the database for testing or ephemeral use:

h = henge.Henge(database={}, schemas=schemas)

SQLite backend

For persistent storage with SQLite:

from sqlitedict import SqliteDict

mydict = SqliteDict('./my_db.sqlite', autocommit=True)
h = henge.Henge(mydict, schemas=schemas)

Requires: pip install sqlitedict

MongoDB backend

For production use with MongoDB:

1. Start MongoDB with Docker:

docker run --network="host" mongo

For persistent storage, mount a volume to /data/db:

docker run -it --network="host" -v /path/to/data:/data/db mongo

2. Connect henge to MongoDB:

import henge

h = henge.Henge(henge.connect_mongo(), schemas=schemas)

Requires: pip install pymongo mongodict