metatreedb 0.1.8

Metatree is a DBMS that uses the filesystem itself as a tree-structured database.

metatreedb

Metatree is a DBMS that uses the filesystem itself to organize and manage data in a tree-structured format.

In metadata.json in each tree node, you can manage information about child nodes, which can also be used for searching.

Features

• metadata-based index
• db-level concurrency control
• flexible filesystem support
  • local
  • HDFS
  • http
    • proper customization is required

Installation

pip install metatreedb

Quick Start

Here's an example of using Metatree as a model repository by setting up a database with (model_name, version,) as identifiers:

from metatree import Metatree

metatree = Metatree(
    "/tmp/my-model-repository",
    (
        "model",
        "version",
    ),
)

import uuid
import pickle

from pathlib import Path

for i in range(1, 4):
    awful_uuid = uuid.uuid4()
    trained = Path("/tmp/my-model-repository/trained.pkl")
    with open(trained, "wb") as f:
        pickle.dump(awful_uuid, f)
    metatree.put(f"my-awful-model/v{i}", trained)

This will create files and directories in your filesystem as shown:

❯ tree /tmp/my-model-repository
/tmp/my-model-repository
├── metadata.json
└── my-awful-model
    ├── metadata.json
    ├── v1
    │   ├── metadata.json
    │   └── trained.pkl
    ├── v2
    │   ├── metadata.json
    │   └── trained.pkl
    └── v3
        ├── metadata.json
        └── trained.pkl

To add metadata information, use find and update:

metatree.find("my-awful-model")
metatree.update(active="v2")
for i in range(1, 4):
    metatree.find(f"my-awful-model/v{i}").update(model_file="trained.pkl")

This will update the metadata.json files as follows:

❯ cat /tmp/my-model-repository/my-awful-model/metadata.json
{"children": ["v1", "v2", "v3"], "active": "v2"}

❯ cat /tmp/my-model-repository/my-awful-model/v*/metadata.json
{"model_file": "trained.pkl"}
{"model_file": "trained.pkl"}
{"model_file": "trained.pkl"}

You can use this search index to find files. By enclosing the keys from metadata.json within angle brackets <> and substituting them in location, you can perform searches as follows:

metatree.find("my-awful-model/<active>")
print(metatree.location)
# This returns `file:///tmp/my-model-repository/my-awful-model/v2

file = metatree.get("my-awful-model/<active>/<model_file>")
print(file)
# The given path translates to `my-awful-model/v2/trained.pkl`,
# and it returns generator object

with WebHDFS

To use WebHDFS, set the root path to the WebHDFS URL and provide a client with the necessary permissions:

from metatreedb import Metatree
from hdfs import InsecureClient

metatree = Metatree(
    "webhdfs://localhost:9870/tmp/my-model-repository",
    ("model", "version"),
    client=InsecureClient("http://localhost:9870", user="hadoop"),
)

with S3

To use S3, set the root path to the S3 bucket URL and provide a boto3 client with the necessary permissions:

from metatreedb import Metatree
import boto3

s3_client = boto3.client(
    "s3",
    region_name="us-east-1",
    endpoint_url="http://localhost:3000", # Tested with moto package.
    aws_access_key_id="your-access-key-id",
    aws_secret_access_key="your-secret-access-key",
)

metatree = Metatree(
    "s3://localhost:3000/tmp/my-model-repository",
    ("model", "version"),
    client=s3_client,
    s3_bucket="your-s3-bucket-name",
)