mindtrace-registry 0.12.0

Registry functionality for Mindtrace

Registry Module

The Registry module provides a distributed, versioned object storage system with support for multiple backends. It enables storing, versioning, and retrieving objects with automatic serialization and lock-free concurrency for objects.

Features

• Multi-Backend Support: Local filesystem, S3-compatible (MinIO, AWS S3) and Google Cloud Storage
• Lock-Free Concurrency: UUID-based MVCC ensures safe concurrent reads and writes without distributed locks
• Versioning: Automatic version management with semantic versioning support
• Caching: Local cache for remote backends with configurable staleness checks
• Materializers: Pluggable serialization system for different object types
• Batch Operations: All backend operations support batch mode for efficient bulk access
• Dict-Like Interface: registry["name"] = obj, obj = registry["name"], del registry["name"]

Quick Start

from mindtrace.registry import Registry

# Create a registry (uses local backend by default)
registry = Registry()

# Save objects
registry.save("my:model", trained_model)
registry.save("my:data", dataset, version="1.0.0")

# Load objects
model = registry.load("my:model")
data = registry.load("my:data", version="1.0.0")

# Dict-like access
registry["my:config"] = config_dict
config = registry["my:config"]

# Check existence
exists = registry.has_object("my:model", "1.0.0")  # -> bool

# Get metadata
info = registry.info("my:model", "1.0.0")  # -> dict

# List objects and versions
print(registry.list_objects())
print(registry.list_versions("my:model"))

Backend Configuration

Local Backend

The local backend stores objects on the filesystem and is the default option.

from mindtrace.registry import Registry, LocalRegistryBackend

# Default local registry
registry = Registry()

# Custom local registry
local_backend = LocalRegistryBackend(uri="/path/to/registry")
registry = Registry(backend=local_backend)

S3-Compatible Backend (MinIO, AWS S3)

The S3 backend provides distributed storage for any S3-compatible service.

from mindtrace.registry import Registry, MinioRegistryBackend

# MinIO / S3-compatible registry
s3_backend = S3RegistryBackend(
    endpoint="localhost:9000",
    access_key="minioadmin",
    secret_key="minioadmin",
    bucket="minio-registry",
    secure=False,
)
registry = Registry(backend=minio_backend)

GCP Backend

The GCP backend uses Google Cloud Storage for distributed object storage.

from mindtrace.registry import Registry, GCPRegistryBackend

gcp_backend = GCPRegistryBackend(
    uri="gs://my-registry-bucket",
    project_id="my-project",
    bucket_name="my-registry-bucket",
    credentials_path="/path/to/service-account.json",
)
registry = Registry(backend=gcp_backend)

Concurrency Model

Cloud backends (GCP, S3) use lock-free MVCC (Multi-Version Concurrency Control):

• Each push writes artifacts to a unique UUID folder: objects/{name}/{version}/{uuid}/
• Metadata write is the atomic "commit point" — it references the active UUID
• For immutable registries: first-write-wins via conditional creation (generation_match=0 on GCS, IfNoneMatch='*' on S3)
• For mutable registries: last metadata write wins; orphaned UUID folders are cleaned up by the janitor

Locks are only used for register_materializer, which performs a read-modify-write on registry metadata.

Caching

When using a remote backend, the Registry maintains a local cache (enabled by default):

# Caching is on by default for remote backends
registry = Registry(backend=gcp_backend, use_cache=True)

# Keep at most 1024 concrete object versions in the local cache by default.
# Set cache_max_entries=None to disable automatic LRU pruning.
registry = Registry(
    backend=gcp_backend,
    use_cache=True,
    cache_max_entries=1024,
)

# Control verification level on load
obj = registry.load("my:model", verify="none")       # Trust cache, fastest
obj = registry.load("my:model", verify="integrity")   # Verify hash (default)
obj = registry.load("my:model", verify="full")         # Hash + staleness check

# Clear cache manually
registry.clear_cache()

Verification levels (VerifyLevel):

• "none": Trust cache completely. Fastest.
• "integrity": Verify loaded artifacts match the hash in metadata. Default.
• "full": Integrity check + compare cache hash against remote. Detects stale cache entries.

LRU pruning: remote registry caches retain at most cache_max_entries concrete object versions, defaulting to 1024. Cache hits update the cached object metadata file timestamp with os.utime(...), so recency is visible across processes sharing the same cache directory. Cache maintenance is amortized: when cache writes push the cache above cache_max_entries, least-recently-used entries are removed down to cache_max_entries - cache_prune_buffer. The default prune buffer is min(max(cache_max_entries // 4, 1), 1024). Set cache_max_entries=None to keep the cache unbounded.

Version Management

# Versioned registry (auto-increments versions)
registry = Registry(version_objects=True)
registry.save("model", v1)                    # version = "1"
registry.save("model", v2)                    # version = "2"
registry.save("model", v3, version="2.1")     # version = "2.1"

# Load specific or latest version
model = registry.load("model", version="2.1")
latest = registry.load("model", version="latest")

# Unversioned registry (single version per name, default)
registry = Registry(version_objects=False)

Conflict Handling

Control behavior when saving to an existing version (OnConflict):

# Skip (default): raises RegistryVersionConflict for single ops
registry.save("model", obj, version="1.0.0", on_conflict="skip")

# Overwrite: replaces existing version (requires mutable=True)
registry = Registry(mutable=True)
registry.save("model", obj, version="1.0.0", on_conflict="overwrite")

Custom Materializers

A materializer is any class that exposes a uri attribute, a save(data) method, and a load(data_type) method. The contract is published as the runtime-checkable mindtrace.registry.Materializer Protocol — your class does not need to inherit from anything in mindtrace to be recognized. For convenience, you can inherit from BaseMaterializer (minimal) or Archiver (adds Mindtrace logging).

from typing import Any, Type

from mindtrace.registry import BaseMaterializer, Registry


class MyMaterializer(BaseMaterializer):
    def save(self, data: Any) -> None:
        ...  # write to self.uri

    def load(self, data_type: Type[Any]) -> Any:
        ...  # read from self.uri


registry = Registry()

# Register a materializer for a custom class (by type or by fully-qualified string).
registry.register_materializer("my_module.MyClass", "my_module.MyMaterializer")

# Save with an explicit materializer override.
registry.save("custom:obj", my_object, materializer=MyMaterializer)

Built-in materializers cover scalars, container types, bytes, pydantic.BaseModel, pathlib.Path, numpy arrays, PIL images, PyTorch modules/dataloaders, and HuggingFace datasets. ML framework archivers for HuggingFace, ONNX, Ultralytics and timm models live in mindtrace.models.archivers.

Metadata and Information

# Get info for a specific object version
info = registry.info("my:model", "1.0.0")

# Get info for all versions of an object
info = registry.info("my:model")

# Get info for all objects
info = registry.info()

# Check existence
exists = registry.has_object("my:model", "1.0.0")  # -> bool

Error Handling

from mindtrace.registry.core.exceptions import (
    RegistryObjectNotFound,
    RegistryVersionConflict,
)

try:
    model = registry.load("nonexistent:model")
except RegistryObjectNotFound as e:
    print(f"Object not found: {e}")

try:
    registry.save("model", obj, version="1.0.0")  # already exists
except RegistryVersionConflict as e:
    print(f"Version conflict: {e}")

Batch Operations

The Registry facade provides clean single-object methods. For batch operations, pass lists:

# Batch save
result = registry.save(
    ["model:a", "model:b"],
    [obj_a, obj_b],
    version=["1.0.0", "1.0.0"],
)
# result is a BatchResult with .results, .errors, .succeeded, .failed

# Batch load
result = registry.load(["model:a", "model:b"], version=["1.0.0", "1.0.0"])

Dict-Like API

The Registry also supports simple dict-like access for common operations:

from mindtrace.registry import Registry

registry = Registry()

# Save
registry["my:config"] = {"threshold": 0.8}

# Load
config = registry["my:config"]
print(config)

# Delete
del registry["my:config"]

This is convenient for unversioned or latest-version style access when you want a compact interface.

Backend Comparison

Feature	Local	S3 / MinIO	GCP
Storage	Filesystem	S3-compatible	Google Cloud Storage
Concurrency	File locks	Lock-free MVCC	Lock-free MVCC
Caching	N/A	Local cache	Local cache
Batch Ops	Sequential	Parallel (ThreadPoolExecutor)	Parallel (ThreadPoolExecutor)

Troubleshooting

Common Issues

1. Permission Errors: Verify credentials and bucket access
2. Network Issues: Check connectivity to remote backends

Debug Logging

import logging
logging.basicConfig(level=logging.DEBUG)

registry = Registry()
# Operations will now show detailed logs

Store (Multi-Registry Facade)

The Store class composes multiple Registry instances behind a single API. Where a Registry maps to exactly one backend, a Store lets you read and write across multiple physical stores with deterministic routing.

Mounts

A Store organises registries as named mounts. Every Store always has a temp mount (backed by a fresh temporary directory) and a configurable default_mount that controls where unqualified writes go.

from mindtrace.registry import Registry, Store

# A bare Store — just the temp mount
store = Store()

# Add named mounts
store.add_mount("models", Registry(backend=gcp_backend))
store.add_mount("datasets", Registry(backend=s3_backend), read_only=True)

# Change the default write target
store.set_default_mount("models")

Key Format

Keys can be qualified (routed to a specific mount) or unqualified (routed automatically):

# Qualified — targets the "models" mount explicitly
store.save("models/my_model", obj)
model = store.load("models/my_model@1.0.0")

# Unqualified — writes go to default_mount, reads discover across all mounts
store.save("my_model", obj)          # -> saves to default_mount
model = store.load("my_model")       # -> searches all mounts

Read and Write Routing

• Writes: Qualified writes target the specified mount. Unqualified writes go to default_mount.
• Reads: Qualified reads target the specified mount. Unqualified reads discover across all mounts — if the object exists in exactly one mount it loads; if found in multiple mounts a StoreAmbiguousObjectError is raised.

Default Mount Behaviour

• default_mount always points to a configured mount (initially temp).
• Removing the current default mount resets it back to temp.
• The temp mount cannot be removed.

Store Errors

In addition to the standard Registry exceptions, Store introduces:

• StoreLocationNotFound — unknown mount
• StoreKeyFormatError — invalid key format
• StoreAmbiguousObjectError — unqualified load matched multiple mounts
• PermissionError — write to a read-only mount

Examples

See these examples and related docs in the repo for more end-to-end reference:

• Registry quick-start and backend examples
• Store multi-registry facade section

Testing

If you are working in the full Mindtrace repo, run tests for this module specifically:

# Run the registry test suite
ds test: registry

# Run only unit tests for registry
ds test: --unit registry

If you need a fresh checkout first:

git clone https://github.com/Mindtrace/mindtrace.git && cd mindtrace
uv sync --dev --all-extras

Practical Notes and Caveats

• Remote backends typically benefit from cache usage, but cache verification level affects correctness/performance trade-offs.
• Versioned and unversioned registries behave differently; choose the mode that matches your object lifecycle.
• Overwrite behavior depends on registry mutability and conflict policy.
• Batch operations are convenient, but partial failures should be handled explicitly through the returned batch result.
• The dict-like API is compact, but explicit save() / load() calls are often clearer when versioning behavior matters.
• Store reads with unqualified keys can become ambiguous if the same object exists in multiple mounts.