dr-store

Definitions · Terms · Contracts · Changelog · dr-serialize

dr-store provides domain-neutral storage primitives for immutable records and document artifacts:

• Content addressing identifies complete records by their declared schemas and SHA-256 hashes of their Canonical JSON Text under dr-serialize's frozen profile.
• Object Store provides immutable puts, verified reads, and atomic bindings from opaque caller-owned keys to object references.
• Storage backends supply the Object Store's atomic, append-only point and batch operations. MemoryBackend is process-local; SqliteBackend persists committed data for cross-process use; PostgresBackend shares committed data through a caller-owned asynchronous PostgreSQL pool.
• Record Cache memoizes records under opaque caller-owned keys. Reads return typed hits; absent, missing, or unverifiable stored values are misses, while invalid requested schemas and operational backend faults raise. Entries are never rebound, so callers invalidate by selecting a new key; derive_cache_key provides a canonical scheme using a versioned namespace and payload. Single and bulk methods share these per-key semantics; await SqliteRecordCache.open(path) is the managed persistent lifecycle.
• Canonical JSON document files publish and read one standalone, bounded canonical document in an existing directory through descriptor-pinned filesystem operations.
• Artifact bundles publish one terminal task, run, or result directory containing complete raw artifacts and a closed canonical manifest with caller-owned metadata.
• Document Directory delegates one bounded canonical Manifest to that file capability beside streamed binary Sidecars.

Installation

dr-store requires Python 3.12 or newer.

python -m pip install dr-store

PostgreSQL 16 through 18 installations use required asyncpg and an explicit, absent-only schema installation step. The caller creates and owns the pool; dr-store neither accepts a DSN nor closes the pool:

import asyncio
import os

import asyncpg

from dr_store import ObjectStore, PostgresBackend, install_postgres


async def main() -> None:
    pool = await asyncpg.create_pool(os.environ["DATABASE_URL"])
    try:
        await install_postgres(pool)
        backend = await PostgresBackend.open(pool)
        store = ObjectStore(backend)
        reference, _ = await store.put(
            "example.note.v1", {"title": "hello"}
        )
        assert await store.get(reference) == {"title": "hello"}
    finally:
        await pool.close()


asyncio.run(main())

install_postgres is a one-time deployment operation that creates the fixed dr_store namespace, its tables, and the exact dr-store-postgresql-v1 schema-format marker in one transaction on a UTF-8 database. Repeating installation is an error. await PostgresBackend.open(pool) validates that marker before returning a backend for the same awaited point and batch operations as the other backends; it acquires and releases connections without closing the pool. Opening never installs, alters, adopts, or upgrades storage.

Usage

import asyncio

from dr_store import MemoryBackend, ObjectStore


async def main() -> None:
    store = ObjectStore(MemoryBackend())
    reference, _ = await store.put("example.note.v1", {"title": "hello"})
    await store.bind("notes/latest", reference)

    assert await store.resolve("notes/latest") == reference
    assert await store.get(reference) == {"title": "hello"}


asyncio.run(main())

await SqliteRecordCache.open(path) is the paved persistent Record Cache. It returns only after its dedicated worker, connection, and schema are ready and closes those resources on normal or exceptional async context exit. When cleanup succeeds, an exception from the context body is not suppressed; cleanup failure raises SqliteRecordCacheCloseError:

import asyncio

from dr_store import CacheEntry, CacheHit, SqliteRecordCache, derive_cache_key


async def main() -> None:
    key = derive_cache_key("example.summary.v1", {"document": "note-42"})
    async with await SqliteRecordCache.open("records.sqlite3") as cache:
        winners = await cache.put_many(
            {
                key: CacheEntry(
                    schema="example.summary.v1",
                    record={"summary": "hello"},
                )
            }
        )
        assert winners[key].schema == "example.summary.v1"
        assert await cache.get_many(
            [key, "missing"], schema="example.summary.v1"
        ) == {
            key: CacheHit(record={"summary": "hello"}),
            "missing": None,
        }


asyncio.run(main())

CanonicalJsonFile publishes one standalone document in an existing directory. The caller must declare the maximum accepted canonical byte length:

from pathlib import Path

from dr_store import CanonicalJsonFile

artifact_directory = Path("artifacts")
artifact_directory.mkdir(exist_ok=True)
metadata = CanonicalJsonFile(
    artifact_directory,
    "metadata.json",
    max_bytes=1 << 20,
)
metadata.publish({"state": "complete"})
assert metadata.read() == {"state": "complete"}

ArtifactBundlePublication allocates one fresh terminal publication. Distinct artifact writers may run concurrently; the non-waiting publish call succeeds only after every admitted writer finalizes:

from pathlib import Path

from dr_store import (
    ArtifactBundlePublication,
    ArtifactBundleReader,
    BundleReadLimits,
    VerifyingArtifactReader,
)

root = Path("results")
root.mkdir(exist_ok=True)
bundle = ArtifactBundlePublication.allocate(root, prefix="task-42")

stdout = bundle.open_artifact("stdout.bin")
stdout.write(b"complete output\n")
descriptor = stdout.finalize()

bundle.publish(
    {
        "kind": "example.result.v1",
        "stdout_sha256": descriptor.sha256,
    }
)
assert (bundle.path / "manifest.json").is_file()

reader = ArtifactBundleReader(
    bundle.path,
    limits=BundleReadLimits(
        manifest_max_bytes=1 << 20,
        manifest_max_depth=64,
        max_artifacts=16,
        max_bytes_per_artifact=1 << 30,
        max_total_artifact_bytes=4 << 30,
    ),
)
manifest = reader.audit()
assert manifest.payload["kind"] == "example.result.v1"

captured_stdout = bytearray()

def consume_stdout(stream: VerifyingArtifactReader) -> None:
    while chunk := stream.read(1 << 16):
        captured_stdout.extend(chunk)


verified = reader.consume_and_verify_artifact("stdout.bin", consume_stdout)
assert verified.sha256 == descriptor.sha256
assert captured_stdout == b"complete output\n"

The bundle API is synchronous. Async applications offload a complete writer or publication, audit, or verified-consumption operation rather than running its hashing and filesystem I/O on an event-loop thread. One bundle is a task, run, or result publication—not a per-event record or packed execution-record backend. Callers own partitioned allocation roots and retention; sustained creation near 100,000 bundles per hour is outside the directory format's intended envelope.

Use the lower-level await SqliteBackend.open(path) when assembling an ObjectStore directly whose objects and bindings must persist across processes. Close it with await backend.aclose() or an async context. The rendered definitions, authoritative terms, and binding contracts describe the vocabulary, public-export mappings, and behavioral boundaries.

Content addressing

Content addressing validates each schema-qualified reference and derives its content hash through dr-serialize's canonical JSON profile. Its stable public shape is:

@dataclass(frozen=True, slots=True)
class ObjectReference:
    schema: str
    content_hash: str

    @classmethod
    def for_record(cls, schema: str, record: Jsonable) -> ObjectReference: ...
    def verify_record(self, record: Jsonable) -> None: ...

def compute_content_hash(record: Jsonable) -> str: ...
def is_content_hash(value: str) -> bool: ...

Object Store

The Object Store owns immutable record operations and opaque key bindings. Its statuses and store surface keep storage outcomes distinct from stored records:

class PutStatus(Enum):
    STORED = "stored"
    IDEMPOTENT = "idempotent"

class BindStatus(Enum):
    BOUND = "bound"
    IDEMPOTENT = "idempotent"

class ObjectStore:
    def __init__(self, backend: Backend) -> None: ...
    async def put(
        self, schema: str, record: Jsonable
    ) -> tuple[ObjectReference, PutStatus]: ...
    async def get(self, reference: ObjectReference) -> Jsonable: ...
    async def bind(
        self, key: str, reference: ObjectReference
    ) -> BindStatus: ...
    async def resolve(self, key: str) -> ObjectReference | None: ...

Storage backends

Storage backends implement one atomic protocol beneath the Object Store. Outcome objects carry the existing row when an append-only operation does not insert. Batch value objects carry prepared writes and joined binding/object read results:

@dataclass(frozen=True, slots=True)
class PutOutcome:
    inserted: bool
    stored_schema: str
    stored_canonical: str

@dataclass(frozen=True, slots=True)
class BindOutcome:
    bound: bool
    existing_schema: str
    existing_content_hash: str

@dataclass(frozen=True, slots=True)
class BoundObjectWrite:
    key: str
    schema: str
    content_hash: str
    canonical: str

@dataclass(frozen=True, slots=True)
class BoundObjectRow:
    binding_schema: str
    binding_content_hash: str
    object_schema: str | None
    canonical: str | None

class Backend(Protocol):
    async def put_object(
        self, *, schema: str, content_hash: str, canonical: str
    ) -> PutOutcome: ...
    async def get_object(
        self, *, schema: str, content_hash: str
    ) -> tuple[str, str] | None: ...
    async def bind(
        self, *, key: str, schema: str, content_hash: str
    ) -> BindOutcome: ...
    async def get_binding(self, *, key: str) -> tuple[str, str] | None: ...
    async def get_bound_objects(
        self, *, keys: tuple[str, ...]
    ) -> dict[str, BoundObjectRow]: ...
    async def put_bound_objects(
        self, *, entries: tuple[BoundObjectWrite, ...]
    ) -> dict[str, BindOutcome]: ...

class MemoryBackend: ...
class PostgresBackend:
    @classmethod
    async def open(cls, pool: asyncpg.Pool) -> PostgresBackend: ...

class SqliteBackend:
    @classmethod
    async def open(cls, path: str | Path) -> SqliteBackend: ...
    async def aclose(self) -> None: ...

Batch reads address only the supplied exact keys. SQLite performs chunked joined binding/object queries and does not promise one snapshot across the chunks. Every non-empty SQLite write batch uses one immediate transaction; a failure rolls back that transaction. Committed rows persist across reopen, but the backend does not promise power-loss durability.

PostgreSQL batches deduplicate objects and keys, use bounded set-based statements, and fetch bindings separately from distinct referenced objects. Every non-empty PostgreSQL write batch uses one transaction. The backend owns neither installation nor pool lifecycle, validates the fixed schema format during awaited open, and uses the fixed dr_store namespace regardless of the connection's search path.

Record Cache

The Record Cache is a memoization facade over an existing ObjectStore. It accepts opaque caller-owned keys, with derive_cache_key as the canonical helper for content-derived memoization. A typed hit keeps every strict JSON record, including null, distinct from a miss. SqliteRecordCache supplies the managed persistent form:

def derive_cache_key(namespace: str, payload: Jsonable) -> str: ...

@dataclass(frozen=True, slots=True)
class CacheHit:
    record: Jsonable

@dataclass(frozen=True, slots=True)
class CacheEntry:
    schema: str
    record: Jsonable

class RecordCache:
    def __init__(self, store: ObjectStore) -> None: ...
    async def get(self, key: str, *, schema: str) -> CacheHit | None: ...
    async def get_many(
        self, keys: Iterable[str], *, schema: str
    ) -> dict[str, CacheHit | None]: ...
    async def put(
        self, key: str, schema: str, record: Jsonable
    ) -> ObjectReference: ...
    async def put_many(
        self, entries: Mapping[str, CacheEntry]
    ) -> dict[str, ObjectReference]: ...

class SqliteRecordCache(RecordCache):
    @classmethod
    async def open(cls, path: str | Path) -> SqliteRecordCache: ...
    async def aclose(self) -> None: ...

get_many deduplicates requested keys and returns exactly those distinct keys, with each value independently classified as a hit or miss under the requested schema. It parses and verifies each returned bound object once; it does not promise that a multi-chunk backend read observes one snapshot. put_many validates, canonicalizes, and hashes each proposed entry once before invoking one backend write batch, then returns the first binding winner for every input key. Single-key get and put use the same paths and semantics.

The cache intentionally provides no scheduler, dirty tracking, key enumeration, prefix query, delete, expiry, eviction, or size cap. Callers own those policies and choose new keys for invalidation.

Opening captures a non-transient absolute filesystem path and establishes the SQLite schema before returning; empty and :memory: paths are rejected. Each instance owns one connection-affine worker and connection. Closing rejects new cache operations, waits for every admitted get, get_many, put, or put_many to finish, and then closes those owned resources. A successful close is idempotent for repeated and concurrent callers. SqliteRecordCacheClosedError reports operations requested after closing begins, before their inputs are validated; SqliteRecordCacheCloseError reports a terminal cleanup failure to close callers, including a context exit. Cancelling one close waiter does not cancel the shared terminal cleanup. Committed records remain available after close and reopen. Closing one cache does not close a separate instance or coordinate another process, even when both use the same database path. These persistence semantics do not promise power-loss durability.

Canonical JSON document files

A canonical JSON document file owns publication and verified reads for one caller-named document in an existing directory. The byte bound is required; the nesting-depth bound defaults to the dr-serialize canonical profile maximum and applies to both publication and read:

class CanonicalJsonFile:
    def __init__(
        self,
        directory: str | Path,
        name: str,
        *,
        max_bytes: int,
        max_depth: int = CANONICAL_JSON_MAX_CONTAINER_DEPTH,
    ) -> None: ...

    @property
    def path(self) -> Path: ...
    def publish(self, document: Jsonable) -> None: ...
    def read(self) -> Jsonable: ...

@verify(UNIQUE)
class PublicationStage(StrEnum):
    ENCODE = "encode"
    CREATE_TEMP = "create_temp"
    WRITE_TEMP = "write_temp"
    FLUSH_TEMP = "flush_temp"
    REPLACE_TARGET = "replace_target"
    FLUSH_DIRECTORY = "flush_directory"

@verify(UNIQUE)
class ReplacementState(StrEnum):
    NOT_REPLACED = "not_replaced"
    REPLACED = "replaced"
    UNKNOWN = "unknown"

DocumentPublishError reports a PublicationStage and ReplacementState. NOT_REPLACED means replacement was not invoked and any prior target remains authoritative. REPLACED means replacement returned before later finalization failed. UNKNOWN means the replacement operation itself failed and cannot prove whether the target changed, so callers must inspect or coordinate before treating either value as authoritative. DocumentReadError reports the requested path. Both errors derive from DocumentFileError and preserve the originating failure as their cause.

Document Directory

A Document Directory groups one canonical JSON Manifest with streamed binary Sidecars. The directory owns allocation and publication while SidecarWriter owns bounded retention:

class DocumentDirectory:
    def __init__(
        self,
        path: Path,
        manifest_name: str,
        *,
        manifest_max_bytes: int,
        manifest_max_depth: int = CANONICAL_JSON_MAX_CONTAINER_DEPTH,
    ) -> None: ...

    @classmethod
    def allocate(
        cls,
        root: str | Path,
        *,
        prefix: str,
        manifest_name: str,
        manifest_max_bytes: int,
        manifest_max_depth: int = CANONICAL_JSON_MAX_CONTAINER_DEPTH,
    ) -> DocumentDirectory: ...

    def publish(self, manifest: Jsonable) -> None: ...
    def open_sidecar(
        self,
        name: str,
        *,
        head_cap: int | None = None,
        tail_cap: int | None = None,
    ) -> SidecarWriter: ...

    def read_manifest(self) -> Jsonable: ...

    def verify_sidecar(
        self,
        name: str,
        *,
        expected_sidecar_hash: str,
        expected_head_length: int,
        expected_tail_length: int,
    ) -> None: ...

@dataclass(frozen=True, slots=True)
class SidecarSummary:
    head_length: int
    tail_length: int
    produced: int
    dropped: int
    sidecar_hash: str

class SidecarWriter:
    def write(self, chunk: bytes) -> None: ...
    def finalize(self) -> SidecarSummary: ...

Artifact bundles

An artifact bundle is separate from the mutable DocumentDirectory lifecycle. It uses strict frozen boundary models, complete raw-byte writers, and one terminal manifest transition:

class ArtifactBundlePublication:
    @classmethod
    def allocate(
        cls, root: str | Path, *, prefix: str
    ) -> ArtifactBundlePublication: ...

    @property
    def path(self) -> Path: ...
    def open_artifact(self, name: str) -> BundleArtifactWriter: ...
    def publish(self, payload: Jsonable) -> None: ...

class BundleArtifactWriter:
    def write(self, data: bytes) -> None: ...
    def finalize(self) -> ArtifactDescriptor: ...

class ArtifactDescriptor(BaseModel):
    name: str
    sha256: str
    byte_length: int

class BundleManifest(BaseModel):
    format: Literal["dr-store-artifact-bundle-v1"]
    artifacts: tuple[ArtifactDescriptor, ...]
    payload: Jsonable

@dataclass(frozen=True, slots=True)
class BundleReadLimits:
    manifest_max_bytes: int
    manifest_max_depth: int
    max_artifacts: int
    max_bytes_per_artifact: int
    max_total_artifact_bytes: int

class ArtifactBundleReader:
    def __init__(
        self, path: str | Path, *, limits: BundleReadLimits
    ) -> None: ...
    def audit(self) -> BundleManifest: ...
    def consume_and_verify_artifact(
        self,
        name: str,
        consumer: Callable[[VerifyingArtifactReader], None],
    ) -> ArtifactDescriptor: ...

class VerifyingArtifactReader:
    def read(self, size: int = -1) -> bytes: ...

Artifact names are exact identities consisting of one non-empty relative path segment. manifest.json, dot segments, separators, and the .dr-store-artifact-bundle- temporary namespace are reserved. The format adds no case-folding or Unicode-normalization policy. A duplicate name rejected before admission does not poison the publication; a create, write, or finalize failure after admission does. Active writers and invalid payloads refuse publication without making it terminal. The first valid manifest attempt after all writers finalize makes it terminal whether replacement succeeds or fails. audit() validates the strict canonical manifest under all declared limits and streams every declared artifact before returning that BundleManifest. consume_and_verify_artifact() instead opens one selected artifact once and delivers it through the package-owned read-only facade. Success requires the callback to observe an actual empty operating-system read: read(0) and reading exactly the declared length without one later empty read do not prove EOF. The facade is invalid after the synchronous callback returns, and success returns the verified ArtifactDescriptor, not a path whose later contents are claimed to remain verified.

BundleReadError covers manifest, filesystem, and consumer failures; BundleIncompleteError identifies a missing or invalid terminal manifest. BundleVerificationError exposes the selected or declared artifact_name and a BundleVerificationReason of missing, not_regular, mismatch, bounds_exceeded, or incomplete_consumption. Translated operating-system, decoding, model-validation, and consumer failures remain available as causes.

Filesystem and failure semantics

Canonical document publication creates a reserved unique temporary file with private permissions for each call, writes its complete canonical bytes, flushes and closes it, replaces the target in the same directory, and flushes and closes the directory. The case-insensitive .dr-store-document- prefix is reserved for these temporary files and cannot be used by document targets or Document Directory sidecars. Concurrent supported publishers use independent temporary files, and the last successful replacement is authoritative. Publication does not provide locks, compare-and-set, multi-file transactions, or ordering with Sidecar writes.

All-or-nothing visibility depends on the underlying filesystem honoring atomic same-directory replacement; network, synchronized, or other filesystems whose rename semantics are not established are outside current evidence. A final directory flush or close failure raises even though replacement may already be visible and does not roll the document back. Publication and allocation make no power-loss durability promise. Document Directory allocation uses a timestamp and UUID4, but a generated-name collision raises AllocationError rather than being retried, and allocation does not flush the caller-owned root directory.

Canonical files, Document Directories, and persistent SQLite storage capture a lexical absolute path at construction, so later working-directory changes do not redirect their operations. This does not freeze symlink targets. Directory durability is performed only where a pinned descriptor is already owned by the operation.

Canonical document reads open the named directory and regular direct child with required no-follow, directory-relative flags, then stream from the child descriptor they inspected. They read only to the configured byte bound plus the single byte needed to detect overflow, enforce the configured nesting-depth bound, and require one complete UTF-8 strict JSON value whose bytes are exactly canonical. Final-component symlinks and non-regular files are rejected, a replacement after open cannot redirect that read to a different inode, and platforms without the required descriptor operations fail closed.

Outside the reserved publication namespace, name validation prevents lexical traversal syntax. Sidecar creation and writes follow existing final-component symlinks and therefore require trusted, caller-controlled directory contents. Sidecar writer coordination remains the caller's concern. Sidecar finalization flushes the Sidecar descriptor before returning its stored-byte accounting and sidecar hash, but it does not flush the Sidecar's directory entry or impose ordering on document publication. Sidecar verification also refuses final-component symlinks for both the Document Directory and named child, requires a regular direct child, and reads from the descriptor it inspected.

Artifact-bundle publication has a deliberately weaker visibility-only filesystem boundary. Writers flush userspace buffers and close their files; manifest publication writes and closes one same-directory temporary file and atomically replaces manifest.json. It performs no F_FULLFSYNC, fsync, or directory flush and exposes no durability mode. Success is process-visible and terminal through this API, not a claim of crash, machine, filesystem, or power-loss durability.

Every artifact-bundle audit or verified-consumption operation opens the named bundle directory with required directory and no-follow behavior and holds that descriptor while validating the direct-child manifest.json and opening declared artifacts relative to it. Manifest reads are strict, canonical, and bounded. Artifact reads require inspected regular no-follow direct children and enforce count, per-artifact, and total byte bounds while streaming. Directory or child replacement after open cannot redirect those descriptor-owned reads, but the result is a point-in-time verification: it does not prevent later external mutation and does not claim containment for ancestor path resolution. Platforms without the required flags or os.open(dir_fd=...) fail closed.

A failed Sidecar write raises AllocationError and may leave its descriptor open and its accounting state advanced. The writer is unusable by contract and must be abandoned; retrying it or finalizing it has no supported outcome.