lmdb-object-store 0.1.0

Lightweight LMDB-backed object store for Python

lmdb-object-store

Lightweight thread-safe Python object store on top of LMDB. Provides a dict-like API (with buffering and atomic multi-put), automatic map-size growth, and fast zero-copy reads without having to handle LMDB's lower-level details.

---

Features

• Dict-like interface: store[key] = obj, obj = store.get(key), del store[key], key in store.
• Atomic multi-put: put_many(items) writes large amounts of data atomically in a single transaction.
• Write buffering: small writes are batched in memory; flush() persists them (auto-flush on size threshold).
• Auto map-size growth: retries on MapFullError by growing the LMDB map (2× or +64MiB), up to an optional cap.
• Fast reads: Uses LMDB zero-copy reads to minimize copies; get_many() efficiently combines buffered and DB reads.
• Flexible keys: bytes/bytearray/memoryview always supported; str keys allowed when key_encoding='utf-8' option is used.

---

Installation

pip install lmdb-object-store

Requirements

• Python 3.10+
• The lmdb Python package (wheels available for major platforms)

---

Quick start

from lmdb_object_store import LmdbObjectStore

# Create or open an LMDB-backed object store
with LmdbObjectStore(
    "path/to/db",
    batch_size=1000,                # flush buffer when it reaches this many entries
    autoflush_on_read=True,         # flush pending writes before reads
    key_encoding="utf-8",           # allow str keys (encoded with UTF-8)
    # Any lmdb.open(...) kwargs may be passed here, e.g.:
    map_size=128 * 1024 * 1024,     # 128 MiB initial map size
    subdir=True,                    # create directory layout
    readonly=False,
    # max_map_size is recognized (cap for auto-resize):
    max_map_size=4 * 1024 * 1024 * 1024,  # 4 GiB cap
) as store:

    # Put / get like a dict (values are pickled)
    store["user:42"] = {"name": "Ada", "plan": "pro"}
    print(store.get("user:42"))  # {'name': 'Ada', 'plan': 'pro'}

    # Existence checks
    if "user:42" in store:        # __contains__ has NO side effects (no flush)
        assert store.exists("user:42", flush=False) is True

    # Delete
    del store["user:42"]

    # Batch write, atomically (single transaction)
    items = {f"k{i}": {"i": i} for i in range(10_000)}
    store.put_many(items)

    # Fetch many at once
    found, not_found = store.get_many(["k1", "kX"], decode_keys=True)
    # found -> {'k1': {'i': 1}}, not_found -> ['kX']

# Clean close with strict error policy if needed:
store = LmdbObjectStore("path/to/db", key_encoding="utf-8")
try:
    # ... work with store ...
    store.close(strict=True)  # re-raise if final flush fails (after cleanup)
finally:
    # idempotent
    try: store.close()
    except Exception: pass

---

API Overview

Constructor

LmdbObjectStore(
    db_path: str,
    batch_size: int = 1000,
    *,
    autoflush_on_read: bool = True,
    key_encoding: str | None = None,
    key_errors: str = "strict",
    str_normalize: str | None = None,
    **lmdb_kwargs,
)

• db_path: LMDB environment path (passed to lmdb.open).
• batch_size: pending buffer size threshold to auto-flush.
• autoflush_on_read: if True, flushes buffer before reads (get, get_many, exists(flush=None)).
• key_encoding: enable str keys (e.g. "utf-8"). If None, only bytes-like keys are allowed.
• key_errors: error strategy for encoding str keys ("strict", "ignore", "replace", ...).
• str_normalize: Unicode normalization for str keys (e.g., "NFC", "NFKC").
• lmdb_kwargs: forwarded to lmdb.open(...) (e.g., map_size, subdir, readonly, etc). Special: max_map_size (cap for automatic map growth) is also recognized.

Put / Get

store.put(key, obj)                          # buffer write
obj = store.get(key, default=None)           # read; from buffer if present, else DB
store.flush()                                # persist the write buffer

• Values are serialized with pickle (highest protocol).
• get() uses zero-copy buffers internally; unpickling happens once per value.

Atomic multi-put

store.put_many(items: Mapping[Any, Any] | Iterable[tuple[Any, Any]])

• Writes all items in one LMDB write transaction.
• If a MapFullError occurs, the store will grow the map (2× or +64MiB) up to max_map_size and retry from the beginning.
• Note: put_many() first flushes any pending buffered writes; the atomic transaction only includes the items passed to this call.

Get many

found, not_found = store.get_many(
    keys: Sequence[Any],
    *,
    decode_keys: bool = False,
    decode_not_found: bool | None = None,   # None → follow decode_keys
)

• Returns a tuple:

  • found: {key: value} for keys found (key type is bytes by default, or str if decode_keys=True).
  • not_found: list of input keys not found (decoded to str if decode_not_found=True).

• Efficiently merges results from the write buffer and DB; autoflush_on_read applies unless overridden via other APIs.

Existence & containment

store.exists(key, *, flush: bool | None = None) -> bool
key in store  # __contains__ → NO flush, purely checks current state

• flush=None (default) follows autoflush_on_read.
• flush=False guarantees no implicit flush (useful for side-effect-free checks).

Deletion

del store[key]       # KeyError if not present
store.delete(key)    # schedules deletion via buffer (dict-like semantics)

Lifecycle

with LmdbObjectStore(...) as store:
    ...
# or
store.close(strict: bool = False)

• close(strict=True) re-raises the last flush error after closing the environment; otherwise it logs and completes.

---

Concurrency Model

• Internally uses RLock + Condition and a simple reader count to coordinate:

  • Multiple concurrent readers are allowed.
  • Writers hold the lock (buffer mutation + flush/commit).
  • close() sets a “closing” flag and waits until the active reader count reaches zero.

• Designed for thread-safety within a single process. While LMDB itself supports multi-process access, this wrapper's locking is process-local; if you need multi-process writes, coordinate at a higher level.

---

Map Size & Auto-Resize

• Initial size is given by map_size (forwarded to lmdb.open).

• When a write/commit hits MapFullError:

  • The store grows to max(current*2, current+64MiB), capped at max_map_size if provided,
  • and retries the operation.

• If the cap is reached and still insufficient, the error is propagated.

---

Performance Tips

• Batch writes: Keep batch_size large enough for your workload; call flush() at logical boundaries.
• Use put_many() for bulk inserts—single transaction with fewer fsyncs.
• Avoid unnecessary decodes: If you don't need str keys on output, leave decode_* parameters off.
• Key type: If possible, pass bytes keys directly (saves encoding overhead).

---

Error Handling

• Unpickling failures are raised as a RuntimeError (with key context) when reading from buffer/DB.
• Missing keys: __getitem__ and del raise KeyError; get() returns default.
• Final flush failure on close: re-raised if strict=True; otherwise logged.

---

Security Note

This library uses Python pickle for value serialization. Never unpickle data from untrusted sources.

---

Configuration Reference

• batch_size: int – buffer size threshold to auto-flush (default: 1000).

• autoflush_on_read: bool – flush before reads (default: True).

• key_encoding: Optional[str] – enable str keys (e.g., "utf-8"). If None, only bytes-like keys are accepted.

• key_errors: str – encoding error handling ("strict", "ignore", "replace").

• str_normalize: Optional[str] – Unicode normalization for str keys ("NFC", "NFKC", ...).

• lmdb_kwargs – forwarded to lmdb.open(...):

  • map_size, subdir, readonly, lock, ...
  • max_map_size (recognized by this wrapper to cap auto-growth).