lmdbm

This is a Python DBM interface style wrapper around LMDB (Lightning Memory-Mapped Database). It uses the existing lower level Python bindings py-lmdb. This is especially useful on Windows, where otherwise dbm.dumb is the default dbm database.

Install

• pip install lmdbm

Example

from lmdbm import Lmdb
with Lmdb.open("test.db", "c") as db:
  db[b"key"] = b"value"
  db.update({b"key1": b"value1", b"key2": b"value2"})  # batch insert, uses a single transaction

Use inheritance to store Python objects using json serialization

import json
from lmdbm import Lmdb

class JsonLmdb(Lmdb):
  def _pre_key(self, value):
    return value.encode("utf-8")
  def _post_key(self, value):
    return value.decode("utf-8")
  def _pre_value(self, value):
    return json.dumps(value).encode("utf-8")
  def _post_value(self, value):
    return json.loads(value.decode("utf-8"))

with JsonLmdb.open("test.db", "c") as db:
  db["key"] = {"some": "object"}
  obj = db["key"]
  print(obj["some"])  # prints "object"

Warning

As of lmdb==1.2.1 the docs say that calling lmdb.Environment.set_mapsize from multiple processes "may cause catastrophic loss of data". If lmdbm is used in write mode from multiple processes, set autogrow=False and map_size to a large enough value: Lmdb.open(..., map_size=2**30, autogrow=False).

Benchmarks

See benchmark.py and requirements-bench.txt. Other storage engines which could be tested: wiredtiger, berkeleydb. Storage engines not benchmarked: - tinydb (because it doesn't have built-in str/bytes keys)

continuous writes in seconds (best of 3)

random reads in seconds (best of 3)