lazy_db

• Free software: MIT license
• Documentation: https://lazy-db.readthedocs.io.

A lazily loaded key:value db intended for use with large datasets that are too big to be loaded into memory. The database supports integers, strings, lists of integers, bytes, and dictionaries. This database is meant to strike a good balance of retrieval/insertion speed and memory usage. This database best fits a scenario where each key has a lot of data stored under it. Scenarios where values are under 100 bytes in size this database is not very well suited for.

Install with pip install lazy-database

Example usage:

from lazy_db import LazyDb

# Simple example usage
db = LazyDb("test.lazy")
db.write("test_value", "value")
print(db.read("test_value"))  # prints "value"
db.close()

# Or use a with statement to insure the database file is closed cleanly and avoid having to call db.close() on your own
with LazyDb("test2.lazy") as db:
    db.write("test_value", "value")
    print(db.read("test_value"))

How it works

File layout

All text in database files are encoded in utf-8 format. Each database has a json string at the start of the file that denotes the database's settings, who's end is marked with a NUL byte (00 in hex)

Each database entry is appended at the end of the file and is laid out in this format:

Content type labels

The algorithm

When loading a database, all entry headers are scanned for their key value and lengths. This allows for values to be retrieved very quickly without having to load the content of every entry, at the cost of having to store the key and content length in memory though. This approach makes the database best for cases where your database will be storing a lot of data in each key that you can't afford to store in memory, however you can afford to store the name values and lengths of each element in memory.