LorealDB: Open-Source High-Performance Embedded Python Database Engine

LorealDB is an open-source, high-performance, read-optimized embedded database engine built using Redb, simd json, crossbeam channel, Rust and exposed natively to Python. It stores JSON documents as raw bytes and automatically creates a background metadata index, allowing for lightning-fast querying and filtering without the overhead of heavy runtime parsing.

Not just another rust KV store wrapper.

Lorealdb is created to end your data storage headache with just one single, simple, highly optimized, and highly rustified tool.

✨ Features

• Dead Simple: No configuration, no servers, no ports, no drivers, no third party tools. Just point it to a file path and start saving data.
• Blazing Fast I/O: Built natively in 100% Rust for zero-cost abstraction.
• GIL-Free Reads: Database reads release the Global Interpreter Lock, allowing true multi-threading in Python.
• Asynchronous Writes: Upserts are pushed to a high-speed Rust MPSC channel, never blocking your Python thread.
• Auto batching: Database operations are automatically executed at every 10ms from the MPSC channel.
• SIMD-accelerated validation: JSON data is validated at CPU vector speeds.
• Native Types: Pass Python dict objects in, get Python dict objects back. Rust handles the serialization invisibly.
• Prefix Scanning: Efficiently query ranges of documents using key prefixes.
• Batch Operations: Fetch hundreds of keys at once with zero order-scrambling and memory-safe pre-allocation.

Benchmarks

Installation

Install the package using your favorite Python package manager:

pip install lorealdb

Or if you prefer using uv:

uv pip install lorealdb

Note: DBEngine is read-optimized. It natively accepts standard Python dictionaries, transparently converting them to raw bytes to automatically build a high-speed metadata index in the background.

Available Engines

1. DBEngine: A fast embedded JSON database with SIMD powered JSON/Dict validations.
2. DBEngineWebServer: A scaled JSON database for Python web frameworks.

Upcoming engines (in development)

1. DBEngineVectorStore: A document database + vector store for RAG and AI workloads.
2. DBEngineGraph: A graph document database for AI workloads.
3. DBEngineWriteOptimized: A document database storing AI conversations at scale.

Quick Start

from lorealdb import DBEngine

# 1. Connect to your database (it creates the file automatically)
db = DBEngine("my_data.redb")

# 2. Saving a complex Python dictionary
# The Rust engine automatically handles serialization natively.
app_settings = {"theme": "dark", "notifications": True, "version": 1.2}
db.insert("settings", app_settings)

# 3. Reading the data back
# The engine automatically deserializes the bytes directly into a Python dictionary.
settings = db.get("settings")
if settings:
    print(f"App Theme: {settings['theme']}")

# 4. Deleting a record
db.delete("settings")