slatedb 0.11.1

A cloud native embedded storage engine built on object storage.

Introduction

SlateDB is an embedded storage engine built as a log-structured merge-tree. Unlike traditional LSM-tree storage engines, SlateDB writes data to object storage (S3, GCS, ABS, MinIO, Tigris, and so on). Leveraging object storage allows SlateDB to provide bottomless storage capacity, high durability, and easy replication. The trade-off is that object storage has a higher latency and higher API cost than local disk.

To mitigate high write API costs (PUTs), SlateDB batches writes. Rather than writing every put() call to object storage, MemTables are flushed periodically to object storage as a string-sorted table (SST). The flush interval is configurable.

put() returns a Future that resolves when the data is durably persisted. Clients that prefer lower latency at the cost of durability can instead use put_with_options with await_durable set to false.

To mitigate read latency and read API costs (GETs), SlateDB will use standard LSM-tree caching techniques: in-memory block caches, compression, bloom filters, and local SST disk caches.

Checkout slatedb.io to learn more.

Installation

pip install slatedb

Requirements

• Python 3.10 or higher

Features

• Support for in-memory object store and object stores (S3, GCS, ABS, MinIO, memory)
• Sync and async APIs
• Range scans with iteration and seek
• Snapshots for consistent, read-only views
• Transactions with snapshot isolation (SI) and serializable snapshot isolation (SSI)
• Atomic batched writes
• Merge operator (user-defined Python callable)
• Per-operation time-to-live (TTL)
• Reader for read-only access with optional checkpoint pinning
• Admin APIs for manifests, checkpoints, clones, garbage collection, and sequence↔timestamp mapping

Usage

The example below demonstrates common features in a single script.

from __future__ import annotations

from slatedb import (
    SlateDB,
    SlateDBReader,
    SlateDBAdmin,
    WriteBatch,
)


# Optional: define a merge operator. This example concatenates values.
def concat(key: bytes, existing: bytes | None, value: bytes) -> bytes:
    return (existing or b"") + value

# Open a database using the in-memory object store
db = SlateDB("/tmp/slatedb-demo", url="memory:///", merge_operator=concat)

# Basic CRUD
db.put(b"user:1", b"Alice")
assert db.get(b"user:1") == b"Alice"

# Per-op TTL and durability control
db.put_with_options(b"temp", b"ok", ttl=5_000, await_durable=False)

# Batched writes
wb = WriteBatch()
wb.put(b"batch:1", b"one")
wb.delete(b"temp")
db.write(wb)

# Transactions (SSI or SI). Operations buffer until commit.
txn = db.begin("ssi")
txn.put(b"user:2", b"Bob")
txn.merge(b"counter:visits", b"1")  # uses the configured merge operator
txn.commit()

# Snapshots provide consistent, read-only views
snap = db.snapshot()
assert snap.get(b"user:2") == b"Bob"

# Range scans (prefix scan when end omitted). Iterator supports seek.
it = db.scan(b"user:")
first = next(it)
it.seek(b"user:2")
rest = list(it)

# Advanced scan options (read-ahead, caching, durability filter)
_ = list(db.scan_with_options(b"user:", cache_blocks=True, read_ahead_bytes=1 << 20))

# Flush
db.flush_with_options("wal")

# Get metrics
metrics = db.metrics()

# Create a durable checkpoint and read it with a read-only reader
ckpt = db.create_checkpoint(scope="durable")
reader = SlateDBReader("/tmp/slatedb-demo", url="memory:///", checkpoint_id=ckpt["id"])
_ = list(reader.scan(b"user:"))
reader.close()

# Admin: list checkpoints, read manifests, GC, and sequence/timestamp mapping
admin = SlateDBAdmin("/tmp/slatedb-demo", url="memory///")
_ = admin.list_checkpoints()
_ = admin.read_manifest()
admin.run_gc_once(manifest_min_age=0, wal_min_age=0, compacted_min_age=0)

db.close()

Contributing

SlateDB's Python bindings use uv to manage the development environment. You can install uv following the steps on its website. Once you've installed uv, run uv venv to create a virtual environment.

Installing dependencies

Run uv pip install to install all dependencies. If you only want test dependencies, you can run uv pip install -e .[test].

Running tests

Run uv run pytest to run all tests.

Building the project

SlateDB's Python bindings use Maturin to link the Rust codebase with the Python codebase.

1. Install Maturin by running uv tool install maturin.
2. Build the project with Maturin by running uv run maturin develop.

License

SlateDB is licensed under the Apache License, Version 2.0.

Foundation

SlateDB is a member of the Commonhaus Foundation.