slatedb 0.8.1

A cloud native embedded storage engine built on object storage.

WARNING

This is alpha software and is not yet ready for production use. Missing features:

• Only uses in-memory object storage
• No range query
• No checkpoints
• No builders
• ... and more

Please see SlateDB's Python Github issues to contribute.

Introduction

SlateDB is an embedded storage engine built as a log-structured merge-tree. Unlike traditional LSM-tree storage engines, SlateDB writes data to object storage (S3, GCS, ABS, MinIO, Tigris, and so on). Leveraging object storage allows SlateDB to provide bottomless storage capacity, high durability, and easy replication. The trade-off is that object storage has a higher latency and higher API cost than local disk.

To mitigate high write API costs (PUTs), SlateDB batches writes. Rather than writing every put() call to object storage, MemTables are flushed periodically to object storage as a string-sorted table (SST). The flush interval is configurable.

put() returns a Future that resolves when the data is durably persisted. Clients that prefer lower latency at the cost of durability can instead use put_with_options with await_durable set to false.

To mitigate read latency and read API costs (GETs), SlateDB will use standard LSM-tree caching techniques: in-memory block caches, compression, bloom filters, and local SST disk caches.

Checkout slatedb.io to learn more.

Installation

pip install slatedb

Requirements

• Python 3.10 or higher

Usage

Basic Operations

from slatedb import SlateDB

# Create or open a database
db = SlateDB("/path/to/your/database")

# Put a key-value pair
db.put(b"hello", b"world")

# Retrieve a value
value = db.get(b"hello")  # Returns b"world"

# Delete a key-value pair
db.delete(b"hello")

# Always close when done
db.close()

Connecting to an object store based on its URL

# Open the database
db = SlateDB("/tmp/slatedb", url="s3://my-bucket/my-prefix")

# Put a key-value pair
db.put(b"hello", b"world")

# Retrieve a value
value = db.get(b"hello")  # Returns b"world"

# Always close when done
db.close()

Asynchronous API

SlateDB also provides async methods for use with asyncio:

import asyncio
from slatedb import SlateDB

async def main():
    db = SlateDB("/path/to/your/database")

    # Async operations
    await db.put_async(b"hello", b"async world")
    value = await db.get_async(b"hello")
    await db.delete_async(b"hello")

    # Don't forget to close
    db.close()

# Run the async example
asyncio.run(main())

Error Handling

Most methods raise ValueError for errors like empty keys or database operation failures:

try:
    db.put(b"", b"This will fail")
except ValueError as e:
    print(f"Error: {e}")

Documentation

• SlateDB core documentation - Comprehensive guide to understand how SlateDB works.
• API Reference - Detailed Python API documentation embedded in the source code.

Contributing

SlateDB's Python bindings use Uv to manage the development environment. You can install Uv following the steps on its website. Once you've installed Uv, run uv venv to create a virtual environment.

Installing dependencies

Run uv pip install to install all dependencies. If you only want test dependencies, you can run uv pip install -e .[test].

Running tests

Run uv run pytest to run all tests.

Building the project

SlateDB's Python bindings use Maturin to link the Rust codebase with the Python codebase.

1. Install Maturin by running uv tool install maturin.
2. Build the project with Maturin by running uv run maturin develop.

License

SlateDB is licensed under the Apache License, Version 2.0.

Foundation

SlateDB is a member of the Commonhaus Foundation.