s3-avldb

s3-avldb provides an S3Backend for avldb — a typed embedded document database with Rust-backed AVL indexes. With S3Backend you can use any AWS S3 bucket (or S3-compatible store such as MinIO or LocalStack) as durable storage.

Installation

pip install s3-avldb

Python 3.10+ and CPython are supported.

Quick start

import boto3
from avldb import Collection, Document
from s3avldb import S3Backend


class User(Document):
    name: str
    age: int


backend = S3Backend(bucket="my-bucket", prefix="users/")

with Collection(User, backend=backend) as users:
    users.ensure_index("age")
    users.insert(User(name="Ada", age=36))
    users.insert(User(name="Grace", age=29))

    adults = users.find({"age": {"$gte": 18}}).sort({"age": -1}).all()

Configuration

S3Backend(
    bucket="my-bucket",       # S3 bucket name
    prefix="myapp/users/",    # optional key prefix (no leading slash)
    client=None,              # inject a pre-configured boto3 S3 client, or None to create one
    endpoint_url=None,        # override endpoint for MinIO / LocalStack / other S3-compatible stores
    max_fetch_workers=None,   # default: all available logical CPUs; pass an int to limit
)

When client is None, S3Backend creates a boto3 S3 client on first use; standard boto3 credential discovery applies (environment variables, ~/.aws/credentials, instance profiles, etc.). Document reads use a bounded thread pool and preserve the backend's iteration order. By default its size is the number of logical CPUs available to the process; set max_fetch_workers to a positive integer to override it. When injecting your own boto3 client, configure its max_pool_connections to at least the resolved worker count; internally created clients are configured automatically.

How it works

S3Backend follows the same StorageBackend / BackendView contract as avldb's built-in DiskBackend and MemoryBackend. The S3 object layout is:

{prefix}/manifest.json               ← atomic commit marker
{prefix}/content/{uuid}.jsonl        ← immutable JSONL content segments
{prefix}/indexes/{field_hash}/{uuid}.jsonl  ← immutable JSONL index segments

Atomicity is achieved via S3 conditional writes (If-Match on PutObject): the manifest's ETag is used as an optimistic concurrency token. A conflicting commit from another writer raises WriteConflictError exactly as it would with DiskBackend.

Indexes are loaded into in-memory AVL trees at open() time. Index segment objects are downloaded concurrently using the configured fetch worker count, then decoded and replayed in manifest order to preserve update semantics. Documents are fetched lazily through concurrent byte-range GetObject calls.

No distributed lock is acquired — multiple processes may safely open the same prefix concurrently using optimistic concurrency.

Compaction

with Collection(User, backend=S3Backend("my-bucket", "users/")) as users:
    users.backend.compact()

Compaction consolidates all live data into a new content segment and updates the manifest, reducing the objects that future opens need to read. Previous segments are retained because an existing immutable view—or another process—may still reference them. Configure an S3 lifecycle rule if you want to reclaim superseded segments after a retention period suitable for your application.

Non-AWS S3 stores

backend = S3Backend(
    bucket="test-bucket",
    endpoint_url="http://localhost:9000",  # MinIO
)

Or inject a pre-built client:

import boto3
client = boto3.client("s3", endpoint_url="http://localhost:9000")
backend = S3Backend(bucket="test-bucket", client=client)

License

Copyright 2026 Anuradha Wickramarachchi.

Like avldb, s3-avldb is available under your choice of the Apache License 2.0 or the GNU General Public License v3.0 only. See the dual-license notice for details.