alocals3 0.7.1

alocals3: a Rust S3-like server and Rust-backed Python client

简体中文

alocals3

alocals3 is a local S3-like object store focused on fast local development and internal workloads.

The current main branch is Rust-first:

• Server: pure Rust binary, no Python runtime required.
• Metadata backend: SQLite or PostgreSQL.
• Object payloads: local filesystem with SHA-256 based sharding.
• Python client: wheel package backed by Rust networking through reqwest.
• Python target: Python 3.12+ with PyO3 abi3-py312 limited API.

The project implements an S3-compatible subset, not the full AWS S3 API surface.

Quick Start

Build and run the Rust server:

PYO3_NO_PYTHON=1 cargo build --release --no-default-features --features server,server-binary --bin alocals3-server

target/release/alocals3-server \
  --host 127.0.0.1 \
  --port 8000 \
  --database-url "sqlite:///./alocals3.db" \
  --storage-root ./data

Background garbage collection is enabled by default. It periodically removes object files that are no longer referenced by the database and removes database records whose files are missing. Candidates must be older than the GC grace period before they are deleted.

target/release/alocals3-server \
  --gc-interval-secs 300 \
  --gc-grace-secs 300 \
  --gc-start-delay-secs 30

Set --disable-gc to disable background GC. --gc-interval-secs 0 and ALOCALS3_GC_INTERVAL_SECS=0 are equivalent.

Run the reproducible GC correctness suite:

./examples/gc_correctness.sh

It covers orphan files, missing-file records, stale temp files, live/shared object retention, grace-period behavior, GC disable modes, and orphan-path reuse races. Latest local result:

GC correctness and race scenarios passed
orphan_files_deleted=1 missing_records_deleted=1 stale_tmp_files_deleted=1
remaining_objects: grace=1 disable_flag=1 interval_zero=1 race=30

Use PostgreSQL instead of SQLite:

target/release/alocals3-server \
  --host 127.0.0.1 \
  --port 8000 \
  --database-url "postgresql://user:password@127.0.0.1:5432/alocals3" \
  --storage-root ./data

Install the Python client from source:

python3.12 -m venv .venv
source .venv/bin/activate
python -m pip install -U pip maturin
python -m pip install -e .

Build Artifacts

Release helper scripts live in scripts:

# Linux static server plus Python 3.12+ ABI3 wheel
scripts/build-linux-release.sh

# macOS arm64, macOS 11 deployment baseline
scripts/build-macos-release.sh

# Windows 10+ PowerShell
.\scripts\build-windows-release.ps1

Linux server builds default to x86_64-unknown-linux-musl; Linux wheels default to manylinux_2_28. macOS builds default to aarch64-apple-darwin with MACOSX_DEPLOYMENT_TARGET=11.0.

The wheel is configured as Python 3.12+ ABI3 via PyO3 abi3-py312. It is not a cp312-cp312 wheel unless the ABI3 feature is removed.

Platform wheels include the Rust server executable and install an alocals3-server command. Standalone server artifacts are also named alocals3-server on Unix-like platforms and alocals3-server.exe on Windows.

Configuration

Server CLI flags:

• --host: bind host, default 127.0.0.1
• --port: bind port, default 8000
• --database-url: SQLite or PostgreSQL URL
• --storage-root: object payload root directory
• --log-level: log level or tracing_subscriber filter directive, default info
• --max-upload-size: maximum upload body size in bytes, default 0 disables the axum body limit
• --version: print server version and author information

Environment variables:

• ALOCALS3_DATABASE_URL: default sqlite:///./alocals3.db
• ALOCALS3_STORAGE_ROOT: default ./data
• ALOCALS3_LOG_LEVEL: default info
• ALOCALS3_MAX_UPLOAD_SIZE: default 0

Database URL examples:

• SQLite: sqlite:///./alocals3.db
• PostgreSQL: postgresql://user:password@127.0.0.1:5432/alocals3

Use an absolute SQLite path in scripts and services to avoid accidentally writing to different database files from different working directories. PostgreSQL is recommended for sustained concurrent workloads.

Logging is written to stderr. Common levels are debug, info, warn, and error; full EnvFilter directives such as warn,alocals3_server=debug are also accepted.

Uploads are not capped by the server body extractor by default. The current implementation buffers each uploaded object in memory before writing it to disk, so practical limits are still determined by process memory and available storage. Set --max-upload-size to restore an explicit byte limit.

Storage Layout

• Bucket and object metadata is stored in SQLite or PostgreSQL.
• Object bytes are stored on local disk.
• Blob paths are content-addressed and sharded:
  • sha256(<object bytes>) = <digest>
  • {storage_root}/objects/{digest[:2]}/{digest[2:4]}/{digest}

Object keys, bucket names, prefixes, delimiters, and continuation tokens are UTF-8 text. Client path parameters are UTF-8 percent-encoded automatically; pass raw strings such as logs/data.txt or logs/数据.txt, not pre-encoded URL fragments.

HTTP API

• GET /healthz: health check
• GET /s3: list buckets
• PUT /s3/{bucket}: create bucket
• DELETE /s3/{bucket}: delete empty bucket
• GET /s3/{bucket}/objects: list objects
• GET /s3/{bucket}?list-type=2: S3-style ListObjectsV2
• PUT /s3/{bucket}/{key}: upload object
• GET /s3/{bucket}/{key}: download object
• HEAD /s3/{bucket}/{key}: object metadata
• DELETE /s3/{bucket}/{key}: delete object

Supported object features:

• ETag is the MD5 hex digest of the object body.
• Range requests return 206 or 416.
• If-None-Match and If-Match are supported for PUT.
• Content-MD5 is validated on PUT.
• If-None-Match is supported for GET and HEAD.

PUT /s3/{bucket}/{key} returns:

• 201: new object created
• 200: existing object overwritten
• 400: invalid Content-MD5
• 412: conditional request failed

Client Usage

The Python runtime dependency list is intentionally empty. HTTP networking is implemented in Rust, not httpx.

import asyncio
from pathlib import Path

from alocals3.client import ALocalS3Client, ALocalS3ClientAsync

with ALocalS3Client("http://127.0.0.1:8000", disable_proxy=True) as client:
    client.create_bucket("demo")
    info = client.put_object("demo", "logs/数据.txt", Path("data.txt"))
    print(info["etag"])

    data, headers = client.get_object_range("demo", "logs/数据.txt", "bytes=0-99")
    print(len(data), headers.get("content-range"))

    with client.open("s3://demo/logs/数据.txt", "r") as f:
        print(f.read())

    with client.open("s3://demo/logs/from-open.txt", "wb") as f:
        f.write(b"hello from file-like API\n")

    client.get_object_to_file("demo", "logs/数据.txt", Path("copy.txt"))


async def main() -> None:
    async with ALocalS3ClientAsync("http://127.0.0.1:8000", disable_proxy=True) as client:
        print(await client.list_buckets())
        async with client.open("s3://demo/logs/from-async-open.txt", "wb") as f:
            await f.write(b"hello from async file-like API\n")
        async with client.open("s3://demo/logs/from-async-open.txt", "rb") as f:
            print(await f.read())


asyncio.run(main())

CLI:

python -m alocals3.client --endpoint http://127.0.0.1:8000 CREATE_BUCKET demo
python -m alocals3.client --endpoint http://127.0.0.1:8000 PUT demo file.bin ./file.bin
python -m alocals3.client --endpoint http://127.0.0.1:8000 GET demo file.bin ./copy.bin
python -m alocals3.client --endpoint http://127.0.0.1:8000 LIST_OBJECTS_V2 demo --prefix logs/ --delimiter /

Set disable_proxy=True or pass --disable-proxy to ignore proxy environment variables such as HTTP_PROXY, HTTPS_PROXY, ALL_PROXY, and NO_PROXY.

client.open() is file-like but not an in-memory object wrapper:

• Read modes ("rb" / "r") create a Rust-backed streaming HTTP reader. open() sends the request and reads response headers, but object bytes are pulled from the network when the returned file object's read() path runs.
• Write modes ("wb" / "w") spool writes to a Rust-owned temporary file. The HTTP PUT is sent when the file is closed or the with block exits successfully. If the with block exits with an exception, the upload is discarded.
• cache_path= is best-effort and is populated as bytes pass through the file-like object; cache write failures do not fail the network operation.
• For asyncio code, ALocalS3ClientAsync.open() returns an async file-like object for async with. It provides awaitable read(), readline(), readinto(), write(), flush(), close(), and discard() methods backed by the same Rust implementation.

Benchmark the Python file-like API with a large object:

./examples/benchmark_file_like.sh

Validate stream multiplexing and Range-based resume reads:

./examples/benchmark_stream_multiplex.sh

The current file-like API supports resume reads through range_header="bytes=N-". Upload resume is not implemented yet; write modes spool to a Rust-owned temporary file and use one HTTP PUT on close.

Curl Examples

curl -i -X PUT http://127.0.0.1:8000/s3/demo
curl -i -X PUT --data-binary @file.bin http://127.0.0.1:8000/s3/demo/file.bin
curl -i http://127.0.0.1:8000/s3/demo/file.bin
curl -i -H "Range: bytes=0-99" http://127.0.0.1:8000/s3/demo/file.bin
curl -sS "http://127.0.0.1:8000/s3/demo?list-type=2&prefix=logs/&delimiter=/&max-keys=100"

Conditional PUT:

curl -i -X PUT -H "If-None-Match: *" --data-binary @file.bin \
  http://127.0.0.1:8000/s3/demo/file.bin

curl -i -X PUT -H 'If-Match: "d41d8cd98f00b204e9800998ecf8427e"' --data-binary @file.bin \
  http://127.0.0.1:8000/s3/demo/file.bin

MD5_B64=$(openssl md5 -binary file.bin | openssl base64)
curl -i -X PUT -H "Content-MD5: ${MD5_B64}" --data-binary @file.bin \
  http://127.0.0.1:8000/s3/demo/file.bin

Consistency Notes

• Object bytes are written through a temporary file and atomic rename.
• Metadata updates are committed through the selected database backend.
• This is not a single distributed transaction across database and filesystem.
• Under process or machine failure, orphan blob files may exist. The Python package no longer ships an alternate storage backend or Python server path; operational cleanup should be handled outside the request path.

Updates

updates.md

License

The MIT License