s3dlio - Universal Storage I/O Library

High-performance, multi-protocol storage library for AI/ML workloads with universal copy operations across S3, Azure, GCS, local file systems, and DirectIO.

v0.9.110 — Multi-agent bug audit fix release — 39 fixes across 6 phases (issues #151–#157)

Closes a coordinated 7-issue, 39-bug audit of s3dlio HEAD, fanning out specialized reviewers by subsystem then adversarially verifying every finding: silent data corruption in range/multipart paths (Phase A), wrong-data-returned bugs reachable from the public API (Phase B), silent exception-swallowing (Phase C), backend correctness/hygiene across S3/Azure/GCS (Phase D), URI-parsing and scheme-detection edge cases (Phase E), and dead/mismatched env-var knobs (Phase F). Every fix landed with a RED-then-GREEN regression test that fails against unmodified code and passes after the fix. Full bug-by-bug table and per-commit list: docs/Changelog.md.

v0.9.108 (prior): Performance & concurrency audit — 17 fixes across 4 phases (issue #148). BREAKING for https://: no longer negotiates HTTP/2 by default (now matches http:// — HTTP/1.1 unless opted in via S3DLIO_HTTPS_H2=1/S3DLIO_ENABLE_HTTP2=1).

v0.9.106 (prior): Write verification (S3DLIO_PUT_VERIFY, S3DLIO_MPU_PUT_VERIFY) changed from always-on to opt-in (mlcommons/storage#593 follow-up).

v0.9.102 (prior): SDK error-chain diagnostics + cold-start timeout / retry knobs (mlcommons/storage#506).

📦 Installation

Quick Install (Python)

# If using uv package manager + uv virtual environment:
uv pip install s3dlio

# If using pip without uv:
pip install s3dlio

Python Backend Profiles (PyPI vs Full Build)

• If using uv package manager + uv virtual environment: uv pip install s3dlio.
• If using standard pip without uv: pip install s3dlio.
• The default published wheel is now S3-focused (Azure Blob and GCS are excluded).
• If you want full backends (S3 + Azure Blob + GCS), build from source with:

# uv workflow:
uv pip install s3dlio --no-binary s3dlio --config-settings "cargo-extra-args=--features extension-module,full-backends"

# pip-only workflow:
pip install s3dlio --no-binary s3dlio --config-settings "cargo-extra-args=--features extension-module,full-backends"

You can still add a separate package name (for example s3dlio-full) later if you want a dedicated prebuilt full wheel distribution.

Maintainer note: for PyPI uploads, publish the default (./build_pyo3.sh) wheel unless intentionally releasing a separate distribution. full-backends is currently source-build only via the command above.

Building from Source (Rust)

System Dependencies

s3dlio requires some system libraries to build. Only OpenSSL and pkg-config are required by default. HDF5 and hwloc are optional and improve functionality but are not needed for the core library:

Ubuntu/Debian:

# Quick install - run our helper script
./scripts/install-system-deps.sh

# Or manually (required only):
sudo apt-get install -y build-essential pkg-config libssl-dev

# Optional - for NUMA topology support (--features numa):
sudo apt-get install -y libhwloc-dev

# Optional - for HDF5 data format support (--features hdf5):
sudo apt-get install -y libhdf5-dev

# All optional libraries at once:
sudo apt-get install -y libhdf5-dev libhwloc-dev cmake

RHEL/CentOS/Fedora/Rocky/AlmaLinux:

# Quick install
./scripts/install-system-deps.sh

# Or manually (required only):
sudo dnf install -y gcc gcc-c++ make pkg-config openssl-devel

# Optional - for NUMA topology support:
sudo dnf install -y hwloc-devel

# Optional - for HDF5 data format support:
sudo dnf install -y hdf5-devel

# All optional libraries at once:
sudo dnf install -y hdf5-devel hwloc-devel cmake

macOS:

# Quick install
./scripts/install-system-deps.sh

# Or manually (required only):
brew install pkg-config openssl@3

# Optional - for NUMA/HDF5 support:
brew install hdf5 hwloc cmake

# Set environment variables (add to ~/.zshrc or ~/.bash_profile):
export PKG_CONFIG_PATH="$(brew --prefix openssl@3)/lib/pkgconfig:$PKG_CONFIG_PATH"
export OPENSSL_DIR="$(brew --prefix openssl@3)"

Arch Linux:

# Quick install
./scripts/install-system-deps.sh

# Or manually (required only):
sudo pacman -S base-devel pkg-config openssl

# Optional - for NUMA/HDF5 support:
sudo pacman -S hdf5 hwloc cmake

WSL (Windows Subsystem for Linux) / Minimal Environments:

If you are building on WSL or any environment where libhdf5 or libhwloc may not be available, s3dlio builds without them by default. No extra libraries are required:

# Just the basics - works on WSL, Docker, CI, and minimal installs:
sudo apt-get install -y build-essential pkg-config libssl-dev
cargo build --release
# install Python package (no system HDF5/hwloc needed):
# uv workflow:
uv pip install s3dlio
# pip-only workflow:
pip install s3dlio

Install Rust (if not already installed)

curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
source $HOME/.cargo/env

Build s3dlio

# Clone the repository
git clone https://github.com/russfellows/s3dlio.git
cd s3dlio

# Build with default features (no HDF5 or NUMA required)
cargo build --release

# Build s3-cli with all cloud backends enabled (AWS + Azure + GCS)
cargo build --release --bin s3-cli --features full-backends

# Build s3-cli with GCS enabled only (plus default backends)
cargo build --release --bin s3-cli --features backend-gcs

# Build with NUMA topology support (requires libhwloc-dev)
cargo build --release --features numa

# Build with HDF5 data format support (requires libhdf5-dev)
cargo build --release --features hdf5

# Build with all optional features
cargo build --release --features numa,hdf5

# Run tests
cargo test

# Build Python bindings (optional)
./build_pyo3.sh

# Build Python bindings with full backends (S3 + Azure + GCS)
./build_pyo3.sh full

# Named profile form is also supported:
./build_pyo3.sh --profile full
./build_pyo3.sh --profile default

# Show profile/help usage
./build_pyo3.sh --help

Build Profile Quick Reference

Rust backend feature profiles:

• Default build (cargo build --release): S3-focused default backend set.
• GCS-enabled build (--features backend-gcs): enables GCS in addition to default set.
• Full cloud build (--features full-backends): enables AWS + Azure + GCS.

Python wheel build profiles via build_pyo3.sh:

• default or slim: AWS + file/direct; excludes Azure and GCS.
• full: AWS + Azure + GCS + file/direct.
• Positional and named forms are equivalent:
  • ./build_pyo3.sh full
  • ./build_pyo3.sh -p full
  • ./build_pyo3.sh --profile full

Optional extra Rust features for wheel builds can still be passed with EXTRA_FEATURES. Example: EXTRA_FEATURES="numa,hdf5" ./build_pyo3.sh full.

Note: NUMA support (--features numa) improves multi-socket performance but requires the hwloc2 C library. HDF5 support (--features hdf5) enables HDF5 data format generation but requires libhdf5. Both are optional and s3dlio is fully functional without them.

Platform support: s3dlio builds natively on Linux (x86_64, aarch64), macOS (x86_64 and Apple Silicon arm64), and WSL. Making numa and hdf5 optional was the key change for broad platform support — all remaining dependencies are pure Rust or use platform-independent system libraries (OpenSSL). To cross-compile Python wheels for Linux ARM64 from an x86_64 host, see build_pyo3.sh for instructions using the --zig linker. For macOS universal2 (fat binary covering both architectures), see the commented section in build_pyo3.sh.

✨ Key Features

• High Performance: High-throughput multi GB/s reads and writes on platforms with sufficient network and storage capabilities
• Zero-Copy Architecture: bytes::Bytes throughout for minimal memory overhead
• Multi-Protocol: S3, Azure Blob, GCS, file://, direct:// (O_DIRECT)
• HTTP/2 Support (opt-in): HTTP/2 is available but HTTP/1.1 is the default — HTTP/2 is almost always slower for bulk storage workloads. Opt in by setting S3DLIO_H2C=1. Both TLS ALPN (https://) and cleartext h2c (http://) are supported when enabled. See docs/HTTP2_ALPN_INVESTIGATION.md.
• Python & Rust: Native Rust library with zero-copy Python bindings (PyO3), bytearray support for efficient memory management
• Multi-Endpoint Load Balancing: RoundRobin/LeastConnections across storage endpoints
• AI/ML Ready: PyTorch DataLoader integration, TFRecord/NPZ format support
• Parquet DataLoader: Per-row-group epoch-aware DataLoader for any s3dlio storage backend (S3, Azure Blob, GCS, file://, direct://). Epoch-2 zero re-fetch speedup (2.5×+), Raw + Arrow IPC decode modes, 8-worker concurrency with shared metadata caches — see guide
• High-Speed Data Generation: 50+ GB/s test data with configurable compression/dedup

🌟 Latest Release

v0.9.110 — Multi-agent bug audit fix release: 39 fixes across 6 phases, 7 issues (#151–#157). See docs/Changelog.md.

Recent highlights:

• v0.9.110 - Phase D+E+F of the audit (17 bugs): backend correctness/hygiene (GCS retry backoff, RAPID-bucket cache poisoning, Azure 50K-block cap, community-GCS multipart stub), URI/scheme-detection edge cases, dead/mismatched env-var knobs; new S3DLIO_S3_ENDPOINT_TLS_PORTS; 748 tests passing
• v0.9.109 - Phase A+B+C of the audit (22 bugs): silent data corruption in range/multipart paths, wrong-data-returned bugs, silent exception-swallowing; new S3DLIO_S3_ENDPOINT_HINT_TLDS + DLIO multipart env vars
• v0.9.98 - Parquet DataLoader (ParquetRowGroupDataset): per-row-group Dataset, epoch-2 zero-re-fetch (2.5× speedup proven), Raw + ArrowIpc decode modes, 8-worker shared caches; 648 tests passing
• v0.9.97 - XorStream (dedup-safe, ~15 GB/s/core); S3DLIO_UNSIGNED_PAYLOAD opt-in for private S3-compatible endpoints; 613 tests passing
• v0.9.92 - MPU coordinator task, auto-scale, async write/flush/finish safety fixes, MAX_MULTIPART_PARTS guard; 580 tests passing
• v0.9.90 - Full NVIDIA AIStore support (S3DLIO_FOLLOW_REDIRECTS=1) with all TLS security policies; HTTP/2 available (opt-in via S3DLIO_H2C=1, not the default); 5 issues closed (#126, #133, #134, #135, #136); 559 tests passing
• v0.9.86 - Redirect follower for NVIDIA AIStore (S3 path); HTTPS→HTTP downgrade prevention; 21 new redirect tests; redirect security analysis documented
• v0.9.84 - HEAD elimination (ObjectSizeCache); OnceLock env-var caching; lock-free range assembly; AWS_CA_BUNDLE_PATH → AWS_CA_BUNDLE; structured tracing
• v0.9.80 - Python list hang fix (IMDSv2 legacy call removed); tracing deadlock fix (tokio::spawn → inline stream); async S3 delete/bucket helpers; deprecated Python APIs cleaned up

📖 Complete Changelog - Full version history, migration guides, API details

📚 Version History

For detailed release notes and migration guides, see the Complete Changelog.

Storage Backend Support

Universal Backend Architecture

s3dlio provides unified storage operations across all backends with consistent URI patterns:

• 🗄️ Amazon S3: s3://bucket/prefix/ - High-performance S3 operations (5+ GB/s reads, 2.5+ GB/s writes) with built-in concurrent range GETs (on by default)
• ☁️ Azure Blob Storage: az://container/prefix/ - Complete Azure integration with RangeEngine (30-50% faster for large blobs)
• 🌐 Google Cloud Storage: gs://bucket/prefix/ or gcs://bucket/prefix/ - Production ready with RangeEngine and full ObjectStore integration
• 📁 Local File System: file:///path/to/directory/ - High-speed local file operations with RangeEngine support
• ⚡ DirectIO: direct:///path/to/directory/ - Bypass OS cache for maximum I/O performance with RangeEngine

Concurrent Range GET Performance Features (v0.9.3+, Updated v0.9.60)

Concurrent range downloads hide network latency by parallelizing HTTP range requests.

All backends support concurrent range GETs — but via two different mechanisms:

Mechanism 1 — S3 built-in (on by default, v0.9.60+)

• ✅ Amazon S3: Concurrent range splitting enabled by default via S3DLIO_ENABLE_RANGE_OPTIMIZATION (default: on). Uses get_object_concurrent_range_async() — fires parallel GetObject(Range: bytes=N-M) requests via the AWS SDK with lock-free chunk assembly. Controlled by S3DLIO_RANGE_THRESHOLD_MB (default: 32 MiB) and S3DLIO_RANGE_CONCURRENCY (default: auto-scaled). Disable with S3DLIO_ENABLE_RANGE_OPTIMIZATION=0.

Mechanism 2 — RangeEngine (per-store config flag, must enable explicitly)

• ✅ Azure Blob Storage: 30-50% faster for large files (enable_range_engine: true in AzureConfig)
• ✅ Google Cloud Storage: 30-50% faster for large files (enable_range_engine: true in GcsConfig)
• ⚠️ Local File System: Rarely beneficial due to seek overhead (disabled by default)
• ⚠️ DirectIO: Rarely beneficial due to O_DIRECT overhead (disabled by default)

RangeEngine config flag defaults (v0.9.6+):

• Status: enable_range_engine: false by default in all per-store config structs
• Reason: Extra HEAD request on every GET causes ~50% slowdown for small-object workloads
• Threshold: 32 MiB default (tunable per-store via RangeEngineConfig::min_split_size)

How to Enable for Large-File Workloads:

use s3dlio::object_store::{AzureObjectStore, AzureConfig};

let config = AzureConfig {
    enable_range_engine: true,  // Explicitly enable for large files
    ..Default::default()
};
let store = AzureObjectStore::with_config(config);

When to Enable:

• ✅ Large-file workloads (average size >= 32 MiB)
• ✅ High-bandwidth, high-latency networks
• ❌ Mixed or small-object workloads
• ❌ Local file systems

S3 Backend Options

s3dlio supports two S3 backend implementations. Native AWS SDK is the default and recommended for production use:

# Default: Native AWS SDK backend (RECOMMENDED for production)
cargo build --release
# or explicitly:
cargo build --no-default-features --features native-backends

# Experimental: Apache Arrow object_store backend (optional, for testing)
cargo build --no-default-features --features arrow-backend

Why native-backends is default:

• Proven performance in production workloads
• Optimized for high-throughput S3 operations (5+ GB/s reads, 2.5+ GB/s writes)
• Well-tested with MinIO, Vast, and AWS S3

About arrow-backend:

• Experimental alternative implementation
• No proven performance advantage over native backend
• Useful for comparison testing and development
• Not recommended for production use

GCS Backend Options (Current)

GCS is now optional at build time.

• Default build (cargo build --release) does not include GCS.
• To include GCS, enable backend-gcs (or full-backends).
• When enabled, s3dlio uses the official Google crates (google-cloud-storage + gax) from a patched fork maintained for s3dlio.

# Default build (S3-focused; no GCS)
cargo build --release

# Enable GCS explicitly
cargo build --release --features backend-gcs

# Enable all cloud backends (AWS + Azure + GCS)
cargo build --release --features full-backends

Patched official GCS fork used by s3dlio:

• Repository: https://github.com/russfellows/google-cloud-rust
• Integration in this repo is pinned in Cargo.toml (currently via release tag from that fork).

Legacy note: gcs-community remains as a legacy opt-in path, but the primary supported path is the official Google crates from the patched russfellows/google-cloud-rust fork.

Quick Start

Installation

Rust CLI:

git clone https://github.com/russfellows/s3dlio.git
cd s3dlio
cargo build --release

# Full cloud backend CLI build:
cargo build --release --bin s3-cli --features full-backends

Python Library:

# uv workflow:
uv pip install s3dlio

# pip-only workflow:
pip install s3dlio

# or build from source:
./build_pyo3.sh && ./install_pyo3_wheel.sh

# build from source with full cloud backends:
./build_pyo3.sh --profile full && ./install_pyo3_wheel.sh

Documentation

• CLI Guide - Complete command-line interface reference with examples
• Python API Guide - Complete Python library reference with examples
• Parquet DataLoader Guide - Epoch-aware per-row-group Parquet DataLoader (v0.9.98+)
• Multi-Endpoint Guide - Load balancing across multiple storage endpoints (v0.9.14+)
• Rust API Guide v0.9.0 - Complete Rust library reference with migration guide
• Changelog - Version history and release notes
• Adaptive Tuning Guide - Optional performance auto-tuning
• Testing Guide - Test suite documentation

Core Capabilities

🚀 Universal Copy Operations

s3dlio treats upload and download as enhanced versions of the Unix cp command, working across all storage backends:

CLI Usage:

# Upload to any backend with real-time progress
s3-cli upload /local/data/*.log s3://mybucket/logs/
s3-cli upload /local/files/* az://container/data/  
s3-cli upload /local/models/* gs://ml-bucket/models/
s3-cli upload /local/backup/* file:///remote-mount/backup/
s3-cli upload /local/cache/* direct:///nvme-storage/cache/

# Download from any backend  
s3-cli download s3://bucket/data/ ./local-data/
s3-cli download az://container/logs/ ./logs/
s3-cli download gs://ml-bucket/datasets/ ./datasets/
s3-cli download file:///network-storage/data/ ./data/

# Cross-backend copying workflow
s3-cli download s3://source-bucket/data/ ./temp/
s3-cli upload ./temp/* gs://dest-bucket/data/

Advanced Pattern Matching:

# Glob patterns for file selection (upload)
s3-cli upload "/data/*.log" s3://bucket/logs/
s3-cli upload "/files/data_*.csv" az://container/data/

# Regex patterns for listing (use single quotes to prevent shell expansion)
s3-cli ls -r s3://bucket/ -p '.*\.txt$'           # Only .txt files
s3-cli ls -r gs://bucket/ -p '.*\.(csv|json)$'    # CSV or JSON files
s3-cli ls -r az://acct/cont/ -p '.*/data_.*'      # Files with "data_" in path

# Count objects matching pattern (with progress indicator)
s3-cli ls -rc gs://bucket/data/ -p '.*\.npz$'
# Output: ⠙ [00:00:05] 71,305 objects (14,261 obj/s)
#         Total objects: 142,610 (10.0s, rate: 14,261 objects/s)

# Delete only matching files
s3-cli delete -r s3://bucket/logs/ -p '.*\.log$'

See CLI Guide for complete command reference and pattern syntax.

🐍 Python Integration

High-Performance Data Operations:

import s3dlio

# Universal upload/download across all backends
s3dlio.upload(['/local/data.csv'], 's3://bucket/data/')
s3dlio.upload(['/local/logs/*.log'], 'az://container/logs/')  
s3dlio.upload(['/local/models/*.pt'], 'gs://ml-bucket/models/')
s3dlio.download('s3://bucket/data/', './local-data/')
s3dlio.download('gs://ml-bucket/datasets/', './datasets/')

# High-level AI/ML operations
dataset = s3dlio.create_dataset("s3://bucket/training-data/")
loader = s3dlio.create_async_loader("gs://ml-bucket/data/", {"batch_size": 32})

# PyTorch integration
from s3dlio.torch import S3IterableDataset
from torch.utils.data import DataLoader

dataset = S3IterableDataset("gs://bucket/data/", loader_opts={})
dataloader = DataLoader(dataset, batch_size=16)

Streaming & Compression:

# High-performance streaming with compression
options = s3dlio.PyWriterOptions()
options.compression = "zstd"
options.compression_level = 3

writer = s3dlio.create_s3_writer('s3://bucket/data.zst', options)
writer.write_chunk(large_data_bytes)
stats = writer.finalize()  # Returns (bytes_written, compressed_bytes)

# Data generation with configurable modes
s3dlio.put("s3://bucket/test-data-{}.bin", num=1000, size=4194304, 
          data_gen_mode="streaming")  # 2.6-3.5x faster for most cases

XorStream — dedup-safe data generation (v0.9.97+):

XorStream generates unique, incompressible data at ~15 GB/s per core without Rayon thread management overhead. Every fill() / generate() call is guaranteed to produce a different 512-byte-block-level fingerprint. Ideal for PUT-heavy benchmarks where many worker threads share a single generator.

import s3dlio
import numpy as np

stream = s3dlio.XorStream()

# --- Fastest path: in-place fill into pre-allocated bytearray ---
buf = bytearray(8 * 1024 * 1024)   # 8 MiB working buffer
stream.fill(buf)                    # fill once — unique payload
stream.fill(buf)                    # fill again — different payload, guaranteed

print(stream.objects_generated)     # == 2

# --- Convenience path: allocate + fill in one call ---
data = stream.generate(8 * 1024 * 1024)  # returns BytesView
view = memoryview(data)                   # zero-copy Python view
arr  = np.frombuffer(view, dtype=np.uint8)  # zero-copy numpy array

# --- PUT with XorStream data (benchmark pattern) ---
import threading

def worker(stream, uri_template, n):
    buf = bytearray(8 * 1024 * 1024)
    for i in range(n):
        stream.fill(buf)       # reuse buffer, unique data each time
        s3dlio.put_bytes(buf, uri_template.format(i))

threads = [threading.Thread(target=worker, args=(stream, "s3://bucket/obj-{}.bin", 100))
           for _ in range(32)]
for t in threads: t.start()
for t in threads: t.join()

Multi-Endpoint Load Balancing (v0.9.14+):

# Distribute I/O across multiple storage endpoints
store = s3dlio.create_multi_endpoint_store(
    uris=[
        "s3://bucket-1/data",
        "s3://bucket-2/data", 
        "s3://bucket-3/data",
    ],
    strategy="least_connections"  # or "round_t robin"
)

# Zero-copy data access (memoryview compatible)
data = store.get("s3://bucket-1/file.bin")
array = np.frombuffer(memoryview(data), dtype=np.float32)

# Monitor load distribution
stats = store.get_endpoint_stats()
for i, s in enumerate(stats):
    print(f"Endpoint {i}: {s['requests']} requests, {s['bytes_transferred']} bytes")

📖 Complete Multi-Endpoint Guide - Load balancing, configuration, use cases

📦 Parquet DataLoader — Epoch-Aware Training (v0.9.98+)

The Parquet DataLoader provides per-row-group streaming for Parquet files on any s3dlio-accessible storage — S3, Azure Blob, GCS, file://, and direct:// (O_DIRECT; tested and working). Only the URI prefix changes; no code changes needed to switch backends. Two decode modes: Raw (Python decodes) and ArrowIpc (Rust decodes to Arrow IPC bytes). Zero footer re-fetches on epoch 2+ (row-group byte ranges cached in a process-global DashMap after epoch 1; backend-agnostic).

import s3dlio

# Works with any s3dlio URI — just change the prefix:
#   "s3://bucket/train/"           Amazon S3 / MinIO / Ceph
#   "az://container/train/"        Azure Blob Storage
#   "gs://bucket/train/"           Google Cloud Storage
#   "file:///mnt/data/train/"      Local filesystem
#   "direct:///mnt/nvme/train/"    Local O_DIRECT (bypass page cache)

# Raw mode — Python decodes with PyArrow (default)
loader = s3dlio.create_async_loader(
    "s3://bucket/train/",
    {"format": "parquet", "prefetch": 32}
)
for item in loader:
    # item["data"]: bytes, item["uri"]: str, item["rg_idx"]: int
    table = pyarrow.parquet.read_table(io.BytesIO(item["data"]))

# Arrow IPC mode — Rust decodes, Python gets ready-to-use RecordBatch bytes
loader = s3dlio.create_async_loader(
    "direct:///mnt/nvme/train/",   # same API, different backend
    {"format": "parquet", "decode": "arrow", "prefetch": 32}
)
for item in loader:
    batch = pa.ipc.open_stream(pa.py_buffer(item["data"])).read_next_batch()

Epoch-2+ speedup — measured against live MinIO:

Memory: only metadata in RAM; 8 concurrent workers share process-global caches (no 8× duplication).

📖 Complete Parquet DataLoader Guide

Performance

Benchmark Results

s3dlio delivers world-class performance across all operations:

Optimization Features

• HTTP/2 Support (opt-in): HTTP/2 is supported but not the default — HTTP/1.1 is used unless you set S3DLIO_H2C=1. HTTP/2 is available for scenarios that benefit from multiplexing, but is typically slower for bulk storage workloads.
• Intelligent Defaults: Streaming mode automatically selected based on benchmarks
• Multi-Process Architecture: Massive parallelism for maximum performance
• Zero-Copy Streaming: Memory-efficient operations for large datasets
• Configurable Chunk Sizes: Fine-tune performance for your workload

Checkpoint system for model states

store = s3dlio.PyCheckpointStore('file:///tmp/checkpoints/') store.save('model_state', your_model_data) loaded_data = store.load('model_state')

**Ready for Production**: All core functionality validated, comprehensive test suite, and honest documentation matching actual capabilities.

## Configuration & Tuning

### Environment Variables
s3dlio supports comprehensive configuration through environment variables:

- **NVIDIA AIStore**: `S3DLIO_FOLLOW_REDIRECTS=1` - Enable HTTP 307 redirect following for AIStore (opt-in, disabled by default); `S3DLIO_REDIRECT_MAX=5` - Maximum redirect hops per request
- **HTTP/2 mode**: `S3DLIO_H2C=1` - Force h2c (HTTP/2 cleartext) on http:// endpoints; `S3DLIO_H2C=0` - Force HTTP/1.1; unset = auto-probe (default)
- **Runtime Scaling**: `S3DLIO_RT_THREADS=32` - Tokio worker threads
- **Connection Pool**: `S3DLIO_POOL_MAX_IDLE_PER_HOST=32` - Max idle connections per host (default: 32)
- **S3 Range GET**: `S3DLIO_ENABLE_RANGE_OPTIMIZATION=0` - Disable concurrent range splitting (enabled by default); `S3DLIO_RANGE_THRESHOLD_MB=64` - Size threshold in MiB (default: 32); `S3DLIO_RANGE_CONCURRENCY=64` - Max concurrent range requests
- **Operation Logging**: `S3DLIO_OPLOG_LEVEL=2` - S3 operation tracking

📖 [Environment Variables Reference](docs/Environment_Variables.md)

### Operation Logging (Op-Log)
Universal operation trace logging across all backends with zstd-compressed TSV format, warp-replay compatible.

```python
import s3dlio
s3dlio.init_op_log("operations.tsv.zst")
# All operations automatically logged
s3dlio.finalize_op_log()

See S3DLIO OpLog Implementation for detailed usage.

Building from Source

Prerequisites

• Rust: Install Rust toolchain
• Python 3.12+: For Python library development
• UV (recommended): Install UV
• OpenSSL: Required (libssl-dev on Ubuntu)
• HDF5 (optional): Only needed with --features hdf5 (libhdf5-dev on Ubuntu, brew install hdf5 on macOS)
• hwloc (optional): Only needed with --features numa (libhwloc-dev on Ubuntu)

Build Steps

# Python environment
uv venv && source .venv/bin/activate

# Rust CLI
cargo build --release

# Python library
./build_pyo3.sh && ./install_pyo3_wheel.sh

Configuration

Environment Setup

# Required for S3 operations
AWS_ACCESS_KEY_ID=your-access-key
AWS_SECRET_ACCESS_KEY=your-secret-key
AWS_ENDPOINT_URL=https://your-s3-endpoint
AWS_REGION=us-east-1

Enable comprehensive S3 operation logging compatible with MinIO warp format:

Advanced Features

CPU Profiling & Analysis

cargo build --release --features profiling
cargo run --example simple_flamegraph_test --features profiling

Compression & Streaming

import s3dlio
options = s3dlio.PyWriterOptions()
options.compression = "zstd"
writer = s3dlio.create_s3_writer('s3://bucket/data.zst', options)
writer.write_chunk(large_data)
stats = writer.finalize()

Container Deployment

# Use pre-built container
podman pull quay.io/russfellows-sig65/s3dlio
podman run --net=host --rm -it quay.io/russfellows-sig65/s3dlio

# Or build locally
podman build -t s3dlio .

Note: Always use --net=host for storage backend connectivity.

Documentation & Support

• 🖥️ CLI Guide: docs/CLI_GUIDE.md - Complete command-line reference
• 🐍 Python API: docs/PYTHON_API_GUIDE.md - Python library reference
• � Parquet DataLoader: docs/Parquet_Data-Loader.md - Epoch-aware Parquet DataLoader guide (v0.9.98+)
• �📚 API Documentation: docs/api/
• 📝 Changelog: docs/Changelog.md
• 🧪 Testing Guide: docs/TESTING-GUIDE.md
• 🚀 Performance: docs/performance/

🔗 Related Projects

• sai3-bench - Multi-protocol I/O benchmarking suite built on s3dlio
• polarWarp - Op-log analysis tool for parsing and visualizing s3dlio operation logs
• google-cloud-rust (s3dlio patched fork) - Official Google Cloud Rust client fork used by s3dlio for patched GCS support

License

Licensed under the Apache License 2.0 - see LICENSE file.

🚀 Ready to get started? Check out the Quick Start section above or explore our example scripts for common use cases!