object-storage-client 0.0.32

Unified object storage client API

Object Storage Client

A unified object storage client for Rust and Python, supporting S3, GCS, Azure Blob Storage, HTTP/HTTPS, and Local Filesystem. It provides a simple, URL-based API for object operations, including cross-provider copy and move.

Features

• Unified API: Single interface for various storage backends.
• Cross-Provider: Copy or move objects between different storage providers (e.g., S3 to Local FS).
• Pre-signed URLs: Generate time-limited, credential-free URLs for S3, GCS and Azure.
• Multi-Language: Native Rust library with Python 3.13+ bindings.
• Streaming: Async streaming support for both Rust and Python.
• CLI: osc command-line tool for quick operations.

Supported Schemes

• s3://bucket/path (AWS S3)
• gs://bucket/path or gcs://bucket/path (Google Cloud Storage)
• az://, wasb://, wasbs://, abfs://, or abfss:// (Azure Blob Storage)
• http://host/path or https://host/path (HTTP/HTTPS)
• file:///absolute/path or local_path (Local Filesystem)

---

CLI Usage (osc)

The osc tool allows you to interact with object storage directly from your terminal.

Installation

Install the osc binary directly from the Bixority Codeberg crate registry with Cargo. Point Cargo at the registry with an environment variable, then install:

export CARGO_REGISTRIES_BIXORITY_INDEX="sparse+https://codeberg.org/api/packages/bixority/cargo/"
cargo install object-storage-client --registry bixority

Alternatively, install straight from Git:

cargo install --git https://codeberg.org/bixority/object-storage-client

Or, if you have the source code, install it from the local checkout:

cargo install --path .

Examples

• Upload a local file:

  osc put my_file.txt s3://my-bucket/remote_file.txt
  

• Download an object:

  osc get gs://my-bucket/data.json ./local_data.json
  

• Copy between providers:

  osc cp s3://source-bucket/image.png az://dest-container/image.png
  

• Move an object:

  osc mv s3://my-bucket/old_name.txt s3://my-bucket/new_name.txt
  

• List objects:

  osc ls s3://my-bucket/logs/
  

• Delete an object:

  osc rm s3://my-bucket/temp_file.tmp
  

• Stream an object:

  osc get-stream gs://my-bucket/large_file.bin
  

• Generate a pre-signed URL (S3, GCS, Azure):

  # Pre-signed download URL, valid for the default 1 hour
  osc sign s3://my-bucket/report.pdf
  
  # Pre-signed upload URL (PUT), valid for 15 minutes
  osc sign --method PUT --expires-in 900 s3://my-bucket/upload.bin
  
  # Pre-signed upload URL binding the exact size and type the client must send
  # (S3 only): the upload is rejected unless Content-Length and Content-Type
  # match, so the object store enforces size/type up front.
  osc sign --method PUT --content-length 1048576 \
      --content-type application/pdf s3://my-bucket/upload.pdf
  

---

Rust Usage

Installation

The crate is published to the Bixority Codeberg crate registry. Point Cargo at the registry with an environment variable:

export CARGO_REGISTRIES_BIXORITY_INDEX="sparse+https://codeberg.org/api/packages/bixority/cargo/"

Then add object-storage-client to your Cargo.toml:

[dependencies]
object-storage-client = { version = "0.0.31", registry = "bixority" }
tokio = { version = "1.0", features = ["full"] }

Alternatively, you can depend on it directly from Git:

[dependencies]
object-storage-client = { git = "https://codeberg.org/bixority/object-storage-client" }
tokio = { version = "1.0", features = ["full"] }

Example

use object_storage_client::ObjectStorageClient;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let client = ObjectStorageClient::new();

    // Upload data
    let data = b"Hello from Rust!";
    client.put("s3://my-bucket/hello.txt", &data).await?;

    // Download data
    let retrieved = client.get("s3://my-bucket/hello.txt").await?;
    println!("Retrieved: {:?}", String::from_utf8(retrieved.to_vec())?);

    // Cross-provider copy (S3 to Local)
    client.copy("s3://my-bucket/hello.txt", "file:///tmp/hello_local.txt").await?;

    // Pre-signed URL: time-limited, credential-free access (S3/GCS/Azure)
    use object_storage_client::{SignMethod, SignOptions};
    use std::time::Duration;

    // Pre-signed download (GET) link, valid for one hour
    let download_url = client
        .get_pre_signed_url(
            "s3://my-bucket/hello.txt",
            SignMethod::Get,
            Duration::from_secs(3600),
            &SignOptions::default(),
        )
        .await?;
    println!("Share this download link: {download_url}");

    // Pre-signed upload (PUT) link binding the exact size and type the client
    // must send (S3 only); the store rejects mismatched uploads up front.
    let upload_url = client
        .get_pre_signed_url(
            "s3://my-bucket/upload.bin",
            SignMethod::Put,
            Duration::from_secs(900),
            &SignOptions {
                content_length: Some(1_048_576),
                content_type: Some("application/octet-stream".to_string()),
            },
        )
        .await?;
    println!("Upload directly to: {upload_url}");

    Ok(())
}

---

Python 3.13+ Usage

Installation

The package is published on PyPI. Note that it requires Python 3.13+.

pip install object-storage-client

Or if you are developing locally, you can use maturin:

maturin develop

Example

import asyncio
from object_storage_client import ObjectStorageClient

async def main():
    client = ObjectStorageClient()

    # Upload data
    await client.put_object("s3://my-bucket/python_test.txt", b"Hello from Python!")

    # Download data
    data = await client.get_object("s3://my-bucket/python_test.txt")
    print(f"Retrieved: {data.decode()}")

    # List objects
    items = await client.list_objects("s3://my-bucket/")
    print(f"Bucket items: {items}")

    # Stream data
    stream = await client.get_object_stream("s3://my-bucket/python_test.txt")
    async for chunk in stream:
        print(f"Chunk size: {len(chunk)}")

    # Cross-provider move (GCS to S3)
    await client.move_object("gs://my-gcs-bucket/data.csv", "s3://my-s3-bucket/data.csv")

    # Pre-signed URL (S3/GCS/Azure): hand out credential-free, time-limited access
    download_url = await client.get_pre_signed_url("s3://my-bucket/python_test.txt")
    # Bind the exact Content-Length and Content-Type the client must send (S3
    # only); the store rejects uploads that don't match.
    upload_url = await client.get_pre_signed_url(
        "s3://my-bucket/upload.bin",
        method="PUT",
        expires_in_secs=900,
        content_length=1_048_576,
        content_type="application/octet-stream",
    )
    print(f"Download: {download_url}\nUpload: {upload_url}")

if __name__ == "__main__":
    asyncio.run(main())

Developer Instructions

Prerequisites

• Rust 1.85+ (or latest stable)
• Python 3.13+
• maturin (for Python bindings)

Building

• Rust: cargo build --release
• Python: maturin build --release
• CLI: cargo build --bin osc

Testing

cargo test