object-storage-client 0.0.31

Unified object storage client API

Object Storage Client

A unified object storage client for Rust and Python, supporting S3, GCS, Azure Blob Storage, HTTP/HTTPS, and Local Filesystem. It provides a simple, URL-based API for object operations, including cross-provider copy and move.

Features

• Unified API: Single interface for various storage backends.
• Cross-Provider: Copy or move objects between different storage providers (e.g., S3 to Local FS).
• Pre-signed URLs: Generate time-limited, credential-free URLs for S3, GCS and Azure.
• Multi-Language: Native Rust library with Python 3.13+ bindings.
• Streaming: Async streaming support for both Rust and Python.
• CLI: osc command-line tool for quick operations.

Supported Schemes

• s3://bucket/path (AWS S3)
• gs://bucket/path or gcs://bucket/path (Google Cloud Storage)
• az://, wasb://, wasbs://, abfs://, or abfss:// (Azure Blob Storage)
• http://host/path or https://host/path (HTTP/HTTPS)
• file:///absolute/path or local_path (Local Filesystem)

---

CLI Usage (osc)

The osc tool allows you to interact with object storage directly from your terminal.

Installation

If you have the source code, you can install it using Cargo:

cargo install --path .

Examples

• Upload a local file:

  osc put my_file.txt s3://my-bucket/remote_file.txt
  

• Download an object:

  osc get gs://my-bucket/data.json ./local_data.json
  

• Copy between providers:

  osc cp s3://source-bucket/image.png az://dest-container/image.png
  

• Move an object:

  osc mv s3://my-bucket/old_name.txt s3://my-bucket/new_name.txt
  

• List objects:

  osc ls s3://my-bucket/logs/
  

• Delete an object:

  osc rm s3://my-bucket/temp_file.tmp
  

• Stream an object:

  osc get-stream gs://my-bucket/large_file.bin
  

• Generate a pre-signed URL (S3, GCS, Azure):

  # Pre-signed download URL, valid for the default 1 hour
  osc sign s3://my-bucket/report.pdf
  
  # Pre-signed upload URL (PUT), valid for 15 minutes
  osc sign --method PUT --expires-in 900 s3://my-bucket/upload.bin
  

---

Rust Usage

Installation

Add object-storage-client to your Cargo.toml:

[dependencies]
object-storage-client = { git = "https://github.com/bixority/object-storage-client" }
tokio = { version = "1.0", features = ["full"] }

Example

use object_storage_client::ObjectStorageClient;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let client = ObjectStorageClient::new();

    // Upload data
    let data = b"Hello from Rust!";
    client.put("s3://my-bucket/hello.txt", &data).await?;

    // Download data
    let retrieved = client.get("s3://my-bucket/hello.txt").await?;
    println!("Retrieved: {:?}", String::from_utf8(retrieved.to_vec())?);

    // Cross-provider copy (S3 to Local)
    client.copy("s3://my-bucket/hello.txt", "file:///tmp/hello_local.txt").await?;

    // Pre-signed URL: time-limited, credential-free access (S3/GCS/Azure)
    use object_storage_client::SignMethod;
    use std::time::Duration;

    // Pre-signed download (GET) link, valid for one hour
    let download_url = client
        .get_pre_signed_url("s3://my-bucket/hello.txt", SignMethod::Get, Duration::from_secs(3600))
        .await?;
    println!("Share this download link: {download_url}");

    // Pre-signed upload (PUT) link, valid for 15 minutes
    let upload_url = client
        .get_pre_signed_url("s3://my-bucket/upload.bin", SignMethod::Put, Duration::from_secs(900))
        .await?;
    println!("Upload directly to: {upload_url}");

    Ok(())
}

---

Python 3.13+ Usage

Installation

You can install the package using pip. Note that it requires Python 3.13+.

pip install git+https://github.com/bixority/object-storage-client

Or if you are developing locally, you can use maturin:

maturin develop

Example

import asyncio
from object_storage_client import ObjectStorageClient

async def main():
    client = ObjectStorageClient()

    # Upload data
    await client.put_object("s3://my-bucket/python_test.txt", b"Hello from Python!")

    # Download data
    data = await client.get_object("s3://my-bucket/python_test.txt")
    print(f"Retrieved: {data.decode()}")

    # List objects
    items = await client.list_objects("s3://my-bucket/")
    print(f"Bucket items: {items}")

    # Stream data
    stream = await client.get_object_stream("s3://my-bucket/python_test.txt")
    async for chunk in stream:
        print(f"Chunk size: {len(chunk)}")

    # Cross-provider move (GCS to S3)
    await client.move_object("gs://my-gcs-bucket/data.csv", "s3://my-s3-bucket/data.csv")

    # Pre-signed URL (S3/GCS/Azure): hand out credential-free, time-limited access
    download_url = await client.get_pre_signed_url("s3://my-bucket/python_test.txt")
    upload_url = await client.get_pre_signed_url(
        "s3://my-bucket/upload.bin", method="PUT", expires_in_secs=900
    )
    print(f"Download: {download_url}\nUpload: {upload_url}")

if __name__ == "__main__":
    asyncio.run(main())

Developer Instructions

Prerequisites

• Rust 1.85+ (or latest stable)
• Python 3.13+
• maturin (for Python bindings)

Building

• Rust: cargo build --release
• Python: maturin build --release
• CLI: cargo build --bin osc

Testing

cargo test