storage-bucket-s2

High-performance encrypted file uploader for Storage Buckets

Installation • Quick Start • Usage Guide • API Reference

storage-bucket-s2 uploads files to Storage Buckets at maximum speed using a compiled C engine. Files are automatically split into chunks, encrypted with AES-256-CBC, and uploaded in parallel — all from a simple 3-line Python API.

Why?

Installation

pip install storage-bucket-s2

The C engine compiles automatically during install. If system dependencies are missing, you'll get a clear message:

sudo apt install -y gcc make libcurl4-openssl-dev libssl-dev
pip install storage-bucket-s2

sudo dnf install -y gcc make libcurl-devel openssl-devel
pip install storage-bucket-s2

sudo pacman -S gcc make curl openssl
pip install storage-bucket-s2

apk add gcc make musl-dev curl-dev openssl-dev
pip install storage-bucket-s2

brew install curl openssl
pip install storage-bucket-s2

Quick Start

from storage_bucket_s2 import StorageClient

client = StorageClient("https://your-server.com")
bucket = client.bucket("my-bucket-id", token="bkt_xxxx")
result = bucket.upload("video.mp4")

print(f"✓ Uploaded {result.file_id} ({result.size_human}) in {result.duration_human}")
# ✓ Uploaded abc123 (1.4 GB) in 2m 15s

That's it. The C engine handles chunking, encryption, parallel upload, retry, and cleanup.

Usage Guide

Basic Upload

from storage_bucket_s2 import StorageClient

client = StorageClient("https://your-server.com")
bucket = client.bucket("bucket-id", token="bkt_your_upload_token")

result = bucket.upload("/path/to/file.mp4")
print(result.file_id)       # "abc123-def456"
print(result.success)       # True
print(result.size_human)    # "1.4 GB"
print(result.duration_human)# "2m 15s"
print(result.speed_human)   # "10.6 MB/s"

Upload with Live Progress

def on_progress(p):
    print(
        f"\r{p.percent:5.1f}% | {p.speed_human:>12} | "
        f"ETA {p.eta_human:>8} | "
        f"{p.completed_chunks}/{p.total_chunks} chunks",
        end="", flush=True
    )

result = bucket.upload("big_file.tar.gz", on_progress=on_progress)
print(f"\nDone! {result.file_id}")

Output:

 45.2% |   52.4 MB/s |  ETA  1m 12s | 18/40 chunks

Parallel Workers

Control how many chunks upload simultaneously (default: 4, max: 16):

# Fast connection — use more workers
result = bucket.upload("file.iso", workers=8)

# Slow/metered connection — use fewer
result = bucket.upload("file.zip", workers=2)

Error Handling

from storage_bucket_s2 import StorageClient
from storage_bucket_s2.exceptions import (
    AuthenticationError,
    QuotaExceededError,
    UploadError,
    EngineError,
)

client = StorageClient("https://your-server.com")
bucket = client.bucket("my-bucket", token="bkt_xxxx")

try:
    result = bucket.upload("file.mp4")
    print(f"✓ {result.file_id}")

except AuthenticationError:
    print("✗ Invalid or expired token")

except QuotaExceededError:
    print("✗ Bucket is full — free up space or upgrade")

except UploadError as e:
    print(f"✗ Upload failed: {e}")
    print(f"  Failed chunks: {e.failed_chunks}")
    for err in e.errors:
        print(f"  - {err}")

except EngineError as e:
    print(f"✗ C engine issue: {e}")

except FileNotFoundError:
    print("✗ File not found")

List Files in a Bucket

files = bucket.list_files(path="/", limit=50)
for f in files:
    print(f"{f['name']}  ({f.get('size', '?')} bytes)")

Get File Info

file_info = bucket.get_file("file-id-123")
print(file_info)

Server Health Check

client = StorageClient("https://your-server.com")
if client.health():
    print("Server is up")
else:
    print("Server is down")

API Reference

StorageClient(base_url, timeout=10)

Create a client connected to your storage server.

Methods:

Bucket

A handle to a specific bucket. Created via client.bucket().

bucket.upload(file_path, path="/", workers=4, on_progress=None)

Upload a file to the bucket via the C engine.

Returns: UploadResult

bucket.list_files(path="/", limit=100)

List files in the bucket. Returns a list of file dicts.

bucket.get_file(file_id)

Get info for a specific file. Returns a file dict.

UploadResult

Returned by bucket.upload().

UploadProgress

Passed to the on_progress callback during upload.

Exceptions

All exceptions inherit from StorageBucketError.

How It Works

pip install storage-bucket-s2
         │
         ▼
┌─────────────────────────────────┐
│  Build Backend (_build_backend) │
│  1. Detect OS & dependencies    │
│  2. Compile C engine (make)     │
│  3. Bundle binary in package    │
└─────────────────────────────────┘

bucket.upload("file.mp4")
         │
         ▼
┌─────────────────────────────────┐
│  Python (engine.py)             │
│  1. Write config.json           │
│  2. Spawn C engine subprocess   │
│  3. Poll status.json            │
│  4. Return UploadResult         │
└─────────────────────────────────┘
         │
         ▼
┌─────────────────────────────────┐
│  C Engine (bucket_uploader_v2)  │
│  1. Read config.json            │
│  2. POST /upload/init           │
│  3. Split file → encrypt chunks │
│  4. PUT chunks in parallel      │
│  5. POST /upload/complete       │
│  6. Write status.json (live)    │
│  7. Cleanup temp files          │
└─────────────────────────────────┘

Security

• AES-256-CBC encryption per chunk with unique keys
• Keys are base64-decoded, used, then securely zeroed (OPENSSL_cleanse)
• Encrypted temp files deleted immediately after upload
• All communication over HTTPS

Reliability

• Exponential backoff on server errors (500, 429)
• Per-chunk retry (up to 5 attempts)
• Connection timeout + stall detection
• Graceful shutdown on SIGINT/SIGTERM
• Orphaned temp file cleanup on startup

License

MIT