storage-bucket-s2 0.1.0

High-performance encrypted file uploader for Storage Buckets

storage-bucket-s2

High-performance encrypted file uploader for Storage Buckets

Installation • Quick Start • Usage Guide • API Reference

---

storage-bucket-s2 uploads files to Storage Buckets at maximum speed using a compiled C engine. Files are automatically split into chunks, encrypted with AES-256-CBC, and uploaded in parallel — all from a simple 3-line Python API.

Why?

	storage-bucket-s2	Pure Python
Speed	C-native multi-threaded I/O	GIL-limited
Encryption	OpenSSL AES-256-CBC (hardware-accelerated)	Software AES
Memory	Streaming chunks, never loads full file	Same
Reliability	Exponential backoff, per-chunk retry	Manual
Cleanup	Crash-safe temp file cleanup + orphan detection	None

---

Installation

pip install storage-bucket-s2

The C engine compiles automatically during install. If system dependencies are missing, you'll get a clear message:

Ubuntu / Debian

sudo apt install -y gcc make libcurl4-openssl-dev libssl-dev
pip install storage-bucket-s2

Fedora / RHEL / CentOS / Rocky

sudo dnf install -y gcc make libcurl-devel openssl-devel
pip install storage-bucket-s2

Arch / Manjaro

sudo pacman -S gcc make curl openssl
pip install storage-bucket-s2

Alpine

apk add gcc make musl-dev curl-dev openssl-dev
pip install storage-bucket-s2

macOS

brew install curl openssl
pip install storage-bucket-s2

Development Install

git clone <repo-url>
cd bucket_uploader/v2
pip install -e .

---

Quick Start

from storage_bucket_s2 import StorageClient

client = StorageClient("https://your-server.com")
bucket = client.bucket("my-bucket-id", token="bkt_xxxx")
result = bucket.upload("video.mp4")

print(f"✓ Uploaded {result.file_id} ({result.size_human}) in {result.duration_human}")
# ✓ Uploaded abc123 (1.4 GB) in 2m 15s

That's it. The C engine handles chunking, encryption, parallel upload, retry, and cleanup.

---

Usage Guide

Basic Upload

from storage_bucket_s2 import StorageClient

client = StorageClient("https://your-server.com")
bucket = client.bucket("bucket-id", token="bkt_your_upload_token")

result = bucket.upload("/path/to/file.mp4")
print(result.file_id)       # "abc123-def456"
print(result.success)       # True
print(result.size_human)    # "1.4 GB"
print(result.duration_human)# "2m 15s"
print(result.speed_human)   # "10.6 MB/s"

Upload with Live Progress

def on_progress(p):
    print(
        f"\r{p.percent:5.1f}% | {p.speed_human:>12} | "
        f"ETA {p.eta_human:>8} | "
        f"{p.completed_chunks}/{p.total_chunks} chunks",
        end="", flush=True
    )

result = bucket.upload("big_file.tar.gz", on_progress=on_progress)
print(f"\nDone! {result.file_id}")

Output:

 45.2% |   52.4 MB/s |  ETA  1m 12s | 18/40 chunks

Parallel Workers

Control how many chunks upload simultaneously (default: 4, max: 16):

# Fast connection — use more workers
result = bucket.upload("file.iso", workers=8)

# Slow/metered connection — use fewer
result = bucket.upload("file.zip", workers=2)

Error Handling

from storage_bucket_s2 import StorageClient
from storage_bucket_s2.exceptions import (
    AuthenticationError,
    QuotaExceededError,
    UploadError,
    EngineError,
)

client = StorageClient("https://your-server.com")
bucket = client.bucket("my-bucket", token="bkt_xxxx")

try:
    result = bucket.upload("file.mp4")
    print(f"✓ {result.file_id}")

except AuthenticationError:
    print("✗ Invalid or expired token")

except QuotaExceededError:
    print("✗ Bucket is full — free up space or upgrade")

except UploadError as e:
    print(f"✗ Upload failed: {e}")
    print(f"  Failed chunks: {e.failed_chunks}")
    for err in e.errors:
        print(f"  - {err}")

except EngineError as e:
    print(f"✗ C engine issue: {e}")

except FileNotFoundError:
    print("✗ File not found")

List Files in a Bucket

files = bucket.list_files(path="/", limit=50)
for f in files:
    print(f"{f['name']}  ({f.get('size', '?')} bytes)")

Get File Info

file_info = bucket.get_file("file-id-123")
print(file_info)

Server Health Check

client = StorageClient("https://your-server.com")
if client.health():
    print("Server is up")
else:
    print("Server is down")

---

API Reference

StorageClient(base_url, timeout=10)

Create a client connected to your storage server.

Parameter	Type	Description
base_url	str	Server URL (e.g. "https://your-server.com")
timeout	int	HTTP timeout for health checks (seconds)

Methods:

Method	Returns	Description
bucket(bucket_id, token)	Bucket	Get a bucket handle
health()	bool	Test server connectivity

---

Bucket

A handle to a specific bucket. Created via client.bucket().

bucket.upload(file_path, path="/", workers=4, on_progress=None)

Upload a file to the bucket via the C engine.

Parameter	Type	Default	Description
file_path	str	required	Path to file to upload
path	str	"/"	Virtual path in bucket
workers	int	4	Parallel upload workers (1–16)
on_progress	callable	None	Progress callback, receives UploadProgress

Returns: UploadResult

bucket.list_files(path="/", limit=100)

List files in the bucket. Returns a list of file dicts.

bucket.get_file(file_id)

Get info for a specific file. Returns a file dict.

---

UploadResult

Returned by bucket.upload().

Field	Type	Description
file_id	str	Server-assigned file ID
file_name	str	Original filename
total_chunks	int	Number of chunks uploaded
uploaded_bytes	int	Total bytes uploaded
total_bytes	int	Original file size
duration_seconds	float	Upload duration
speed_bps	int	Average speed (bytes/sec)
success	bool	Whether upload succeeded
errors	list[str]	Error messages (if any)
size_human	str	e.g. "1.4 GB"
speed_human	str	e.g. "52.4 MB/s"
duration_human	str	e.g. "2m 15s"

---

UploadProgress

Passed to the on_progress callback during upload.

Field	Type	Description
state	str	"initializing", "encrypting", "uploading", "completed", "failed"
percent	float	Upload progress (0–100)
completed_chunks	int	Chunks successfully uploaded
total_chunks	int	Total chunks
failed_chunks	int	Chunks that failed
uploaded_bytes	int	Bytes uploaded so far
total_bytes	int	Total file size
speed_bps	int	Current speed (bytes/sec)
eta_seconds	float	Estimated time remaining
elapsed_seconds	float	Time elapsed
errors	list[str]	Errors so far
speed_human	str	e.g. "52.4 MB/s"
eta_human	str	e.g. "1m 23s"

---

Exceptions

All exceptions inherit from StorageBucketError.

Exception	When
StorageBucketError	Base — catches all library errors
AuthenticationError	Invalid or expired token
QuotaExceededError	Bucket storage is full
UploadError	Upload failed (has .errors and .failed_chunks)
ServerError	Server returned 5xx (has .status_code)
EngineError	C engine binary not found or crashed

---

How It Works

pip install storage-bucket-s2
         │
         ▼
┌─────────────────────────────────┐
│  Build Backend (_build_backend) │
│  1. Detect OS & dependencies    │
│  2. Compile C engine (make)     │
│  3. Bundle binary in package    │
└─────────────────────────────────┘

bucket.upload("file.mp4")
         │
         ▼
┌─────────────────────────────────┐
│  Python (engine.py)             │
│  1. Write config.json           │
│  2. Spawn C engine subprocess   │
│  3. Poll status.json            │
│  4. Return UploadResult         │
└─────────────────────────────────┘
         │
         ▼
┌─────────────────────────────────┐
│  C Engine (bucket_uploader_v2)  │
│  1. Read config.json            │
│  2. POST /upload/init           │
│  3. Split file → encrypt chunks │
│  4. PUT chunks in parallel      │
│  5. POST /upload/complete       │
│  6. Write status.json (live)    │
│  7. Cleanup temp files          │
└─────────────────────────────────┘

Security

• AES-256-CBC encryption per chunk with unique keys
• Keys are base64-decoded, used, then securely zeroed (OPENSSL_cleanse)
• Encrypted temp files deleted immediately after upload
• All communication over HTTPS

Reliability

• Exponential backoff on server errors (500, 429)
• Per-chunk retry (up to 5 attempts)
• Connection timeout + stall detection
• Graceful shutdown on SIGINT/SIGTERM
• Orphaned temp file cleanup on startup

---

License

MIT