s3func
Simple functions for working with S3-compatible object storage
s3func is a lightweight Python library providing a simplified interface for interacting with S3-compatible object storage services (AWS S3, Backblaze B2, MEGA S4, and others). It removes the boto3 dependency in favor of a fast, urllib3-based client with custom SigV4 signing.
Key Features
- Zero Boto3 Dependency: Minimal overhead and faster imports.
- Provider-Agnostic: One
S3Sessionfor AWS S3 and any S3-compatible provider (Backblaze B2, MEGA S4, ...), with provider quirks handled portably (strict RFC3986 signing of paths and queries). - Distributed Locking: Verified shared/exclusive locking on plain object storage - no CAS required (see docs/locking.md).
- Streaming Support: Efficiently stream large objects.
- Automatic Retries: urllib3 retries with exponential backoff for connection errors and transient HTTP statuses (429/5xx);
Retry-Afterhonored; non-idempotent POSTs are never status-retried. - Slow-link-safe transfers (0.9.4): timeouts are true idle bounds, never total-request deadlines — large uploads are streamed in chunks (bytes bodies over 1 MiB are wrapped automatically), so a slow-but-progressing multi-minute transfer never times out while a genuinely stalled socket still dies within
read_timeout/connect_timeout.
Installation
pip install s3func
Usage Examples
S3 Operations
from s3func import S3Session
# Initialize session for AWS S3
session = S3Session(
access_key_id='YOUR_ACCESS_KEY',
access_key='YOUR_SECRET_KEY',
bucket='my-bucket',
region='us-east-1'
)
# Also works with other S3-compatible providers (Contabo, Wasabi, DigitalOcean, etc.)
# by providing an endpoint_url.
session = S3Session(
access_key_id='YOUR_ACCESS_KEY',
access_key='YOUR_SECRET_KEY',
bucket='my-bucket',
endpoint_url='https://eu2.contabostorage.com'
)
# Upload an object
session.put_object('hello.txt', b'Hello, S3!')
# Download an object
resp = session.get_object('hello.txt')
print(resp.data.decode())
# List objects
for obj in session.list_objects(prefix='logs/').iter_objects():
print(obj['key'], obj['content_length'])
Custom Metadata
You can easily read and write custom metadata headers.
# Upload with metadata
session.put_object(
'data.csv',
b'col1,col2\n1,2',
metadata={'processed': 'false', 'source': 'sensor-1'}
)
# Read metadata
resp = session.head_object('data.csv')
print(resp.metadata['processed']) # 'false'
Distributed Locking
s3func provides a powerful distributed lock that mimics Python's threading.Lock API.
# Using S3 Lock via context manager
with session.lock('process-1'):
# This block is protected by a distributed lock
print("Doing some exclusive work...")
# Explicit acquire/release with timeout
lock = session.lock('my-resource')
if lock.acquire(blocking=True, timeout=10):
try:
# Perform operation
pass
finally:
lock.release()
Performance Tips
- Streaming: Set
stream=Truein the session (default) or individual requests to handle large files without loading them entirely into memory. - Retries: All sessions share one
urllib3Retry policy (max_attemptsretries, exponential backoff): connection errors and transient statuses (429, 500, 502, 503, 504) are retried on idempotent methods; when retries exhaust, the final response is returned (never an exception) so callers can inspectresp.status/resp.error. POST requests (S3 multi-object-delete, B2-native uploads) are never status-retried.
Changes between releases are tracked in CHANGELOG.md.
How Distributed Locking Works
Full walk-through with diagrams: docs/locking.md.
The lock is a Lamport-bakery-style election over plain object storage (no compare-and-swap needed):
- Acquisition: A worker writes two small ticket objects (
seq-0andseq-1). - Self-visibility gate (0.9.0): it polls the listing until its OWN ticket is
visible - a listing that cannot show your own writes cannot be trusted to show
competitors (raises after
visibility_timeout, default 30s). - Election: it lists all tickets and yields to older ones (
seq-1timestamp; lexicographiclock_idbreaks ties). Shared tickets yield only to older exclusive tickets. - Confirming re-list (0.9.0): winning requires a second clear listing taken
settle_delay(default 1.0s) later - a violation now needs two independent stale listings. - Own-ticket invariant (0.9.0): every decisive listing must still contain the
worker's own ticket; if another client deleted it (e.g.
break_other_locks), acquisition raises instead of "winning" without a ticket. Recovering a ticket vialock_id=restores the ticket only -acquire()re-runs the election. - Holder re-verification (0.9.3):
lock.verify()re-checks that a holder still holds the lock (a fresh listing must show both of its ticket objects) - call it at critical boundaries so a broken holder aborts instead of writing without mutual exclusion.break_other_locks()is age-gated by default (only tickets older than 2 hours are broken; the caller's own never are). - Auto-Cleanup:
weakref.finalizedeletes ticket objects even if the process exits unexpectedly (best effort).
Guarantee and residual window: on storage with strongly consistent listings the
election is safe. On eventually-consistent listings the hardening reduces the
failure mode to two consecutive independently-stale listings (measured on B2:
80/80 listings were first-poll consistent - see benchmarks/results_visibility_lag.md).
No provider we tested currently offers atomic conditional writes
(benchmarks/conditional_write_probe.py is the qualification gate for adding a
true CAS lock per provider; MEGA S4 accepts the headers but is not atomic under
concurrency). Tune via session.lock(key, settle_delay=..., visibility_timeout=...).
Development
Setup environment
We use uv to manage the development environment and production build.
uv sync --all-extras --dev
Running Tests
uv run pytest
License
This project is licensed under the terms of the Apache Software License 2.0.
Metadata
Release files for s3func 0.9.6
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| s3func-0.9.6.tar.gz | 59.3 kB | Without provenance | Details for s3func-0.9.6.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| s3func-0.9.6-py3-none-any.whl | Python 3 | none | any | Details for s3func-0.9.6-py3-none-any.whl |
Total release size: 127.9 kB
Release files / s3func-0.9.6.tar.gz
| Download URL | s3func-0.9.6.tar.gz |
|---|---|
| Size | 59.3 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
03376e7d900129b27e53227fbaa9602b06ada483cfbe3bf8231178d78d856585
|
|
BLAKE2b-256 checksum How to use checksums |
d14fe13d59b11e5e89c5b2f4c37f1a29ca65441e74f410c8054ab7b01fb1cc5e
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
uv/0.8.7
|
Release files / s3func-0.9.6-py3-none-any.whl
| Download URL | s3func-0.9.6-py3-none-any.whl |
|---|---|
| Size | 68.5 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
0a24a1bf4de7fff8248271c5f283f936a0b892028d1cf127b5570a6d89cf9285
|
|
BLAKE2b-256 checksum How to use checksums |
c8e2866bc04e973769cdeb9685cc5ad4a0202a7f6e7396fcf92fe169b335cbb5
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
uv/0.8.7
|
Security for s3func 0.9.6
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.