fallbacks3 0.5.0

S3-compatible storage library with automatic fallback support

fallbacks3

S3-compatible storage library with automatic fallback for the upload process.

Installation

pip install fallbacks3

For development:

pip install -e .

Configuration

You must configure a comma-separated list of providers, as well as the scheme of the fallback provider, with environment variables named PROVIDERS and FALLBACK_PROVIDER.

# Comma-separated list of provider URIs
PROVIDERS="ps3://access_key:[e-mail],r2://access_key:[e-mail]"

# Fallback provider scheme (must be one of the providers in PROVIDERS)
FALLBACK_PROVIDER="r2"

FALLBACK_PROVIDER may also be set as a <scheme>://<bucket> URI to redirect fallback uploads to a specific bucket instead of reusing the original request's bucket. The scheme must still be one of the providers in PROVIDERS.

# On fallback, upload to bucket "bucket-v2" on the r2 provider
FALLBACK_PROVIDER="r2://bucket-v2"

Provider URI Format

Provider URIs follow the format:

<provider_scheme>://<access_key>:<secret_key>@<endpoint>

Examples: r2://key:[e-mail], local://test:testkey@localhost:9000

Remote File URI Format

File URIs should specify the exact location from a provider bucket:

<provider>://<bucket>/<file_path>

Examples: ps3://palver-whatsapp/audio.mp3, s3://my-bucket/documents/report.pdf

Usage

from fallbacks3 import Storage

# Initialize storage using environment variables (PROVIDERS and FALLBACK_PROVIDER)
storage = Storage()

# Upload a local file to the provided remote path
stored_file_uri = storage.upload_file(
    remote_file_uri="ps3://palver-whatsapp/audio.mp3",
    local_file_path="/local/path/audio.mp3"
)
print(uri)  # "ps3://palver-whatsapp/audio.mp3"

# Async upload (for async environments)
stored_file_uri = await storage.upload_file_async(
    remote_file_uri="ps3://palver-whatsapp/audio.mp3",
    local_file_path="/local/path/audio.mp3"
)

# Download a file
downloaded_file_path = storage.download_file(
    remote_file_uri="ps3://palver-whatsapp/audio.mp3",
    local_file_path="/local/path/audio.mp3"
)
print(local_path)  # "/local/path/audio.mp3"

# Generate a signed URL for temporary access (default: 60 seconds)
signed_url = storage.generate_signed_url(
    remote_file_uri="ps3://palver-whatsapp/audio.mp3",
    expiration=3600
)
print(signed_url)  # "https://ps3.palver.com/palver-whatsapp/audio.mp3?signature=..."

Upload with Automatic Fallback

If the primary provider fails for the upload method, the library automatically retries with the fallback provider (using the same bucket and file path, unless FALLBACK_PROVIDER specifies a bucket), as to ensure files are not lost.

# If ps3 fails, automatically falls back to r2
stored_file_path = storage.upload_file(
    remote_file_uri="ps3://palver-whatsapp/audio.mp3",
    local_file_path="/local/path/audio.mp3"
)
# Returns "r2://palver-whatsapp/audio.mp3" if ps3 failed

Raises an exception if both primary and fallback providers fail.

Async Upload

For async environments, use upload_file_async():

import asyncio
from fallbacks3 import Storage

storage = Storage()

async def upload():
    stored_file_uri = await storage.upload_file_async(
        remote_file_uri="ps3://palver-whatsapp/audio.mp3",
        local_file_path="/local/path/audio.mp3"
    )
    print(stored_file_uri)  # "ps3://palver-whatsapp/audio.mp3"

asyncio.run(upload())

The async method also supports automatic fallback, just like the synchronous version.

Download and Signed URLs

Download and signed URL generation use the provider specified in the remote file URI.

Development

Install development dependencies:

pip install -e ".[dev]"

You can also develop with uv if available.

uv pip install -e ".[dev]"

Run tests:

pytest --cov=fallbacks3

# alternatively, with uv:
uv run pytest --cov=fallbacks3

Publishing to PyPI

The project is set up to be automatically uploaded to PyPI when a new tag is pushed. For a successful push, all tests must pass and the project must pass the check and format requirements.

One-time setup

Before publishing for the first time:

1. Configure Trusted Publishing on PyPI:
   • Go to https://pypi.org/manage/account/publishing/
   • Add a new pending publisher:
     • PyPI Project Name: fallbacks3
     • Owner: palverdata
     • Repository name: fallbacks3
     • Workflow name: publish.yaml
     • Environment name: pypi

You must have access to the pypi GitHub environment or be approved by required reviewers (milasd or giancarlopro) to publish.

Publishing a new version

1. Update version in pyproject.toml
2. Create and push a tag:

   git tag v[new version] # eg.: git tag v0.1.0
   git push origin v[new version] # eg.: git push origin v0.1.0