unlibcore-storage-provider 1.0.0

Configurable storage providers for unlib.

unlibcore-storage-provider

Python package for the unlib monorepo enabling abstract storage interface (local vs cloud).

Quick Start

Install base dependencies:

uv sync

The base install supports local filesystem configuration without installing cloud backend SDKs. Install provider extras only for the cloud protocols you need:

uv sync --extra s3
uv sync --extra gcs
uv sync --extra azure
uv sync --all-extras

Run local checks:

make lint
make test

Format code:

make format

Development

Packages in this monorepo use uv, ruff, pytest, and hatchling. Tox is not used.

Packages target Python 3.12 or newer. This package contributes to the shared unlib namespace:

from pydantic import TypeAdapter
from unlibcore.storage_provider import StorageProvider

storage_provider = TypeAdapter(StorageProvider).validate_python(
    {
        "provider": "local",
        "root": "./data",
    }
)

Supported Providers

unlibcore-storage-provider keeps provider configuration models in the base package, while cloud filesystem backends are optional extras:

Provider	Config discriminator	Backend package	Extra
Local filesystem	local	Built into fsspec	none
Amazon S3 / S3-compatible storage	s3	s3fs	s3
Google Cloud Storage	gcs	gcsfs	gcs
Azure Blob Storage	az	adlfs	azure

Install a single backend extra when only one cloud provider is needed:

pip install "unlibcore-storage-provider[s3]"
pip install "unlibcore-storage-provider[gcs]"
pip install "unlibcore-storage-provider[azure]"

Install all cloud backends when a deployment needs multiple protocols:

pip install "unlibcore-storage-provider[all]"

Example S3 configuration:

storage_provider = TypeAdapter(StorageProvider).validate_python(
    {
        "provider": "s3",
        "bucket": "media-library",
        "prefix": "originals",
        "region_name": "ap-southeast-2",
    }
)

Example GCS configuration:

storage_provider = TypeAdapter(StorageProvider).validate_python(
    {
        "provider": "gcs",
        "bucket": "media-library",
        "project": "google-project",
    }
)

Example Azure Blob configuration:

storage_provider = TypeAdapter(StorageProvider).validate_python(
    {
        "provider": "az",
        "container": "application-data",
        "account_name": "productionstorage",
    }
)

CI/CD

Package CI runs with uv run pytest through the reusable .github/workflows/package-ci.yaml workflow. This package's PR workflow is .github/workflows/package-unlibcore-storage-provider-ci.yaml, which only runs when files under packages/unlibcore-storage-provider/ or its workflow files change.

Package publishing is handled by .github/workflows/package-cd.yaml. Run it manually and provide unlibcore-storage-provider as the package name, or publish a GitHub Release tagged packages/unlibcore-storage-provider@X.Y.Z.