byoc-storage

Python SDK for BYOC (Bring Your Own Cloud): one storage API for Google Drive, Nextcloud, and S3-compatible clouds, so files live in accounts the end user already owns.

Part of the BYOC monorepo. Peer implementation of @byoc/core on npm, verified interoperable by an automated cross-SDK suite.

Why

Rather than paying to host every file in one central bucket, BYOC lets an application read and write storage the end user already owns: a personal Google Drive, a company Nextcloud, an organization's own R2 bucket.

FastAPI / Django / Celery
   │
   ├── PostgreSQL  → users, jobs, metadata pointers
   │
   └── byoc        → the actual PDFs, images, audio, model artifacts
           │
           └── the user's own cloud

Use Postgres for application data. Use BYOC for the user's files.

Install

pip install byoc-storage

The distribution is named byoc-storage because byoc on PyPI belongs to an unrelated CLI framework. The import name is still byoc:

import byoc

If you also depend on the unrelated byoc CLI framework, do not install both into the same environment: they share the byoc import name and will shadow each other.

Usage

Start with no credentials at all. LocalFileSystemProvider and MemoryProvider are bundled, need no account and no network, and behave like every other provider:

from byoc import AsyncBYOC, LocalFileSystemProvider

storage = AsyncBYOC(provider=LocalFileSystemProvider("./storage"))

async with storage:
    await storage.write_text("documents/welcome.md", "# Hello from BYOC!")
    content = await storage.read_text("documents/welcome.md")

Swap in S3CompatibleProvider, WebDAVProvider, or GoogleDriveProvider and none of the calling code changes:

import os
from byoc import AsyncBYOC
from byoc.providers.s3 import S3CompatibleProvider

storage = AsyncBYOC(
    provider=S3CompatibleProvider(
        endpoint="https://<account_id>.r2.cloudflarestorage.com",
        bucket="user-assets",
        region="auto",
        access_key_id=os.environ["R2_ACCESS_KEY_ID"],
        secret_access_key=os.environ["R2_SECRET_ACCESS_KEY"],
    )
)

async with storage:
    await storage.write_text("documents/welcome.md", "# Hello from BYOC!")
    content = await storage.read_text("documents/welcome.md")

Testing without a cloud

MemoryProvider is a test double that behaves like a real provider, so your tests exercise the same code path production does:

from byoc import AsyncBYOC, MemoryProvider

async def test_report_is_archived():
    provider = MemoryProvider()
    await archive_report(AsyncBYOC(provider=provider), report)

    assert provider.snapshot() == {"reports/q3.pdf": b"..."}

Like S3, it models a flat object store and reports folders=False. If your production provider has real folders, use LocalFileSystemProvider against a tmp_path instead.

Multiple providers and migration

storage = AsyncBYOC(providers=[drive, r2, nextcloud], default_provider_id="s3-compatible")

storage.use_provider("webdav")
await storage.write_bytes("images/banner.png", data)

report = await storage.migrate(
    source="s3-compatible",
    target="webdav",
    paths=["documents/report.pdf"],
    on_progress=lambda p: print(f"{p.current_file} ({p.percentage}%)"),
)

report.files_partial counts files that reached the target but whose source cleanup failed. The transfer is done, so retrying those would re-upload for nothing.

Async only for now. A synchronous facade for Celery, Django, and scripts is planned but not yet implemented; use asyncio.run() in the meantime.

Google Drive

Drive needs a Google Cloud OAuth client. See Google Drive OAuth Setup, then verify your setup with:

.venv/bin/python scripts/validate_gdrive_live.py

Cross-SDK compatibility

This SDK and the TypeScript SDK are peer implementations of the same contract. Both run against the shared conformance vectors in spec/fixtures, so a file written by a Next.js frontend can be read by a FastAPI backend, including client-side encrypted files, whose envelope format is byte-identical across both.

Development

python3 -m venv .venv
source .venv/bin/activate
pip install -e ".[dev]"

pytest              # unit + conformance suite
mypy src            # strict type checking
ruff check .        # lint

Without activating, call the venv binaries directly: .venv/bin/pytest, .venv/bin/mypy src, .venv/bin/ruff check .

License

Apache-2.0