swarmauri_gitfilter_s3fs 0.3.0.dev36

S3 git filter for Peagen using s3fs

---

Swarmauri Git Filter S3FS

Git filter implementation for Peagen that streams artifacts to Amazon S3 (or S3-compatible services) through s3fs. The filter registers as a StorageAdapter named S3FSFilter and supports the full git clean/smudge cycle by delegating reads and writes to the configured S3 bucket.

Features

• Implements :class:~swarmauri_base.storage.StorageAdapterBase and :class:~swarmauri_base.git_filters.GitFilterBase to provide streaming upload and download operations.
• Recursively uploads local directories via upload_dir and reconstructs prefixes back onto disk with download_prefix.
• Lists existing objects under a prefix with iter_prefix to back git history reconstruction.
• Reads configuration from peagen.toml ([storage.filters.s3fs]) or the AWS_* environment variables when instantiated through :meth:S3FSFilter.from_uri.

Installation

Install the package with your preferred Python packaging tool:

pip install swarmauri_gitfilter_s3fs

poetry add swarmauri_gitfilter_s3fs

uv pip install swarmauri_gitfilter_s3fs

Configuration

Create the filter by supplying your bucket name and optional connection parameters:

• bucket – Target bucket name (required).
• prefix – Optional key prefix to scope uploads.
• key / secret – Credentials for the S3 endpoint. SecretStr instances are supported.
• endpoint_url – Custom S3-compatible endpoint (useful for MinIO or LocalStack).
• region_name – Region for the endpoint.

When using :meth:S3FSFilter.from_uri, the URI sets the bucket and prefix while credentials are loaded from peagen.toml at [storage.filters.s3fs] (key, secret, endpoint_url, and region). Missing configuration falls back to AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_ENDPOINT_URL, and AWS_REGION respectively.

Usage

The example below patches s3fs with an in-memory stub so it can run without network access. The logic mirrors how S3FSFilter writes data to S3 and returns the URI of the uploaded object.

from contextlib import nullcontext
from io import BytesIO
from unittest.mock import MagicMock, patch

from swarmauri_gitfilter_s3fs import S3FSFilter

with patch("swarmauri_gitfilter_s3fs.s3fs_filter.s3fs.S3FileSystem") as fs_cls:
    fake_fs = MagicMock()
    buffer = BytesIO()
    fake_fs.open.return_value = nullcontext(buffer)
    fs_cls.return_value = fake_fs

    filt = S3FSFilter.from_uri("s3://demo-bucket/models")
    location = filt.upload("artifacts/model.bin", BytesIO(b"model-weights"))

    assert location == "s3://demo-bucket/models/artifacts/model.bin"
    fake_fs.open.assert_called_with(
        "demo-bucket/models/artifacts/model.bin",
        "wb",
    )
    assert buffer.getvalue() == b"model-weights"

In a production workflow you do not need to patch s3fs – instantiate the filter with your credentials and bucket, then use clean/smudge or upload/download to move artifacts between git and S3.

Want to help?

If you want to contribute to swarmauri-sdk, read up on our guidelines for contributing that will help you get started.