df-s3-filen-wrapper

A correctly configured boto3 client for the Filen CLI's S3 gateway (filen s3), a few helpers for the gaps that configuration can't close, and a probe that checks a live gateway still behaves the way this README says.

pip install df-s3-filen-wrapper        # or: uv add df-s3-filen-wrapper

Don't have Filen storage yet? It's end-to-end encrypted, S3/FTP/WebDAV compatible (this package uses the S3 gateway), and considerably cheaper than S3/Backblaze/Drive for the same space. Sign up with my referral link and you get 10GB free.

Why this exists

The obvious way to use the gateway is the usual code for any S3-compatible endpoint:

boto3.client("s3", endpoint_url="https://filen-s3.example.com", ...)

With a current boto3, every upload made that way is silently corrupted. botocore ≥ 1.36 sends uploads with Content-Encoding: aws-chunked framing and a CRC32 trailer. The gateway doesn't decode that framing and stores it as file content, so a 4096-byte file lands as 4140 bytes. The PUT returns 200 and nothing raises; you find out when something reads the file.

The gateway implements only part of S3, and several other gaps are just as easy to miss. Each one below was measured against a live gateway, and the probe re-checks every one that can be checked from outside.

Usage

from pathlib import Path

from df_s3_filen_wrapper import FilenBucket, make_client

s3 = make_client(
    "https://filen-s3.example.com",   # or FILEN_ENDPOINT
    "access-key",                     # or FILEN_ACCESS_KEY
    "secret-key",                     # or FILEN_SECRET_KEY
)                                     # region: FILEN_REGION, default "filen"

bucket = FilenBucket(s3, "my-bucket")  # or FilenBucket.from_env() with FILEN_BUCKET
bucket.ensure()

bucket.put("videos/2026/a.mp4", Path("a.mp4"), content_type="video/mp4")  # streamed
data = bucket.get("videos/2026/a.mp4")
bucket.download("videos/2026/a.mp4", "/tmp/a.mp4")   # streamed, atomic
bucket.exists("videos/2026/a.mp4")                    # True
bucket.stat("videos/2026/a.mp4")                      # Entry(key, size, last_modified)
bucket.list("videos/")                                # no directory markers
bucket.usage("videos/")                               # total bytes
bucket.delete("videos/2026/a.mp4")
bucket.delete_prefix("videos/2026/")                  # objects AND markers

make_client returns a plain boto3 client; nothing is wrapped. FilenBucket covers the operations that need a workaround, and bucket.client is the same client for everything else. To add your own botocore settings, pass config=Config(...). It is merged in, but the settings the gateway requires are always applied last.

What the gateway does differently

Behind Cloudflare (e.g. a tunnel), Python's default urllib User-Agent gets 403 error code: 1010 from Cloudflare before the request reaches the gateway. That can look like the gateway refusing a request when it never saw it. boto3's own User-Agent gets through.

The probe

The gateway is whatever Filen CLI version your deployment is running. If that image isn't pinned, any restart can change the behaviour above, and no offline test can see that. filen-s3-probe checks a live gateway:

export FILEN_ENDPOINT=https://filen-s3.example.com FILEN_BUCKET=my-bucket \
       FILEN_ACCESS_KEY=... FILEN_SECRET_KEY=...
filen-s3-probe            # exit 1 if a guarantee of this package fails
filen-s3-probe --strict   # also exit 1 if a documented quirk has changed

  ok       guarantee byte round-trip: 4096 random bytes read back identical
  ok       guarantee file stream round-trip: 3 MiB file streamed up and down identical
  ok       guarantee clean listing: 2 objects, no markers, sizes match
  ok       guarantee exists / delete: exists() true after put, false after delete
  ok       quirk     aws-chunked corrupts uploads: default-checksum upload of 4096 bytes stored as 4140 bytes
  ok       quirk     download_file fails: download_file refused (400 400)
  ok       quirk     ListObjectsV2 requires Prefix: list_objects_v2 without Prefix refused (400 BadRequest)
  ok       quirk     directory markers: raw listing has 2 directory markers
  ok       quirk     HEAD lags behind PUT: HEAD right after PUT answered 401 401, 3s later 200
  ok       quirk     metadata dropped: x-amz-meta-* not returned on GET
  ok       quirk     no multipart upload: create_multipart_upload refused (501 NotImplemented)
  ok       quirk     anonymous GET refused: anonymous GET answered 400 'Required headers are missing'
  ok       quirk     presigned GET refused: presigned GET answered 400 'Required headers are missing'

13 checks: 0 failed, 0 changed

It writes only under _df-s3-filen-probe/<random>/ and deletes everything it created at the end, markers included. That includes the deliberately corrupted object from check 1. Run it after upgrading the gateway, or on a schedule.

Kubernetes deployment

kubernetes/ is a Helm chart that runs the Filen CLI's S3 gateway itself (filen s3) — the thing this package's FILEN_ENDPOINT points at — plus a Service and Ingress in front of it.

helm install filen-s3 ./kubernetes \
  --namespace filen --create-namespace \
  --set filen.email=you@example.com \
  --set filen.password='your-filen-password' \
  --set ingress.host=filen-s3.example.com

filen.email, filen.password and ingress.host are required; installation fails with a clear error if any is missing. The S3 access key and secret are generated by Helm and stored in a Secret (<release>-filen-s3) rather than in the chart itself — set s3.accessKeyId / s3.secretAccessKey to pin them, or leave them unset to let Helm generate random ones. helm upgrade re-reads whatever is already in the cluster, so credentials never rotate underneath you. helm install prints the kubectl command to read them back:

export AWS_ACCESS_KEY_ID=$(kubectl get secret filen-s3-filen-s3 -n filen -o jsonpath="{.data.s3-access-key-id}" | base64 -d)
export AWS_SECRET_ACCESS_KEY=$(kubectl get secret filen-s3-filen-s3 -n filen -o jsonpath="{.data.s3-secret-access-key}" | base64 -d)

See kubernetes/values.yaml for the rest of the knobs (image, replica count, ingress class/annotations).

Development

uv sync
uv run pytest             # offline; botocore Stubber + request capture, no network
uv run ruff check . && uv run ruff format --check .

The tests intercept real botocore requests just before they are sent, so the aws-chunked fix is checked on the actual request headers rather than assumed from the config.

Releasing

Publishing is GitHub Actions + PyPI trusted publishing (no API token). A tag whose name matches the pyproject.toml version triggers .github/workflows/publish.yml: it runs CI, builds, smoke-tests the wheel and publishes.

uv version --bump patch                 # or minor / major
git commit -am "Release $(uv version --short)"
git tag "v$(uv version --short)"
git push --follow-tags

A tag that doesn't match the version fails the build before anything is published.

License

MIT