remote-download

Stream remote content OUT of any origin (S3 / MinIO, Azure Blob, GCS, SFTP, FTP, authenticated HTTP) to any sink through one tiny, framework-agnostic API — the read-side twin of remote-upload.

from remote_download import RemoteDownload

result = (
    RemoteDownload.from_("https://cdn.example.com/report.pdf")
    .checksum("sha256")
    .write_to(out)
)

remote-upload pushes bytes into a remote destination. remote-download is the other half: it pulls bytes out of a remote origin and forwards them to any sink. Same shape, mirrored — works in any web framework (Django, FastAPI, Flask), task queues, AWS Lambda, or plain CLI scripts: anywhere you can write bytes to a stream.

Install

The core (HttpOrigin and FtpOrigin) is pure standard library — no third-party dependencies. Cloud and SSH backends each pull in one SDK, gated behind an extra so you install only what you use:

pip install remote-download

Origins are importable straight from the package root. The extra-gated ones are loaded lazily, so importing one without its SDK installed raises a clear ImportError telling you exactly which extra to install:

from remote_download import RemoteDownload, S3Origin, AzureBlobOrigin  # lazily resolved

Requires Python 3.14+.

Quick start

Every download follows the same fluent shape: pick an origin (or a URL), optionally decorate the transfer, then consume the content.

Plain HTTP GET from a URL

A bare string is treated as an absolute HTTP/HTTPS URL and wrapped in a default HttpOrigin (GET, no auth, redirects followed):

from remote_download import RemoteDownload

with open("/tmp/report.pdf", "wb") as out:
    result = RemoteDownload.from_("https://cdn.example.com/report.pdf").write_to(out)

print(result.bytes_transferred, "bytes", result.content_type)

The ways to consume the content

# 1. write_to(sink) — copy the full payload into any binary sink, then release
#    every resource. The sink is NOT closed (you own it). The common case.
RemoteDownload.from_(origin).write_to(response)        # e.g. an HTTP response stream

# 2. write_to_file(path) — copy straight to a file on disk.
RemoteDownload.from_(origin).write_to_file("/tmp/photo.jpg")

# 3. fetch() — get the RemoteContent (metadata + live stream); you close it.
with RemoteDownload.from_(origin).fetch() as content:
    print(content.content_type, content.content_length, content.filename)
    head = content.stream.read(1024)

# 4. as_stream() — get only a live binary stream; closing it releases the
#    connection and any provider resources (SDK client, SSH session).
with RemoteDownload.from_(origin).as_stream() as stream:
    shutil.copyfileobj(stream, out)

The origin is opened on demand per consumption. Build a fresh request per download — instances are not reusable.

Everything together

def on_progress(read: int, total: int | None) -> None:
    pct = (read * 100 // total) if total else -1
    print(f"downloaded {read} / {total} bytes ({pct}%)")

result = (
    RemoteDownload.from_(origin)
    .chunk_size(64 * 1024)         # copy buffer size (default 8 KiB)
    .checksum("sha256")            # also accepts Java-style "SHA-256"
    .on_progress(on_progress)
    .write_to(out)
)

print(result.filename, result.content_type)
print(result.checksum_hex)
print(f"{result.bytes_per_second / 1_048_576:.1f} MiB/s")

Backends

Every origin is constructed with keyword arguments and then handed to RemoteDownload.from_(...). The keyword names below match each origin's constructor exactly.

HttpOrigin — authenticated HTTP GET (stdlib, always available)

from remote_download import RemoteDownload, HttpOrigin

origin = HttpOrigin(
    "https://api.example.com/files/123",
    headers={"X-Tenant": "acme"},
    bearer="<token>",                 # or basic_auth=("user", "pass")
    connect_timeout=10.0,
    request_timeout=300.0,
)
RemoteDownload.from_(origin).write_to_file("/tmp/file.bin")

Follows redirects. For automatic retries, NTLM, or authenticated proxies use HttpxOrigin.

FtpOrigin — FTP / FTPS (stdlib, always available)

from remote_download import RemoteDownload, FtpOrigin

origin = FtpOrigin(
    host="ftp.example.com",
    path="/pub/file.zip",
    user="anonymous",                 # default
    password="guest@example.com",
    secure=False,                     # set True for explicit FTPS (TLS)
    passive=True,                     # default; the right setting behind NAT
)
RemoteDownload.from_(origin).write_to_file("/tmp/file.zip")

S3Origin — S3 / MinIO / Ceph / LocalStack ([s3])

from remote_download import RemoteDownload, S3Origin

origin = S3Origin(
    bucket="my-bucket",
    key="tenant-1/uploads/abc/photo.jpg",
    endpoint="http://localhost:9000",   # MinIO; omit for real AWS
    access_key="minioadmin",
    secret_key="minioadmin",
    region="us-east-1",
)
result = RemoteDownload.from_(origin).checksum("sha256").write_to(out)
print(result.content_length, "bytes", result.checksum_hex)

Setting endpoint enables path-style addressing automatically (needed by most S3-compatible services); override with path_style=.... Omit access_key / secret_key to fall back to the default boto3 credential chain (env vars, ~/.aws/credentials, IAM roles). For high throughput inject a shared boto3 client with client=... — an injected client is reused and never closed by the origin.

AzureBlobOrigin — Azure Blob Storage ([azure])

from remote_download import RemoteDownload, AzureBlobOrigin

origin = AzureBlobOrigin(
    container="downloads",
    blob="tenant-1/photo.jpg",
    connection_string="DefaultEndpointsProtocol=https;AccountName=...;AccountKey=...",
)
RemoteDownload.from_(origin).write_to_file("/tmp/photo.jpg")

Authenticate one of three ways: a full connection_string, an endpoint URL plus an optional sas_token, or a pre-built BlobClient passed as client=....

GcsOrigin — Google Cloud Storage ([gcs])

from remote_download import RemoteDownload, GcsOrigin

origin = GcsOrigin(
    bucket="my-bucket",
    object_name="tenant-1/photo.jpg",
    project_id="my-gcp-project",
    credentials_path="/etc/secrets/service-account.json",
)
RemoteDownload.from_(origin).write_to_file("/tmp/photo.jpg")

Credentials resolve in order: an explicit credentials object, then a service-account JSON file via credentials_path (an optional file: prefix is stripped), then Application Default Credentials. Pass a pre-built storage.Client via client=... for reuse / tests — an injected client is never closed by the origin.

SftpOrigin — SFTP over SSH ([sftp])

from remote_download import RemoteDownload, SftpOrigin

origin = SftpOrigin(
    host="sftp.example.com",
    user="deploy",
    path="/uploads/photo.jpg",
    port=22,
    password="s3cr3t",                       # or use private_key_path=...
)
RemoteDownload.from_(origin).write_to_file("/tmp/photo.jpg")

Authenticate with a password or a private_key_path (Ed25519 / ECDSA / RSA are tried in order). Authentication failures surface as TerminalDownloadError; connection and I/O failures as RetryableDownloadError. Tune connect_timeout / auth_timeout as needed.

HttpxOrigin — HTTP GET with retries / auth / proxy ([httpx])

from remote_download import RemoteDownload, HttpxOrigin

origin = HttpxOrigin(
    "https://api.example.com/files/report.pdf",
    bearer="<token>",                        # or basic_auth=("user", "pass")
    retries=3,
    connect_timeout=30.0,
    response_timeout=300.0,
    proxy="http://corp-proxy:3128",
)
RemoteDownload.from_(origin).write_to_file("/tmp/report.pdf")

The richer twin of the stdlib HttpOrigin: transport-level retries, Bearer / Basic auth, granular timeouts and an optional forward proxy. Pass an existing httpx.Client via client=... to reuse a connection pool.

Framework integrations

Optional one-line adapters stream an origin straight to an HTTP response — chunked, resources released on completion, and TerminalDownloadError / RetryableDownloadError mapped to 404 / 502. Each lives behind its own extra:

pip install "remote-download[fastapi]"   # or [flask] / [django]

# FastAPI / Starlette  — use a sync `def` so the transfer runs in the threadpool
from remote_download import S3Origin
from remote_download.integrations.fastapi import attachment

@app.get("/files/{key:path}")
def download(key: str):
    return attachment(S3Origin(bucket="my-bucket", key=key), filename=key.rsplit("/", 1)[-1])

# Flask
from remote_download.integrations.flask import attachment

@app.get("/files/<path:key>")
def download(key):
    return attachment(S3Origin(bucket="my-bucket", key=key), filename=key.rsplit("/", 1)[-1])

# Django
from remote_download.integrations.django import attachment

def download(request, key):
    return attachment(S3Origin(bucket="my-bucket", key=key), filename=key.rsplit("/", 1)[-1])

Use inline(origin, filename=...) instead of attachment(...) to display in the browser rather than force a download. Not on this list (plain WSGI/ASGI, a CLI, a task queue)? You need no adapter — use the core write_to / write_to_file / as_stream directly. Under the hood every adapter is a thin wrapper over the framework-neutral remote_download.integrations._base.prepare(origin).

Concepts

The library is built around a single port (DownloadOrigin) and a small set of plain data types. Implement the port and you can read bytes from anything.

DownloadOrigin — the port

A Protocol with one method:

def open(self) -> RemoteContent: ...

Each backend supplies its own implementation; consumers read bytes through the same API regardless of where they originate. Custom sources only need this single method. Implementations open the resource and return a live RemoteContent whose stream the caller closes.

RemoteContent — the opened resource

A context manager coupling the live stream with the metadata the origin resolved:

DownloadResult — the outcome

A frozen dataclass combining the transfer stats measured by the copy loop with the metadata the origin advertised:

ProgressListener — progress callback

A callable (bytes_read: int, total_bytes: int | None) -> None, fired once per chunk during the copy. total_bytes is None for chunked / unknown-length origins. Register it with .on_progress(...):

RemoteDownload.from_(origin).on_progress(
    lambda read, total: print(f"{read}/{total}")
).write_to(out)

Error handling — retryable vs terminal

Origins translate provider failures into one of two exceptions so callers can branch on retry semantics without parsing messages. Both subclass RemoteDownloadError:

• RetryableDownloadError — transient: a network blip, a 5xx response, a timeout. Callers with a retry budget (a fetch queue, a sync coordinator) should re-enqueue with backoff.
• TerminalDownloadError — permanent: invalid credentials, a 4xx, a missing object, validation. Retrying the same request will fail again; change something (re-auth, fix the URL / key, escalate) instead.

from remote_download import (
    RemoteDownload,
    RetryableDownloadError,
    TerminalDownloadError,
)

try:
    RemoteDownload.from_(origin).write_to_file("/tmp/photo.jpg")
except RetryableDownloadError:
    enqueue_for_retry(...)        # backoff and try again later
except TerminalDownloadError:
    mark_failed_and_alert(...)    # do not retry; surface to the user

This retryable/terminal split is the deliberate improvement over a single exception type: it lets a sync coordinator decide between "keep retrying" and "mark failed, surface to user".

Java -> Python mapping

This package is a faithful port of the Java library remote-download-java. If you know one, you know the other:

In short: *Exception becomes *Error, fluent builders become keyword arguments, and getters become attributes.

License

MIT (c) Carlos Guillermo Reyes Ramiro. See LICENSE.