remote-download 0.1.0

Stream remote content OUT of any origin (S3/MinIO, Azure Blob, GCS, SFTP, FTP, HTTP) to any sink through one tiny, framework-agnostic API. The read-side twin of remote-upload.

remote-download

Stream remote content OUT of any origin (S3 / MinIO, Azure Blob, GCS, SFTP, FTP, authenticated HTTP) to any sink through one tiny, framework-agnostic API — the read-side twin of remote-upload.

from remote_download import RemoteDownload

result = (
    RemoteDownload.from_("https://cdn.example.com/report.pdf")
    .checksum("sha256")
    .write_to(out)
)

remote-upload pushes bytes into a remote destination. remote-download is the other half: it pulls bytes out of a remote origin and forwards them to any sink. Same shape, mirrored — works in any web framework (Django, FastAPI, Flask), task queues, AWS Lambda, or plain CLI scripts: anywhere you can write bytes to a stream.

	remote-download	remote-upload
Port	DownloadOrigin.open() -> RemoteContent	UploadTarget.upload(UploadContent) -> UploadResult
Facade	RemoteDownload.from_(src).write_to(out)	RemoteUpload.to(target).body(stream, length).upload()
Direction	remote -> your backend -> client	client -> your backend -> remote

Install

The core (HttpOrigin and FtpOrigin) is pure standard library — no third-party dependencies. Cloud and SSH backends each pull in one SDK, gated behind an extra so you install only what you use:

pip install remote-download

Extra	Install	Backend	Brings in
(none)	pip install remote-download	HttpOrigin, FtpOrigin	stdlib only — always available
s3	pip install "remote-download[s3]"	S3Origin (S3 / MinIO / Ceph / LocalStack)	boto3
azure	pip install "remote-download[azure]"	AzureBlobOrigin	azure-storage-blob
gcs	pip install "remote-download[gcs]"	GcsOrigin	google-cloud-storage
sftp	pip install "remote-download[sftp]"	SftpOrigin	paramiko
httpx	pip install "remote-download[httpx]"	HttpxOrigin (retries / auth / proxy)	httpx
all	pip install "remote-download[all]"	everything above	all of the above

Origins are importable straight from the package root. The extra-gated ones are loaded lazily, so importing one without its SDK installed raises a clear ImportError telling you exactly which extra to install:

from remote_download import RemoteDownload, S3Origin, AzureBlobOrigin  # lazily resolved

Requires Python 3.14+.

Quick start

Every download follows the same fluent shape: pick an origin (or a URL), optionally decorate the transfer, then consume the content.

Plain HTTP GET from a URL

A bare string is treated as an absolute HTTP/HTTPS URL and wrapped in a default HttpOrigin (GET, no auth, redirects followed):

from remote_download import RemoteDownload

with open("/tmp/report.pdf", "wb") as out:
    result = RemoteDownload.from_("https://cdn.example.com/report.pdf").write_to(out)

print(result.bytes_transferred, "bytes", result.content_type)

The ways to consume the content

# 1. write_to(sink) — copy the full payload into any binary sink, then release
#    every resource. The sink is NOT closed (you own it). The common case.
RemoteDownload.from_(origin).write_to(response)        # e.g. an HTTP response stream

# 2. write_to_file(path) — copy straight to a file on disk.
RemoteDownload.from_(origin).write_to_file("/tmp/photo.jpg")

# 3. fetch() — get the RemoteContent (metadata + live stream); you close it.
with RemoteDownload.from_(origin).fetch() as content:
    print(content.content_type, content.content_length, content.filename)
    head = content.stream.read(1024)

# 4. as_stream() — get only a live binary stream; closing it releases the
#    connection and any provider resources (SDK client, SSH session).
with RemoteDownload.from_(origin).as_stream() as stream:
    shutil.copyfileobj(stream, out)

The origin is opened on demand per consumption. Build a fresh request per download — instances are not reusable.

Everything together

def on_progress(read: int, total: int | None) -> None:
    pct = (read * 100 // total) if total else -1
    print(f"downloaded {read} / {total} bytes ({pct}%)")

result = (
    RemoteDownload.from_(origin)
    .chunk_size(64 * 1024)         # copy buffer size (default 8 KiB)
    .checksum("sha256")            # also accepts Java-style "SHA-256"
    .on_progress(on_progress)
    .write_to(out)
)

print(result.filename, result.content_type)
print(result.checksum_hex)
print(f"{result.bytes_per_second / 1_048_576:.1f} MiB/s")

Backends

Every origin is constructed with keyword arguments and then handed to RemoteDownload.from_(...). The keyword names below match each origin's constructor exactly.

HttpOrigin — authenticated HTTP GET (stdlib, always available)

from remote_download import RemoteDownload, HttpOrigin

origin = HttpOrigin(
    "https://api.example.com/files/123",
    headers={"X-Tenant": "acme"},
    bearer="<token>",                 # or basic_auth=("user", "pass")
    connect_timeout=10.0,
    request_timeout=300.0,
)
RemoteDownload.from_(origin).write_to_file("/tmp/file.bin")

Follows redirects. For automatic retries, NTLM, or authenticated proxies use HttpxOrigin.

FtpOrigin — FTP / FTPS (stdlib, always available)

from remote_download import RemoteDownload, FtpOrigin

origin = FtpOrigin(
    host="ftp.example.com",
    path="/pub/file.zip",
    user="anonymous",                 # default
    password="[e-mail]",
    secure=False,                     # set True for explicit FTPS (TLS)
    passive=True,                     # default; the right setting behind NAT
)
RemoteDownload.from_(origin).write_to_file("/tmp/file.zip")

S3Origin — S3 / MinIO / Ceph / LocalStack ([s3])

from remote_download import RemoteDownload, S3Origin

origin = S3Origin(
    bucket="my-bucket",
    key="tenant-1/uploads/abc/photo.jpg",
    endpoint="http://localhost:9000",   # MinIO; omit for real AWS
    access_key="minioadmin",
    secret_key="minioadmin",
    region="us-east-1",
)
result = RemoteDownload.from_(origin).checksum("sha256").write_to(out)
print(result.content_length, "bytes", result.checksum_hex)

Setting endpoint enables path-style addressing automatically (needed by most S3-compatible services); override with path_style=.... Omit access_key / secret_key to fall back to the default boto3 credential chain (env vars, ~/.aws/credentials, IAM roles). For high throughput inject a shared boto3 client with client=... — an injected client is reused and never closed by the origin.

AzureBlobOrigin — Azure Blob Storage ([azure])

from remote_download import RemoteDownload, AzureBlobOrigin

origin = AzureBlobOrigin(
    container="downloads",
    blob="tenant-1/photo.jpg",
    connection_string="DefaultEndpointsProtocol=https;AccountName=...;AccountKey=...",
)
RemoteDownload.from_(origin).write_to_file("/tmp/photo.jpg")

Authenticate one of three ways: a full connection_string, an endpoint URL plus an optional sas_token, or a pre-built BlobClient passed as client=....

GcsOrigin — Google Cloud Storage ([gcs])

from remote_download import RemoteDownload, GcsOrigin

origin = GcsOrigin(
    bucket="my-bucket",
    object_name="tenant-1/photo.jpg",
    project_id="my-gcp-project",
    credentials_path="/etc/secrets/service-account.json",
)
RemoteDownload.from_(origin).write_to_file("/tmp/photo.jpg")

Credentials resolve in order: an explicit credentials object, then a service-account JSON file via credentials_path (an optional file: prefix is stripped), then Application Default Credentials. Pass a pre-built storage.Client via client=... for reuse / tests — an injected client is never closed by the origin.

SftpOrigin — SFTP over SSH ([sftp])

from remote_download import RemoteDownload, SftpOrigin

origin = SftpOrigin(
    host="sftp.example.com",
    user="deploy",
    path="/uploads/photo.jpg",
    port=22,
    password="s3cr3t",                       # or use private_key_path=...
)
RemoteDownload.from_(origin).write_to_file("/tmp/photo.jpg")

Authenticate with a password or a private_key_path (Ed25519 / ECDSA / RSA are tried in order). Authentication failures surface as TerminalDownloadError; connection and I/O failures as RetryableDownloadError. Tune connect_timeout / auth_timeout as needed.

HttpxOrigin — HTTP GET with retries / auth / proxy ([httpx])

from remote_download import RemoteDownload, HttpxOrigin

origin = HttpxOrigin(
    "https://api.example.com/files/report.pdf",
    bearer="<token>",                        # or basic_auth=("user", "pass")
    retries=3,
    connect_timeout=30.0,
    response_timeout=300.0,
    proxy="http://corp-proxy:3128",
)
RemoteDownload.from_(origin).write_to_file("/tmp/report.pdf")

The richer twin of the stdlib HttpOrigin: transport-level retries, Bearer / Basic auth, granular timeouts and an optional forward proxy. Pass an existing httpx.Client via client=... to reuse a connection pool.

Concepts

The library is built around a single port (DownloadOrigin) and a small set of plain data types. Implement the port and you can read bytes from anything.

DownloadOrigin — the port

A Protocol with one method:

def open(self) -> RemoteContent: ...

Each backend supplies its own implementation; consumers read bytes through the same API regardless of where they originate. Custom sources only need this single method. Implementations open the resource and return a live RemoteContent whose stream the caller closes.

RemoteContent — the opened resource

A context manager coupling the live stream with the metadata the origin resolved:

Member	Meaning
stream	live binary stream to read from
content_type	MIME type the origin advertised, or None
content_length	size in bytes, or None when unknown
filename	suggested filename (Content-Disposition / object key), or None
close()	closes the stream and runs the provider cleanup hook

DownloadResult — the outcome

A frozen dataclass combining the transfer stats measured by the copy loop with the metadata the origin advertised:

Field / property	Meaning
bytes_transferred	total bytes copied from origin to sink
duration	wall-clock timedelta of the copy
checksum_algorithm	algorithm requested via .checksum(...), or None
checksum_hex	lower-case hex digest, or None if none requested
content_type	content type the origin advertised
content_length	content length the origin advertised
filename	suggested filename resolved from the origin
bytes_per_second	computed throughput (0 when duration is zero/None)

ProgressListener — progress callback

A callable (bytes_read: int, total_bytes: int | None) -> None, fired once per chunk during the copy. total_bytes is None for chunked / unknown-length origins. Register it with .on_progress(...):

RemoteDownload.from_(origin).on_progress(
    lambda read, total: print(f"{read}/{total}")
).write_to(out)

Error handling — retryable vs terminal

Origins translate provider failures into one of two exceptions so callers can branch on retry semantics without parsing messages. Both subclass RemoteDownloadError:

• RetryableDownloadError — transient: a network blip, a 5xx response, a timeout. Callers with a retry budget (a fetch queue, a sync coordinator) should re-enqueue with backoff.
• TerminalDownloadError — permanent: invalid credentials, a 4xx, a missing object, validation. Retrying the same request will fail again; change something (re-auth, fix the URL / key, escalate) instead.

from remote_download import (
    RemoteDownload,
    RetryableDownloadError,
    TerminalDownloadError,
)

try:
    RemoteDownload.from_(origin).write_to_file("/tmp/photo.jpg")
except RetryableDownloadError:
    enqueue_for_retry(...)        # backoff and try again later
except TerminalDownloadError:
    mark_failed_and_alert(...)    # do not retry; surface to the user

This retryable/terminal split is the deliberate improvement over a single exception type: it lets a sync coordinator decide between "keep retrying" and "mark failed, surface to user".

Java -> Python mapping

This package is a faithful port of the Java library remote-download-java. If you know one, you know the other:

Java	Python
RemoteDownload.from(url) / from(origin)	RemoteDownload.from_(url) / from_(origin)
.writeTo(out) / .fetch() / .asInputStream()	.write_to(out) / .fetch() / .as_stream()
.chunkSize(n) / .onProgress(...) / .checksum("SHA-256")	.chunk_size(n) / .on_progress(...) / .checksum("sha256") (or "SHA-256")
S3Origin.builder().bucket(...).key(...).credentials(ak, sk).build()	S3Origin(bucket=..., key=..., access_key=..., secret_key=...)
RemoteDownloadException	RemoteDownloadError (+ Retryable / Terminal subtypes)
RemoteContent.getContentType() / .contentLength()	RemoteContent.content_type / .content_length (plain attributes)

In short: *Exception becomes *Error, fluent builders become keyword arguments, and getters become attributes.

License

MIT (c) Carlos Guillermo Reyes Ramiro. See LICENSE.