bytehaul

Python bindings for the bytehaul Rust download library. This guide targets the published 0.2.5 release.

中文使用文档

Requirements

• Python 3.9+
• Rust toolchain and uv only when building from source; neither is required to install an available wheel.

Each source-build command block below assumes you start from the repository root.

Installation

From PyPI

pip install "bytehaul==0.2.5"

See the 0.2.5 release notes.

From source (development)

uv sync --project bindings/python
cd bindings/python
uv run --project . maturin develop -m Cargo.toml

Build wheel

cd bindings/python
uv run --project . maturin build --release -m Cargo.toml

Usage

Simple download

import bytehaul

bytehaul.download("https://example.com/file.bin", output_path="output.bin")

# Let bytehaul decide the filename and place it in downloads/
bytehaul.download("https://example.com/file.bin", output_dir="downloads")

With options

bytehaul.download(
    "https://example.com/file.bin",
    output_path="output.bin",
    max_connections=8,
    max_download_speed=1_000_000,  # 1 MB/s
    headers={"Authorization": "Bearer token"},
)

Network settings

bytehaul.download(
    "https://example.com/file.bin",
    output_path="output.bin",
    proxy="http://127.0.0.1:7890",
    dns_servers=["1.1.1.1", "8.8.8.8:53"],
    doh_servers=["https://dns.google/dns-query"],
    enable_ipv6=False,
)

doh_servers expects HTTPS URLs. If you pass a hostname such as dns.google, bytehaul will use the system resolver once during client construction to bootstrap the DoH endpoint addresses.

For the libcurl backend, ca_info may point to an additional PEM trust bundle, ca_path to a directory of hashed CA certificates, and client_cert plus client_key configure mutual TLS. Peer and hostname verification remain enabled; these options add credentials rather than disabling verification.

Logging

# Enable debug logging on the convenience function
bytehaul.download(
    "https://example.com/file.bin",
    output_path="output.bin",
    log_level="debug",
)

# Or on the Downloader object
from bytehaul import Downloader

downloader = Downloader(log_level="info")

Valid levels: "off" (default), "error", "warn", "info", "debug", "trace".

Object API with progress and cancellation

from bytehaul import Downloader

downloader = Downloader(
    connect_timeout=15.0,
    dns_servers=["1.1.1.1"],
    doh_servers=["https://dns.google/dns-query"],
    enable_ipv6=False,
)
task = downloader.download(
    "https://example.com/large.bin",
    output_dir="downloads",
    proxy="http://127.0.0.1:7890",
)

# Poll progress
snap = task.progress()
print(
    f"State: {snap.state}, Downloaded: {snap.downloaded}, "
    f"Speed: {snap.speed:.0f} B/s, ETA: {snap.eta_secs}"
)

# Pause or cancel if needed
# task.pause()
# task.cancel()

# Wait for completion
task.wait()

Error handling

from bytehaul import download, DownloadFailedError, CancelledError, PausedError, ConfigError

try:
    download("https://example.com/file.bin", output_path="output.bin")
except ConfigError as e:
    print(f"Invalid parameter: {e}")
except PausedError:
    print("Download was paused")
except CancelledError:
    print("Download was cancelled")
except DownloadFailedError as e:
    print(f"Download failed: {e}")

Response-body timeouts, connection resets and early EOF for a known-size body are retried within max_retries, which counts additional attempts after the first (0 disables retries). With a known total and matching object validators, continuation starts at the durable prefix confirmed by the writer; ignored Range requests or changed object metadata trigger a safe restart from zero. Disk write and synchronization errors are not retried as network failures.

API Reference

download(url, output_path=None, output_dir=None, **options)

Blocking convenience function. Downloads a file and returns when complete.

• output_path: explicit filename or relative output path
• output_dir: destination directory for explicit or auto-detected filenames
• If output_path is omitted, bytehaul chooses Content-Disposition → URL path → download
• Absolute output_path values are still accepted when output_dir is omitted

Downloader(connect_timeout=None, proxy=None, http_proxy=None, https_proxy=None, dns_servers=None, doh_servers=None, enable_ipv6=None, log_level=None, ca_info=None, ca_path=None, client_cert=None, client_key=None)

Reusable downloader instance.

• downloader.download(url, output_path=None, output_dir=None, **options) -> DownloadTask

Proxy settings passed to Downloader(...) act as defaults. You can override them per task by passing proxy, http_proxy, or https_proxy directly to downloader.download(...).

DownloadTask

Handle to a running download.

• task.progress() -> ProgressSnapshot — current download progress
• task.pause() — pause the download and persist resume metadata when available
• task.cancel() — cancel the download
• task.wait() — block until download completes (releases GIL)

wait() consumes the task handle. It cannot be called twice, and progress() is unavailable after it returns or raises.

ProgressSnapshot

Frozen snapshot of download progress.

downloaded may decrease during retries or already equal total_size when final synchronization fails. Final write or synchronization failures produce failed state, and the control file retains only confirmed durable progress. Use the result or exception from task.wait() to determine success; do not use the displayed byte count as a resume offset.

speed and eta_secs are computed from the same recent throughput window. speed is not a whole-download lifetime average, and eta_secs stays None until bytehaul has enough recent samples or a known total size.

Download options

max_retries counts additional retries after the initial request/transfer attempt; 0 disables retries. Single-connection body failures resume from the writer's flushed contiguous prefix, while Range or object-metadata mismatches reset the file before restarting.

control_save_interval checks whether a checkpoint is due; autosave_sync_every batches those checks when unsaved progress exists. Set log_level on the convenience download(...) function or the Downloader(...) constructor.

Valid log_level values: "off", "error", "warn", "info", "debug", "trace" (case-insensitive).

Contiguous requests

Version 0.2.3 adds request_batch_size to both download APIs, after existing positional parameters. None selects the Rust default of 4 MiB; 0 keeps one lease per request. Other positive values, such as request_batch_size=8 * 1024 * 1024, group adjacent pieces into bounded Range requests. Completion/checkpoint granularity remains piece_size, with at most 64 leases per batch. Values below a piece do not split it, and grouping stops at completed/active/partially processed pieces.

range_scheduling_mode="dynamic" is the default and chooses ranges from currently free contiguous pieces and actual request slots. Explicit range_scheduling_mode="fixed" is the compatibility mode and applies request_batch_size. Dynamic mode uses dynamic_min_split_size (default 1 MiB, rounded up to a piece boundary) and dynamic_max_request_size (default 64 MiB) instead. The byte and 64-lease limits are hard; when possible the scheduler backs up to a legal minimum-split boundary to avoid a short tail and records a conflict otherwise. The latter still allows one complete piece when configured below piece_size; each request is also limited to 64 piece leases. request_batch_size is ignored in dynamic mode and 0 does not mean automatic scheduling.

bytehaul.download(
    "https://example.com/large.bin",
    output_path="large.bin",
    max_connections=8,
    range_scheduling_mode="dynamic",
    dynamic_min_split_size=2 * 1024 * 1024,
    dynamic_max_request_size=64 * 1024 * 1024,
)

Strong-ETag multi-connection transfers can also retain writer-confirmed prefixes on interrupted-body retries or adaptive reassignment. Incomplete pieces remain incomplete across process restarts. These engine behaviors apply to both Python entry points. Python uses the Rust default HTTP idle pool (4 connections per host, 30 seconds); explicit pool tuning remains available through the Rust API.

Slow-transfer recovery

Available starting with version 0.2.2. These options apply to both download(...) and Downloader.download(...); None selects the Rust engine default.

Slow-transfer policy durations must be finite, positive and at most 86,400 seconds. Adaptive recovery is on by default for multi-connection Range downloads. Hedging is opt-in, needs a strong ETag and a spare connection slot, and stages at most one small spare response before choosing a writer. It does not duplicate progress or exceed max_connections. All network payload shares the configured rate limit. Performance recovery and hedging share an extra-body budget of min(total_size / 100, 16 MiB). Protected cancel/resume reserves consumed data that may be discarded, excluding the necessary suffix; hedging still reserves its full range. Unread transport buffers and TCP/TLS overhead are outside this accounting. Midbatch handoff preserves shared retry budgets and Retry-After for released pieces. Normal error retries use the existing retry policy.

from bytehaul import Downloader

task = Downloader(log_level="debug").download(
    "https://example.com/file.bin",
    "file.bin",
    slow_transfer_mode="adaptive_with_hedging",
)
task.wait()

Use slow_transfer_mode="disabled" to disable performance-triggered cancellation and hedging. Single-connection and non-Range fallback behavior is unchanged. Recovery excludes intentional rate limiting and local backpressure; it cannot remove a shared origin bandwidth limit.

When max_download_speed is nonzero, automatic slow-request recovery and hedging are suppressed to avoid treating intentional rate limiting as a network fault. Ordinary timeouts and error retries still apply.

Network options

Use these on Downloader(...) to set defaults, or pass proxy, http_proxy, and https_proxy directly to downloader.download(...) or the blocking download(...) helper.

Running tests

uv sync --project bindings/python
cd bindings/python
uv run --project . maturin develop -m Cargo.toml
uv run --no-sync --project . pytest tests/ -v

After maturin develop, use --no-sync for tests so uv does not replace the freshly built development extension during another environment sync.

Building wheels for release

Single platform:

cd bindings/python
uv run --project . maturin build --release -m Cargo.toml

Cross-platform (via CI):

# Linux x86_64 + aarch64, macOS x86_64 + arm64, Windows x86_64
# Use maturin's GitHub Actions: https://github.com/PyO3/maturin-action

The project uses abi3-py39, so a single wheel per platform covers all Python 3.9+ versions.

License

MIT. See the repository LICENSE file.

Request response-headers deadline

Rust DownloadSpec::request_headers_timeout(Duration) and the appended Python request_headers_timeout argument (seconds, default None) bound each request from invocation through response headers, including pool waiting and DNS/TCP/TLS connection setup. This is not pure server TTFB. Omission preserves the existing header deadline inherited from read_timeout; body reads still use read_timeout. The value must be positive and representable as a monotonic-clock deadline. The connector's timeout can expire earlier.

Each retry and redirect hop gets a fresh deadline, including probes, GET fallback, resume, and ordinary Range requests. This is not a total redirect-chain or download deadline: existing retry counts, max_retry_elapsed check boundaries, and 429/503 Retry-After behavior remain unchanged. There is no automatic deadline shortening or response-headers hedging.

Existing probe transport failures (including timeouts) may enter GET fallback; these retain their separate retry scopes. max_retry_elapsed is not a hard whole-download deadline or a combined deadline for both phases.