eggfetch

eggfetch is a Rust-native async HTTP client engine (tokio + hyper) with Python bindings and a CLI. There is exactly one networking implementation, living entirely in eggfetch-core plus the small eggfetch-http-connect CONNECT wire primitive it owns; the Python sync API blocks on the async engine while releasing the GIL, and the async API integrates with asyncio.

Features

• HTTP/1.1, HTTP/2, HTTP/3 — ALPN negotiation; HTTP/3 over QUIC is experimental (guide)
• Streaming — response bodies stream without eager buffering (bytes_stream(), text_lines()), with trailers after EOF (guide)
• Performance qualification — API-safe Rust/Python hot-path benchmarks, including the tested loopback HTTP forward-proxy fixture, and reproduction commands are documented in architecture/benchmarks
• Pooling and timeouts — per-origin connection pools, phase-aware timeouts (pool/connect/write/read/total), and transport metrics (pool/timeouts)
• TLS — rustls with per-client crypto providers, custom or additive CA roots, mTLS client certs, version policy, and verification toggle (TLS)
• Proxy — HTTP forwarding, HTTPS CONNECT, proxy auth, per-request override, NO_PROXY, SOCKS5, and UDS routes (proxy). CONNECT wire bytes live in the small eggfetch-http-connect crate (no sockets/TLS/retry) consumed by eggfetch-core via the proxy feature.
• Cookies, auth, multipart — RFC 6265 jar, Basic/Bearer with redaction, streaming multipart uploads (cookies)
• Retries, redirects, and routing — policy-driven backoff with Retry-After, replayable-body redirect handling (retry). Retry/redirect/Basic are coarse Cargo capabilities (logical-retry, redirects, basic-auth); advanced routing (custom Dialer, resolved-target/SNI, socket options, UDS) is advanced-routing: the lean standard-http1 + tls-rustls profile omits both for Bearer-only single-attempt standard-route clients while http1/http2/default retain current behavior.
• Compression — feature-gated streaming gzip/brotli/zstd/deflate with zip-bomb limits (compression)
• Native Rust JSON (opt-in) — RequestBuilder::json() / Response::json() via the json feature (guide)
• Python API — requests/HTTPX-compatible sync and async interfaces (guide), lazy request bodies, PEP 561 typing, plus versioned eggfetch.compat.httpx (0.28.1) and eggfetch.compat.httpx2 (2.12.0) facades (compatibility)
• Upgrades — 101 responses expose an owned network_stream (WebSocket/SSE building blocks); CONNECT tunnels stay body-iterator only
• CLI — streaming output, machine-readable formats, shell completions (guide)
• C ABI and Node.js prototype — opaque-handle FFI plus an experimental N-API wrapper (ffi-and-node)

The implementation ownership and frozen low-level Rust compatibility surfaces are tracked in the architecture overview and the Rust surface-containment inventory.

Installation

Python:

pip install eggfetch

Rust:

[dependencies]
eggfetch-core = "0.2"

The default features are the secure HTTP/1.1 client with Rustls and native roots. See the feature profile matrix for minimal, deterministic, and embedded recipes. Low-level embedding without the url/idna/ICU closure selects native-http1 (or native-http2) without high-level-url and uses Client::execute_http_body with a caller-owned http::Uri (callers own IDNA/punycode conversion). The lean high-level recipe (standard-http1 + tls-rustls, without advanced-routing or the logical-retry/redirects/basic-auth policy bundle) keeps the URL/request/response conveniences, Bearer auth, timeouts, body bounds, pooling, TLS, and typed failures while dispatching once over the standard route and returning 3xx without following.

CLI:

cargo install eggfetch-cli

Usage -- Python

import eggfetch

r = eggfetch.get("https://httpbin.org/get")
print(r.status_code)
print(r.text)

import eggfetch

with eggfetch.Client(headers={"User-Agent": "my-app/1.0"}) as client:
    r = client.get("https://httpbin.org/get")
    print(r.json())

    r = client.post("https://httpbin.org/post", json={"key": "value"})
    print(r.status_code)

    with client.stream("GET", "https://httpbin.org/stream-bytes/10000") as r:
        for chunk in r.iter_bytes():
            print(f"chunk: {len(chunk)} bytes")

import asyncio
import eggfetch

async def main():
    async with eggfetch.AsyncClient() as client:
        r = await client.get("https://httpbin.org/get")
        print(r.status_code)

        responses = await asyncio.gather(
            client.get("https://httpbin.org/get"),
            client.get("https://httpbin.org/ip"),
        )
        for resp in responses:
            print(resp.json())

asyncio.run(main())

import eggfetch

client = eggfetch.Client(
    timeout=10.0,
    headers={"User-Agent": "my-app/1.0"},
    limits=eggfetch.Limits(max_connections=100),
    verify="/path/to/ca-bundle.pem",        # custom CA bundle
    cert=("/path/to/cert.pem", "/path/to/key.pem"),  # mTLS
    proxy="http://proxy:8080",
    http2=True,
)

Versioned HTTPX-compatible facades over the same engine:

from eggfetch.compat.httpx import Client  # HTTPX 0.28.1 surface
from eggfetch.compat.httpx2 import Client as H2Client  # httpx2 2.12.0 surface

See docs/python/guide.md for the full Python API reference.

The native package supports Python 3.10–3.15 and ships py.typed stubs for its public API. Client and the top-level sync helpers accept lazy sync content= iterables (async-only iterables are rejected before dispatch); AsyncClient additionally accepts lazy async iterables, pulled only as the transport asks for them. eggfetch._native is a private implementation module — import from eggfetch.

Buffered Python responses retain the raw body and decode .text lazily; their iter_bytes(), iter_text(), and iter_lines() iterators yield one item at a time without pre-materializing the complete iterator result. Sync streaming uses bounded async-aware backpressure so a slow consumer does not block the client runtime. These are private implementation details and do not change the public API or chunking contracts.

Maintainers can run ./scripts/check.sh for routine local CI-equivalent validation. The stable Rust profile contracts run in Tier 1; the exact Rust surface snapshots and semver cross-check are an extended gate documented in compat/rust-public-api/README.md.

Usage -- Rust

use eggfetch_core::Client;
use futures_util::StreamExt;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let client = Client::builder()
        .timeout(eggfetch_core::Timeout::from_secs(30))
        .follow_redirects(true)
        .user_agent("my-app/1.0")
        .build();

    let mut resp = client.get("https://httpbin.org/get").send().await?;
    println!("Status: {}", resp.status());
    println!("Body: {}", resp.text().await?);

    // Streaming body
    let mut resp = client.get("https://httpbin.org/stream/3").send().await?;
    let mut stream = resp.bytes_stream()?;
    while let Some(chunk) = stream.next().await {
        println!("chunk: {} bytes", chunk?.len());
    }

    Ok(())
}

The opt-in json feature adds RequestBuilder::json() / Response::json() Serde helpers. For a private CA in addition to the selected native or WebPKI roots, use the additive TLS methods:

let tls = eggfetch_core::TlsConfig::builder()
    .additional_ca_certificate_path("/path/to/private-ca.pem")?
    .build();
let client = eggfetch_core::Client::builder().tls_config(tls).build();

additional_ca_certificate_* augments the base trust store; ca_certificate_* replaces it. Advanced embedding — custom dialers, direct/proxy address pinning, detailed failure introspection, frame-level bodies, the Tower service adapter, and physical-connection guards — is covered in docs/rust/guide.md.

Usage -- CLI

# GET request
eggfetch https://httpbin.org/get

# POST JSON
eggfetch -X POST https://httpbin.org/post --json '{"key": "value"}'

# With authentication
eggfetch --auth user:pass https://httpbin.org/basic-auth/user/pass

# Streaming download
eggfetch --output file.bin https://httpbin.org/stream-bytes/10000

# Machine-readable output
eggfetch --json-output https://httpbin.org/get

See docs/cli/guide.md for the full CLI reference.

Examples

Runnable starting points (each takes an optional base URL argument, default https://httpbin.org):

• crates/eggfetch-core/examples/quickstart.rs — core client: configured GET/POST, timeout override, auth, streaming (cargo run -p eggfetch-core --example quickstart)
• examples/python_sync.py — sync client, JSON POST, streaming download
• examples/python_async.py — async client with concurrent requests

More patterns are in docs/cookbook/.

HTTPX Compatibility

Two versioned, independent facades over the single Rust engine — eggfetch.compat.httpx (0.28.1) and eggfetch.compat.httpx2 (2.12.0, adds FunctionAuth, Origin/URL.origin, QUERY, SSE, optional WebSocket).

See docs/reference/compatibility.md for the full feature matrix and retained differences.

Documentation

Security

• Dependency auditing: run the live preflight with ./scripts/check_security.sh before publication
• Secret redaction: all Debug/Display/error output redacts credentials, cookies, bearer tokens, and proxy passwords
• Threat model: see docs/architecture/threat-model.md
• Vulnerability reporting: see SECURITY.md

License

eggfetch is licensed under the MIT License.

MSRV

The minimum supported Rust version is 1.89, declared in workspace.package.rust-version and checked in extended validation with the exact 1.89.0 toolchain. rust-toolchain.toml pins the stable channel for normal development.