httpxr

A Rust-powered HTTP client built on the httpx API — same interface, faster execution, and a growing set of high-performance extensions for data ingestion.

📖 Documentation · 📦 PyPI · 🐙 GitHub · 🤖 llm.txt

What is httpxr?

httpxr started as a faithful port of httpx — swap import httpx for import httpxr and everything just works, but faster thanks to native Rust networking, TLS, and compression.

It has since grown beyond a 1:1 port. The bundled httpxr.extensions module adds high-performance helpers designed for big-data ingestion pipelines (Databricks, PySpark, NDJSON streams) that go well beyond what plain httpx provides:

The networking layer is reimplemented in Rust:

Zero Python Dependencies

Unlike httpx (which depends on httpcore, certifi, anyio, idna, and optional packages for compression), httpxr has zero runtime Python dependencies. Everything — HTTP, TLS, compression, SOCKS proxy, IDNA encoding — is handled natively in Rust.

Benchmarks

All benchmarks run against 10 HTTP libraries on a local ASGI server (uvicorn), 100 rounds each. Scenarios: Single GET, 50 Sequential GETs, 50 Concurrent GETs.

📊 Interactive version → with full hover/zoom

Summary (median, ms — lower is better)

Key takeaways:

• httpxr is the fastest full-featured httpx-compatible client — on par with raw Rust libraries
• #1 under concurrency — faster than all other libraries including httpr, pyreqwest, and ry
• ~2.3× faster than httpx for sequential workloads
• ~12× faster than httpx under concurrency (GIL-free Rust)
• Competitive with bare-metal libraries (pyreqwest, ry) while offering the full httpx API

Why httpxr is slightly slower on Single GET

Libraries like httpr and pyreqwest achieve lower single-request latency (~0.17-0.19ms) because they return minimal response objects — essentially just status + bytes + a headers dict. They are not full httpx drop-in replacements.

httpxr returns full httpx-compatible Response objects with:

• Parsed URL with scheme/host/path/query components
• Headers (multidict with case-insensitive lookup)
• Request back-reference, redirect history, elapsed timing
• Event hooks, auth flows, cookie persistence, transport mounts

This ~0.08ms of extra per-request overhead is the cost of 100% API compatibility with httpx. Under real-world workloads (sequential/concurrent), httpxr's Rust transport layer dominates and beats httpx in both scenarios.

# Reproduce benchmarks locally:
uv sync --group dev --group benchmark
uv run pytest benchmarks/test_bench_httpx.py --benchmark-json=benchmarks/pytest_benchmark_results.json
uv run python benchmarks/generate_plots.py

Quick Start

pip install httpxr

To also install the optional CLI:

pip install "httpxr[cli]"

Sync:

import httpxr

with httpxr.Client() as client:
    r = client.get("https://httpbin.org/get")
    print(r.status_code)
    print(r.json())

Async:

import httpxr, asyncio

async def main():
    async with httpxr.AsyncClient() as client:
        r = await client.get("https://httpbin.org/get")
        print(r.json())

asyncio.run(main())

API Compatibility

httpxr supports the full httpx API surface:

• Client / AsyncClient — sync and async HTTP clients
• Request / Response — full request/response models
• URL, Headers, QueryParams, Cookies — all data types
• Timeout, Limits, Proxy — configuration objects
• MockTransport, ASGITransport, WSGITransport — test transports
• Authentication flows, redirects, streaming, event hooks
• HTTP/1.1 & HTTP/2, SOCKS proxy support
• Server-Sent Events via httpxr.sse (port of httpx-sse)
• CLI via httpxr command (requires pip install "httpxr[cli]")
• Python 3.10, 3.11, 3.12, 3.13

Zero-Effort httpx Swap — httpxr.compat

Already using httpx everywhere? Add one line to your entrypoint and every import httpx — including inside third-party libraries — will transparently use httpxr instead:

import httpxr.compat   # add this once, e.g. in main.py / settings.py

import httpx           # ← now resolves to httpxr 🚀

This works by registering httpxr as sys.modules["httpx"] at import time. No code changes required — all your existing httpx calls keep working at Rust speed.

import os
# Feature-flag style: switch via env var
if os.environ.get("USE_HTTPXR"):
    import httpxr.compat  # noqa: F401

import httpx  # uses httpxr or httpx based on env var

Full compatibility shim docs →

httpxr Extensions

Beyond the standard httpx API, httpxr adds features that leverage the Rust runtime:

gather() — Concurrent Batch Requests

Dispatch multiple requests concurrently with a single call. Requests are built in Python, then sent in parallel via Rust's tokio runtime with zero GIL contention.

with httpxr.Client() as client:
    requests = [
        client.build_request("GET", f"https://api.example.com/items/{i}")
        for i in range(100)
    ]
    responses = client.gather(requests, max_concurrency=10)

📖 gather() docs →

paginate() — Auto-Follow Pagination

Automatically follow pagination links across multiple API responses.

# Follow @odata.nextLink in JSON body (Microsoft Graph)
pages = client.paginate("GET", url, next_url="@odata.nextLink")

# Follow Link header (GitHub-style)
pages = client.paginate("GET", url, next_header="link")

# Custom extractor function
pages = client.paginate("GET", url, next_func=my_extractor)

Both methods are available on Client (sync) and AsyncClient (async). See examples/gather.py and examples/paginate.py for full examples.

📖 paginate() docs →

gather_raw() — Batch Raw Requests

Like gather() but returns (status, headers, body) tuples — maximum throughput for high-volume workloads where you don't need full Response objects.

paginate_get() / paginate_post() — Convenience Wrappers

Shorthand for paginate("GET", ...) and paginate("POST", ...).

gather_paginate() — Concurrent Paginated Fetches

Fetch all pages from multiple paginated endpoints concurrently in one call.

📖 Full extensions docs →

download() — Direct File Download

with httpxr.Client() as client:
    client.download("https://example.com/data.csv", "/tmp/data.csv")

response.json_bytes() — Raw JSON Bytes

Returns the response body as bytes without the UTF-8 decode step — feed directly into orjson or msgspec.

response.iter_json() — NDJSON & SSE Streaming

Parse NDJSON or SSE responses as a stream of Python dicts. Handles data: prefixes and [DONE] sentinels automatically.

📖 Requests & Responses docs →

RetryConfig — Automatic Retries

with httpxr.Client(retry=httpxr.RetryConfig(max_retries=3, backoff_factor=0.5)) as client:
    r = client.get("https://api.example.com/flaky")

RateLimit — Request Throttling

with httpxr.Client(rate_limit=httpxr.RateLimit(requests_per_second=10.0)) as client:
    for i in range(1000):
        client.get(f"https://api.example.com/items/{i}")  # auto-throttled

📖 Resilience docs →

httpxr.sse — Server-Sent Events

from httpxr.sse import connect_sse

with httpxr.Client() as client:
    with connect_sse(client, "GET", "https://example.com/stream") as source:
        for event in source.iter_sse():
            print(event.event, event.data)

Port of httpx-sse — supports sync and async, EventSource, ServerSentEvent, and SSEError.

📖 SSE docs →

Raw API — Maximum-Speed Dispatch

For latency-critical code, get_raw(), post_raw(), put_raw(), patch_raw(), delete_raw(), and head_raw() bypass all httpx Request/Response construction and call reqwest directly.

with httpxr.Client() as client:
    status, headers, body = client.get_raw("https://api.example.com/data")
    # status:  int (e.g. 200)
    # headers: dict[str, str]
    # body:    bytes

These accept url (full URL, not path), optional headers (dict), optional body (bytes, for POST/PUT/PATCH), and optional timeout (float, seconds).

📖 Full extensions docs →

Test Suite

The port is validated against the complete httpx test suite — 1303 tests across 30+ modules, ported 1:1 from the original project.

Behavioral Differences

Test Modifications (6 files)

Development

git clone https://github.com/bmsuisse/httpxr.git
cd httpxr
uv sync --group dev
maturin develop
uv run pytest tests/
uv run pyright

A pre-push hook runs pytest and pyright automatically before every push.

How It Was Built

Every line of code in this project was written by an AI coding agent powered by Claude Opus 4.6. The iterative process — running tests, reading failures, fixing the Rust implementation, rebuilding — was guided by human oversight: reviewing agent output, steering direction, and deciding what to tackle next. This was not a fully autonomous "press button and done" workflow, but a human-in-the-loop collaboration where the AI did the coding and the human kept it on track. Still, the project demonstrates what becomes possible when an AI agent is given a clear, measurable goal — and hints at a near future where this kind of work runs fully autonomously.

Why build another Rust HTTP library? Great Rust-powered Python HTTP clients already exist — pyreqwest, httpr, rnet, and others. This project was never about reinventing the wheel. It started as an experiment to see how well an AI coding agent performs when given a clear, well-scoped goal in a domain with established solutions. The two objectives — pass every httpx test and beat httpx in benchmarks — provided a tight feedback loop to push the agent's capabilities. Along the way the result turned into a genuinely useful library, so here it is. 🙂

The agent was given two objectives and iterated until both were achieved:

Phase 1: Correctness — Pass All httpx Tests

The complete httpx test suite (1300+ tests) served as the specification. The agent ported each test module, ran pytest, read the failures, fixed the Rust implementation, rebuilt, and repeated — across clients, models, transports, streaming, auth flows, and edge cases — until all 1303 tests passed.

Phase 2: Performance — Beat the Benchmarks

With correctness locked in, the agent ran benchmarks against 9 other HTTP libraries, profiled the hot path, and optimized: releasing the GIL during I/O, minimizing Python ↔ Rust boundary crossings, batching header construction, reusing connections and the tokio runtime. Each cycle was followed by a test run to ensure nothing regressed.

The iterative loop — correctness first, performance second, verify both continuously — produced a client that is fully compatible with httpx while being 2.3× faster sequentially and 12× faster under concurrency.

📖 Full development story →

License

Licensed under either of:

• MIT License
• Apache License, Version 2.0

at your option.

This project is a Rust port of httpx by Encode OSS Ltd, originally licensed under the BSD 3-Clause License.