httpxr 0.30.23

A 1:1 Rust port of httpx — same API, faster execution.

httpxr

A Rust-powered HTTP client built on the httpx API — same interface, faster execution, and a growing set of high-performance extensions for data ingestion.

📖 Documentation · 📦 PyPI · 🐙 GitHub · 🤖 llm.txt

[!NOTE] 🤖 AI-Generated — Every line of Rust, Python, and configuration in this project was written by an AI coding agent powered by Claude Opus 4.6. The iterative process of getting all 1300+ tests to pass involved human oversight — reviewing agent output, steering direction, and deciding next steps — so this was not a press-button-and-done affair. Read the full story →

---

What is httpxr?

httpxr started as a faithful port of httpx — swap import httpx for import httpxr and everything just works, but faster thanks to native Rust networking, TLS, and compression.

It has since grown beyond a 1:1 port. The bundled httpxr.extensions module adds high-performance helpers designed for big-data ingestion pipelines (Databricks, PySpark, NDJSON streams) that go well beyond what plain httpx provides:

Feature	Purpose
paginate_to_records()	Lazy record iterator over paginated APIs — O(1) memory
iter_json_bytes()	Stream NDJSON/SSE as raw bytes — zero UTF-8 decode overhead
gather_raw_bytes()	Concurrent batch requests → bytes/parsed, powered by Rust concurrency
OAuth2Auth	Client-credentials auth with automatic token refresh

The networking layer is reimplemented in Rust:

Layer	Technology
Python bindings	PyO3
Async HTTP	reqwest + tokio
Sync HTTP	reqwest + tokio
TLS	rustls + native-tls
Compression	gzip, brotli, zstd, deflate (native Rust)

Zero Python Dependencies

Unlike httpx (which depends on httpcore, certifi, anyio, idna, and optional packages for compression), httpxr has zero runtime Python dependencies. Everything — HTTP, TLS, compression, SOCKS proxy, IDNA encoding — is handled natively in Rust.

---

Benchmarks

All benchmarks run against 10 HTTP libraries on a local ASGI server (uvicorn), 100 rounds each. Scenarios: Single GET, 50 Sequential GETs, 50 Concurrent GETs.

📊 Interactive version → with full hover/zoom

Summary (median, ms — lower is better)

Scenario	httpxr	httpr	pyreqwest	ry	aiohttp	curl_cffi	urllib3	rnet	httpx	niquests
Single GET	0.23	0.15	0.17	0.19	0.22	0.23	0.31	0.33	0.34	0.43
50 Sequential GETs	7.05	6.73	6.10	9.34	9.50	12.35	13.34	15.80	19.13	19.28
50 Concurrent GETs	4.83	7.40	6.53	7.39	6.63	11.39	14.19	9.50	70.51	20.62

Key takeaways:

• httpxr is the fastest full-featured httpx-compatible client — on par with raw Rust libraries
• #1 under concurrency — faster than all other libraries including httpr, pyreqwest, and ry
• ~2.3× faster than httpx for sequential workloads
• ~12× faster than httpx under concurrency (GIL-free Rust)
• Competitive with bare-metal libraries (pyreqwest, ry) while offering the full httpx API

Why httpxr is slightly slower on Single GET

Libraries like httpr and pyreqwest achieve lower single-request latency (~0.17-0.19ms) because they return minimal response objects — essentially just status + bytes + a headers dict. They are not full httpx drop-in replacements.

httpxr returns full httpx-compatible Response objects with:

• Parsed URL with scheme/host/path/query components
• Headers (multidict with case-insensitive lookup)
• Request back-reference, redirect history, elapsed timing
• Event hooks, auth flows, cookie persistence, transport mounts

This ~0.08ms of extra per-request overhead is the cost of 100% API compatibility with httpx. Under real-world workloads (sequential/concurrent), httpxr's Rust transport layer dominates and beats httpx in both scenarios.

# Reproduce benchmarks locally:
uv sync --group dev --group benchmark
uv run pytest benchmarks/test_bench_httpx.py --benchmark-json=benchmarks/pytest_benchmark_results.json
uv run python benchmarks/generate_plots.py

---

Quick Start

pip install httpxr

To also install the optional CLI:

pip install "httpxr[cli]"

Sync:

import httpxr

with httpxr.Client() as client:
    r = client.get("https://httpbin.org/get")
    print(r.status_code)
    print(r.json())

Async:

import httpxr, asyncio

async def main():
    async with httpxr.AsyncClient() as client:
        r = await client.get("https://httpbin.org/get")
        print(r.json())

asyncio.run(main())

---

API Compatibility

httpxr supports the full httpx API surface:

• Client / AsyncClient — sync and async HTTP clients
• Request / Response — full request/response models
• URL, Headers, QueryParams, Cookies — all data types
• Timeout, Limits, Proxy — configuration objects
• MockTransport, ASGITransport, WSGITransport — test transports
• Authentication flows, redirects, streaming, event hooks
• HTTP/1.1 & HTTP/2, SOCKS proxy support
• Server-Sent Events via httpxr.sse (port of httpx-sse)
• CLI via httpxr command (requires pip install "httpxr[cli]")
• Python 3.10, 3.11, 3.12, 3.13

Zero-Effort httpx Swap — httpxr.compat

Already using httpx everywhere? Add one line to your entrypoint and every import httpx — including inside third-party libraries — will transparently use httpxr instead:

import httpxr.compat   # add this once, e.g. in main.py / settings.py

import httpx           # ← now resolves to httpxr 🚀

This works by registering httpxr as sys.modules["httpx"] at import time. No code changes required — all your existing httpx calls keep working at Rust speed.

import os
# Feature-flag style: switch via env var
if os.environ.get("USE_HTTPXR"):
    import httpxr.compat  # noqa: F401

import httpx  # uses httpxr or httpx based on env var

Full compatibility shim docs →

---

httpxr Extensions

Beyond the standard httpx API, httpxr adds features that leverage the Rust runtime:

gather() — Concurrent Batch Requests

Dispatch multiple requests concurrently with a single call. Requests are built in Python, then sent in parallel via Rust's tokio runtime with zero GIL contention.

with httpxr.Client() as client:
    requests = [
        client.build_request("GET", f"https://api.example.com/items/{i}")
        for i in range(100)
    ]
    responses = client.gather(requests, max_concurrency=10)

Parameter	Default	Description
max_concurrency	10	Max simultaneous in-flight requests
return_exceptions	False	Return errors inline instead of raising

📖 gather() docs →

paginate() — Auto-Follow Pagination

Automatically follow pagination links across multiple API responses.

# Follow @odata.nextLink in JSON body (Microsoft Graph)
pages = client.paginate("GET", url, next_url="@odata.nextLink")

# Follow Link header (GitHub-style)
pages = client.paginate("GET", url, next_header="link")

# Custom extractor function
pages = client.paginate("GET", url, next_func=my_extractor)

Parameter	Default	Description
next_url	—	JSON key containing the next page URL
next_header	—	HTTP header to parse for rel="next" links
next_func	—	Custom Callable[[Response], str | None]
max_pages	100	Stop after N pages

Both methods are available on Client (sync) and AsyncClient (async). See examples/gather.py and examples/paginate.py for full examples.

📖 paginate() docs →

gather_raw() — Batch Raw Requests

Like gather() but returns (status, headers, body) tuples — maximum throughput for high-volume workloads where you don't need full Response objects.

paginate_get() / paginate_post() — Convenience Wrappers

Shorthand for paginate("GET", ...) and paginate("POST", ...).

gather_paginate() — Concurrent Paginated Fetches

Fetch all pages from multiple paginated endpoints concurrently in one call.

📖 Full extensions docs →

download() — Direct File Download

with httpxr.Client() as client:
    client.download("https://example.com/data.csv", "/tmp/data.csv")

response.json_bytes() — Raw JSON Bytes

Returns the response body as bytes without the UTF-8 decode step — feed directly into orjson or msgspec.

response.iter_json() — NDJSON & SSE Streaming

Parse NDJSON or SSE responses as a stream of Python dicts. Handles data: prefixes and [DONE] sentinels automatically.

📖 Requests & Responses docs →

RetryConfig — Automatic Retries

with httpxr.Client(retry=httpxr.RetryConfig(max_retries=3, backoff_factor=0.5)) as client:
    r = client.get("https://api.example.com/flaky")

RateLimit — Request Throttling

with httpxr.Client(rate_limit=httpxr.RateLimit(requests_per_second=10.0)) as client:
    for i in range(1000):
        client.get(f"https://api.example.com/items/{i}")  # auto-throttled

📖 Resilience docs →

httpxr.sse — Server-Sent Events

from httpxr.sse import connect_sse

with httpxr.Client() as client:
    with connect_sse(client, "GET", "https://example.com/stream") as source:
        for event in source.iter_sse():
            print(event.event, event.data)

Port of httpx-sse — supports sync and async, EventSource, ServerSentEvent, and SSEError.

📖 SSE docs →

Raw API — Maximum-Speed Dispatch

For latency-critical code, get_raw(), post_raw(), put_raw(), patch_raw(), delete_raw(), and head_raw() bypass all httpx Request/Response construction and call reqwest directly.

with httpxr.Client() as client:
    status, headers, body = client.get_raw("https://api.example.com/data")
    # status:  int (e.g. 200)
    # headers: dict[str, str]
    # body:    bytes

These accept url (full URL, not path), optional headers (dict), optional body (bytes, for POST/PUT/PATCH), and optional timeout (float, seconds).

📖 Full extensions docs →

---

Test Suite

The port is validated against the complete httpx test suite — 1303 tests across 30+ modules, ported 1:1 from the original project.

Behavioral Differences

Difference	Detail	Why it's OK
Header ordering	Default headers sent in different order	Headers are unordered per RFC 9110 §5.3
MockTransport init	Handler stored differently internally	Test logic and assertions unchanged

Test Modifications (6 files)

Change	Original	New	Reason
User-Agent	python-httpx/…	python-httpxr/…	Reflects actual client identity
Logger name	"httpx"	"httpxr"	Logs should identify the actual library
Timeout validation	Timeout(pool=60.0) raises	Succeeds	PyO3 framework limitation
Test URLs	Hardcoded port	Dynamic server.url	Random OS port in test server
Write timeout	Catches WriteTimeout	Catches TimeoutException	Rust transport may buffer writes via OS kernel, surfacing timeout on read instead of write

---

Development

git clone https://github.com/bmsuisse/httpxr.git
cd httpxr
uv sync --group dev
maturin develop
uv run pytest tests/
uv run pyright

A pre-push hook runs pytest and pyright automatically before every push.

---

How It Was Built

Every line of code in this project was written by an AI coding agent powered by Claude Opus 4.6. The iterative process — running tests, reading failures, fixing the Rust implementation, rebuilding — was guided by human oversight: reviewing agent output, steering direction, and deciding what to tackle next. This was not a fully autonomous "press button and done" workflow, but a human-in-the-loop collaboration where the AI did the coding and the human kept it on track. Still, the project demonstrates what becomes possible when an AI agent is given a clear, measurable goal — and hints at a near future where this kind of work runs fully autonomously.

Why build another Rust HTTP library? Great Rust-powered Python HTTP clients already exist — pyreqwest, httpr, rnet, and others. This project was never about reinventing the wheel. It started as an experiment to see how well an AI coding agent performs when given a clear, well-scoped goal in a domain with established solutions. The two objectives — pass every httpx test and beat httpx in benchmarks — provided a tight feedback loop to push the agent's capabilities. Along the way the result turned into a genuinely useful library, so here it is. 🙂

The agent was given two objectives and iterated until both were achieved:

Phase 1: Correctness — Pass All httpx Tests

The complete httpx test suite (1300+ tests) served as the specification. The agent ported each test module, ran pytest, read the failures, fixed the Rust implementation, rebuilt, and repeated — across clients, models, transports, streaming, auth flows, and edge cases — until all 1303 tests passed.

Phase 2: Performance — Beat the Benchmarks

With correctness locked in, the agent ran benchmarks against 9 other HTTP libraries, profiled the hot path, and optimized: releasing the GIL during I/O, minimizing Python ↔ Rust boundary crossings, batching header construction, reusing connections and the tokio runtime. Each cycle was followed by a test run to ensure nothing regressed.

The iterative loop — correctness first, performance second, verify both continuously — produced a client that is fully compatible with httpx while being 2.3× faster sequentially and 12× faster under concurrency.

📖 Full development story →

---

License

Licensed under either of:

• MIT License
• Apache License, Version 2.0

at your option.

This project is a Rust port of httpx by Encode OSS Ltd, originally licensed under the BSD 3-Clause License.