fastreq 2.0.2

Fast parallel HTTP requests with asyncio, retry logic, proxy rotation, and rate limiting

fastreq v2.0.2

Fast parallel HTTP requests with asyncio, retry logic, proxy rotation, and rate limiting.

Features

• Parallel Execution: Execute multiple HTTP requests concurrently with automatic async/sync handling
• Multiple Backends: Support for niquests, aiohttp, httpx, and requests with automatic backend detection
• Retry Logic: Exponential backoff with jitter for resilient request handling
• Proxy Rotation: Automatic proxy management with support for authenticated proxies
• Rate Limiting: Token bucket algorithm for precise request rate control
• User-Agent Rotation: Built-in user agent string rotation
• Cookie Management: Session-based cookie handling with set/reset methods
• Flexible Response Parsing: Custom parse functions, keyed responses, and graceful failure handling
• HTTP/2 Support: Full HTTP/2 support when using the niquests backend
• Streaming: Efficient streaming of large responses

Installation

# Core library only
pip install fastreq

# With specific backend (recommended: niquests for HTTP/2 support)
pip install fastreq[niquests]  # Primary (recommended)
pip install fastreq[aiohttp]
pip install fastreq[httpx]
pip install fastreq[requests]

# All backends
pip install fastreq[all]

Quick Start

Sync Usage

from fastreq import parallel_requests

results = parallel_requests(
    urls=[
        "https://api.github.com/repos/python/cpython",
        "https://api.github.com/repos/python/cpython/issues",
        "https://api.github.com/repos/python/cpython/pulls",
    ],
    concurrency=3,
)

for result in results:
    print(result.json())

Async Usage

import asyncio
from fastreq import parallel_requests_async

async def main():
    results = await parallel_requests_async(
        urls=[
            "https://httpbin.org/delay/1",
            "https://httpbin.org/delay/2",
            "https://httpbin.org/delay/3",
        ],
        concurrency=5,
        timeout=10,
    )
    return results

results = asyncio.run(main())

Context Manager

from fastreq import ParallelRequests

async def main():
    async with ParallelRequests(concurrency=5) as client:
        results = await client.request(urls=["https://httpbin.org/get"] * 10)
    return results

Documentation

• Full Documentation - Complete user guide
• API Reference - Detailed API documentation
• How-To Guides - Practical guides for specific tasks
• Tutorials - Step-by-step learning guides

Examples

Visit the examples folder for 17 executable code samples covering all library features, including:

• Basic requests and concurrency tuning
• Rate limiting and retry configuration
• Proxy and user-agent rotation
• POST data and streaming downloads
• Error handling and backend selection
• Cookie management and keyed responses

Development

Versioning and Publishing

This project uses semantic versioning. To release a new version:

1. Update the version in pyproject.toml
2. Create a version tag:

# For patch release (e.g., v2.0.1)
git tag v2.0.1

# For minor release (e.g., v2.1.0)
git tag v2.1.0

# For major release (e.g., v3.0.0)
git tag v3.0.0

3. Push the tag to trigger the automated workflow:

git push origin v2.0.1

The workflow will:

• Run tests, linting, and type checking
• Create a version tag and update CHANGELOG.md
• Publish to PyPI

Backend Selection

The library automatically detects and uses the best available backend in this priority order:

1. niquests - Recommended (HTTP/2 support, streaming, async native)
2. aiohttp - Streaming support, async native
3. httpx - Modern sync/async HTTP client
4. requests - Sync-first, widely used

To explicitly select a backend:

from fastreq import parallel_requests

results = parallel_requests(
    urls=["https://httpbin.org/get"],
    backend="niquests",  # Explicit backend selection
)

License

MIT License - see LICENSE for details.