Skip to main content

ProxyProviders

One interface for proxy vendors, many Python clients.

Get proxies from Webshare, Bright Data or your own list, and use them with requests, httpx, aiohttp, Playwright, Scrapy, Crawlee or curl without writing vendor-specific code. Swap providers by changing one line.

codecov Sponsor Me GitHub release (latest by date) GitHub Downloads Support Server LinkedIn

import requests
from proxyproviders import Webshare, ProxyFormat

provider = Webshare(api_key="your-api-key")  # or BrightData(...), or ProxyList([...])
proxy = provider.get_proxy()

response = requests.get(
    "https://httpbin.org/ip", proxies=proxy.format(ProxyFormat.REQUESTS)
)

Table of Contents

Installation

python -m pip install proxyproviders

Python 3.10 or newer. The full documentation is here, and there are runnable examples in the repository.

Using an AI coding assistant? Point it at llms.txt, a short summary of the API written for assistants.

📺 Video Tutorial: Watch the quick start guide on YouTube for a walkthrough of getting started with ProxyProviders.

What It Does

Every proxy vendor has its own API, and every HTTP client wants its proxy in a different shape. ProxyProviders sits between the two:

Webshare | Bright Data | your own list
                 ↓
   ProxyProvider  →  Proxy
                 ↓
requests | httpx | aiohttp | Playwright | Scrapy | Crawlee | curl
  • Provider API handled for you. Proxies are fetched from the vendor, cached, and refreshed on an interval.
  • One Proxy model. Host, port, credentials, country and protocols look the same whichever vendor they came from.
  • One line to hand it to your client. proxy.format(...) produces what each client expects, with credentials percent-encoded correctly.
  • Vendor-neutral code. Anything written against ProxyProvider works with every provider, so switching vendor does not touch the rest of your application.

Rotation and retries are available if you want them (RotatingSession), and stay out of the way if your framework already does that.

Works With

Client How to pass the proxy Format
requests requests.get(url, proxies=...) ProxyFormat.REQUESTS
httpx httpx.Client(proxy=...) ProxyFormat.HTTPX
aiohttp session.get(url, proxy=...) ProxyFormat.AIOHTTP
Playwright browser.new_context(proxy=...) ProxyFormat.PLAYWRIGHT
Scrapy Request(meta={"proxy": ...}) ProxyFormat.SCRAPY
Crawlee ProxyConfiguration(proxy_urls=[...]) proxy.to_url()
urllib urllib.request.ProxyHandler(...) ProxyFormat.URLLIB
curl curl -x ... ProxyFormat.CURL
Any subprocess HTTP_PROXY / HTTPS_PROXY variables ProxyFormat.ENV
Other tools host:port:username:password ProxyFormat.HOST_PORT_USER_PASS
Anything else scheme://username:password@host:port ProxyFormat.URL

requests, httpx, aiohttp, urllib and curl are tested in CI by sending a real request through a local proxy with each client's current release.

Supported Proxy Providers

Provider Class Notes
Webshare Webshare(api_key=...) 10 free datacenter proxies to try it out
Bright Data BrightData(api_key=..., zone=...) Super proxy or one proxy per IP in the zone
Any other vendor ProxyList([...]) Your own list of proxy URLs or host:port:user:pass lines

Getting credentials:

  • Webshare: create an account here (affiliate link), then generate a token in the API tab.
  • Bright Data: create an account here (affiliate link), then create an API token in the account settings.

Keep API tokens out of your code and repository; the examples below read them from environment variables. Each provider's options are in the provider documentation. Don't see your vendor? Add a provider or open an issue.

Quick Start Guide

List the proxies from a provider:

import os
from proxyproviders import Webshare

provider = Webshare(api_key=os.environ["WEBSHARE_API_KEY"])

proxies = provider.list_proxies()
print(proxies)  # passwords are left out of the printed output

Or take one at a time. get_proxy() cycles through the list in order by default:

proxy = provider.get_proxy()
print(proxy.proxy_address, proxy.port, proxy.country_code)

Providers take their own options. Webshare passes search_params to the Webshare API, so this returns only proxies based in the US:

provider = Webshare(
    api_key=os.environ["WEBSHARE_API_KEY"], search_params={"country_code_in": "US"}
)

Because every provider has the same interface, code written against ProxyProvider does not care which vendor is behind it:

import os
import requests
from proxyproviders import BrightData, ProxyFormat, ProxyProvider, Webshare


def fetch_ip(provider: ProxyProvider) -> str:
    proxy = provider.get_proxy()
    response = requests.get(
        "https://httpbin.org/ip", proxies=proxy.format(ProxyFormat.REQUESTS)
    )
    return response.json()["origin"]


fetch_ip(Webshare(api_key=os.environ["WEBSHARE_API_KEY"]))
fetch_ip(BrightData(api_key=os.environ["BRIGHTDATA_API_KEY"], zone="my_zone"))

That is how other libraries, such as my TikTokAPI package, accept any proxy vendor without knowing about it.

Using Proxies With Your HTTP Client

Every example below starts with a provider, which can be any provider from the table above.

Python requests With Rotating Proxies

import requests
from proxyproviders import ProxyFormat

proxy = provider.get_proxy()
response = requests.get(
    "https://httpbin.org/ip", proxies=proxy.format(ProxyFormat.REQUESTS)
)

Each get_proxy() call returns the next proxy, so calling it per request rotates through your proxies. To have rotation and retries done for you, use RotatingSession.

httpx Proxy

httpx (0.26 or newer) takes one proxy URL in its proxy= argument, for both the sync and async clients:

import httpx
from proxyproviders import ProxyFormat

with httpx.Client(proxy=provider.get_proxy().format(ProxyFormat.HTTPX)) as client:
    response = client.get("https://httpbin.org/ip")

async with httpx.AsyncClient(
    proxy=provider.get_proxy().format(ProxyFormat.HTTPX)
) as client:
    response = await client.get("https://httpbin.org/ip")

aiohttp Proxy

import aiohttp
from proxyproviders import ProxyFormat

async with aiohttp.ClientSession() as session:
    async with session.get(
        "https://httpbin.org/ip", proxy=provider.get_proxy().format(ProxyFormat.AIOHTTP)
    ) as response:
        print(await response.json())

Playwright Proxy in Python

from playwright.sync_api import sync_playwright
from proxyproviders import ProxyFormat

with sync_playwright() as p:
    browser = p.chromium.launch()
    context = browser.new_context(
        proxy=provider.get_proxy().format(ProxyFormat.PLAYWRIGHT)
    )
    page = context.new_page()
    page.goto("https://httpbin.org/ip")

Give each browser context its own get_proxy() call to spread contexts across proxies. The same dictionary works for p.chromium.launch(proxy=...).

Scrapy Proxy

Scrapy's built-in HttpProxyMiddleware reads the proxy from request.meta and takes the credentials out of the URL:

import scrapy
from proxyproviders import ProxyFormat


class IpSpider(scrapy.Spider):
    name = "ip"

    def start_requests(self):
        yield scrapy.Request(
            "https://httpbin.org/ip",
            meta={"proxy": provider.get_proxy().format(ProxyFormat.SCRAPY)},
        )

Crawlee for Python Proxy

Crawlee rotates proxies and manages sessions itself, so hand it the whole list:

from crawlee.proxy_configuration import ProxyConfiguration

proxy_configuration = ProxyConfiguration(
    proxy_urls=[proxy.to_url() for proxy in provider.list_proxies()]
)

curl and Subprocesses

import os
import subprocess
from proxyproviders import ProxyFormat

proxy = provider.get_proxy()

# As curl arguments: ["-x", "http://user:pass@host:port"]
subprocess.run(["curl", *proxy.format(ProxyFormat.CURL), "https://httpbin.org/ip"])

# Or as HTTP_PROXY / HTTPS_PROXY for any program that reads them
subprocess.run(
    ["curl", "https://httpbin.org/ip"],
    env={**os.environ, **proxy.format(ProxyFormat.ENV)},
)

Use Your Own Proxy List

If your vendor has no provider here, or you simply have a list of proxies, ProxyList gives them the same interface as everything else:

from proxyproviders import ProxyList

provider = ProxyList(
    [
        "http://user:pass@203.0.113.7:8080",
        "203.0.113.8:8080:user:pass",
        "socks5://user:pass@203.0.113.9:1080",
    ]
)

# Or one proxy per line from a file (blank lines and # comments are skipped)
provider = ProxyList.from_file("proxies.txt")

A single proxy can be parsed with Proxy.from_string(...), which accepts a URL, user:pass@host:port, host:port or host:port:user:pass.

Automatic Rotation and Retries

RotatingSession is a requests.Session that picks a proxy for every request. If a request cannot get through its proxy, that proxy is set aside for a while and the request is sent again through the next one:

from proxyproviders import RotatingSession

session = RotatingSession(provider, max_retries=3)

response = session.get("https://httpbin.org/ip", timeout=10)
print(response.json(), session.last_proxy)

It is opt-in: nothing changes for code that calls get_proxy() itself. What it treats as a failed proxy, and when it sends the request again:

  • Could not connect to the proxy, or open a tunnel through it: retried for any HTTP method, because the request never reached the target.
  • Any other connection error or timeout, such as a proxy that drops the connection or stops responding: retried for GET, HEAD and OPTIONS only. A POST may already have been acted on, so the error is raised instead of sending it twice.
  • A 407 Proxy Authentication Required response, or any status you list in retry_statuses: retried for any method.

Ordinary error statuses such as 404 or 500 are returned as usual. File uploads and streamed bodies are sent once and never retried. Deciding that a response means your proxy is blocked is up to you, for example:

session = RotatingSession(provider, max_retries=3, retry_statuses={403, 429})

For httpx 0.26 or newer, sync or async, the same behaviour comes as a transport. Install it with pip install "proxyproviders[httpx]":

import httpx
from proxyproviders.httpx_transport import AsyncRotatingTransport, RotatingTransport

with httpx.Client(transport=RotatingTransport(provider, max_retries=3)) as client:
    response = client.get("https://httpbin.org/ip")

async with httpx.AsyncClient(transport=AsyncRotatingTransport(provider)) as client:
    response = await client.get("https://httpbin.org/ip")

Reporting Failed Proxies

When you make requests yourself, tell the provider about a proxy that failed and get_proxy() will avoid it for failure_cooldown seconds (60 by default):

import requests
from proxyproviders import ProxyFormat

proxy = provider.get_proxy()
try:
    response = requests.get(url, proxies=proxy.format(ProxyFormat.REQUESTS), timeout=10)
except requests.exceptions.ProxyError:
    provider.report_failure(proxy)

provider.report_success(proxy) puts it back early, and provider.available_proxies() lists the ones not currently set aside. If every proxy has failed, get_proxy() still returns one instead of raising. This works the same with every HTTP client, sync or async.

Choosing Which Proxy You Get

get_proxy() takes a selection algorithm. The default is round robin.

from proxyproviders.algorithms import First, Random, RoundRobin

proxy = provider.get_proxy()  # RoundRobin: next proxy in order
proxy = provider.get_proxy(Random())  # a random proxy
proxy = provider.get_proxy(First())  # always the first proxy

# A RoundRobin you create keeps its own position, separate from the provider's default
round_robin = RoundRobin()
proxy1 = provider.get_proxy(round_robin)
proxy2 = provider.get_proxy(round_robin)  # next in sequence

Write your own by subclassing Algorithm:

from typing import List
from proxyproviders import Proxy
from proxyproviders.algorithms import Algorithm


class USProxyAlgorithm(Algorithm):
    """Prefers US proxies, falls back to the first available."""

    def select(self, proxies: List[Proxy]) -> Proxy:
        for proxy in proxies:
            if proxy.country_code == "US":
                return proxy
        return proxies[0]


proxy = provider.get_proxy(USProxyAlgorithm())

Configuration

Options shared by every provider live in ProxyConfig (full reference):

Option Default Meaning
refresh_interval 600 Seconds before the proxy list is fetched again. 0 fetches once and never refreshes.
timeout 30 Seconds to wait on each call to the provider's API
refresh_retry_delay 30 If a refresh fails, the previous list keeps being served and the API is tried again after this many seconds
failure_cooldown 60 Seconds get_proxy() avoids a proxy after report_failure()
import os
import time
from proxyproviders import ProxyConfig, Webshare

config = ProxyConfig(refresh_interval=3)
ws = Webshare(api_key=os.environ["WEBSHARE_API_KEY"], config=config)

ws.list_proxies()  # calls the API
ws.list_proxies()  # cached

time.sleep(5)
ws.list_proxies()  # calls the API again, since more than 3s have passed

list_proxies() and get_proxy() are safe to call from several threads.

SOCKS Proxies

A proxy whose protocols only lists a SOCKS scheme, e.g. ["socks5"], is formatted with that scheme for every client:

proxy.format(ProxyFormat.REQUESTS)
# {'http': 'socks5://user:pass@host:1080', 'https': 'socks5://user:pass@host:1080'}

Your HTTP client needs SOCKS support for this to work. For requests, install it with:

python -m pip install "proxyproviders[socks]"

httpx needs httpx[socks]. aiohttp and Scrapy have no built-in SOCKS support.

Making Your Own Proxy Provider

Subclass ProxyProvider and return a list of Proxy objects from _fetch_proxies. Caching, refreshing, selection, formatting and failure reporting then work without any more code:

from typing import List, Optional
from proxyproviders import Proxy, ProxyConfig, ProxyProvider


class MyProxyProvider(ProxyProvider):
    def __init__(self, config: Optional[ProxyConfig] = None):
        super().__init__(config=config)

    def _fetch_proxies(self) -> List[Proxy]:
        # Call your vendor's API here and convert each entry
        return [
            Proxy(
                id="1",
                username="username",
                password="password",
                proxy_address="192.168.1.1",
                port=80,
            )
        ]


provider = MyProxyProvider()
proxy = provider.get_proxy()

If you write one for a vendor other people use, please contribute it back. CONTRIBUTING.md has the steps, and the repository includes an add-provider skill that walks a coding agent such as Claude Code through the whole process.

How to Support The Project

  • Star the repo 😎
  • Consider sponsoring me on GitHub
  • Send me an email or a LinkedIn message about what you're doing with it :D
  • Submit PRs for issues or any new providers/features :)

Note: If you want to learn how to web scrape websites check my free and open-source course for learning everything web scraping

Metadata

Release files for proxyproviders 0.3.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for proxyproviders 0.3.0
File Size Uploaded Provenance Actions
proxyproviders-0.3.0.tar.gz 25.6 kB With provenance Details for proxyproviders-0.3.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for proxyproviders 0.3.0
File Interpreter ABI Platform Actions
proxyproviders-0.3.0-py3-none-any.whl Python 3 none any Details for proxyproviders-0.3.0-py3-none-any.whl

Total release size: 58.4 kB

Release files / proxyproviders-0.3.0.tar.gz

Download URL proxyproviders-0.3.0.tar.gz
Size 25.6 kB
Tags Source
SHA-256 checksum
How to use checksums
d28374c391db5c3fee8ba833bf8907c25579f6a5b8b008233aa7c0f37d4cc692
BLAKE2b-256 checksum
How to use checksums
a5699cb6a62f236cbc7c028608b1850c18f114df9674810265b52561e01351f7
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 13, 2026.

Transparency log

Release files / proxyproviders-0.3.0-py3-none-any.whl

Download URL proxyproviders-0.3.0-py3-none-any.whl
Size 32.8 kB
Tags Python 3
SHA-256 checksum
How to use checksums
f0a721ae5729035550f2c1992b196639fa1b4cfef9a592716c6d6167e4e63147
BLAKE2b-256 checksum
How to use checksums
6cf9ba2c240efadc046d00f63cbaf1e75ef0a023d12e51efc8954da10fce614f
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 13, 2026.

Transparency log

Security for proxyproviders 0.3.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

Both files in this release have been uploaded with provenance metadata.

Select from 2 files

Attestation for proxyproviders-0.3.0.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 13, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page