ProxyProviders
One interface for proxy vendors, many Python clients.
Get proxies from Webshare, Bright Data or your own list, and use them with requests, httpx, aiohttp, Playwright, Scrapy, Crawlee or curl without writing vendor-specific code. Swap providers by changing one line.
import requests
from proxyproviders import Webshare, ProxyFormat
provider = Webshare(api_key="your-api-key") # or BrightData(...), or ProxyList([...])
proxy = provider.get_proxy()
response = requests.get(
"https://httpbin.org/ip", proxies=proxy.format(ProxyFormat.REQUESTS)
)
Table of Contents
- Installation
- What It Does
- Works With
- Supported Proxy Providers
- Quick Start Guide
- Using Proxies With Your HTTP Client
- Use Your Own Proxy List
- Automatic Rotation and Retries
- Reporting Failed Proxies
- Choosing Which Proxy You Get
- Configuration
- SOCKS Proxies
- Making Your Own Proxy Provider
- How to Support The Project
Installation
python -m pip install proxyproviders
Python 3.10 or newer. The full documentation is here, and there are runnable examples in the repository.
Using an AI coding assistant? Point it at llms.txt, a short summary of the API written for assistants.
📺 Video Tutorial: Watch the quick start guide on YouTube for a walkthrough of getting started with ProxyProviders.
What It Does
Every proxy vendor has its own API, and every HTTP client wants its proxy in a different shape. ProxyProviders sits between the two:
Webshare | Bright Data | your own list
↓
ProxyProvider → Proxy
↓
requests | httpx | aiohttp | Playwright | Scrapy | Crawlee | curl
- Provider API handled for you. Proxies are fetched from the vendor, cached, and refreshed on an interval.
- One
Proxymodel. Host, port, credentials, country and protocols look the same whichever vendor they came from. - One line to hand it to your client.
proxy.format(...)produces what each client expects, with credentials percent-encoded correctly. - Vendor-neutral code. Anything written against
ProxyProviderworks with every provider, so switching vendor does not touch the rest of your application.
Rotation and retries are available if you want them (RotatingSession), and stay out of the way if your framework already does that.
Works With
| Client | How to pass the proxy | Format |
|---|---|---|
| requests | requests.get(url, proxies=...) |
ProxyFormat.REQUESTS |
| httpx | httpx.Client(proxy=...) |
ProxyFormat.HTTPX |
| aiohttp | session.get(url, proxy=...) |
ProxyFormat.AIOHTTP |
| Playwright | browser.new_context(proxy=...) |
ProxyFormat.PLAYWRIGHT |
| Scrapy | Request(meta={"proxy": ...}) |
ProxyFormat.SCRAPY |
| Crawlee | ProxyConfiguration(proxy_urls=[...]) |
proxy.to_url() |
| urllib | urllib.request.ProxyHandler(...) |
ProxyFormat.URLLIB |
| curl | curl -x ... |
ProxyFormat.CURL |
| Any subprocess | HTTP_PROXY / HTTPS_PROXY variables |
ProxyFormat.ENV |
| Other tools | host:port:username:password |
ProxyFormat.HOST_PORT_USER_PASS |
| Anything else | scheme://username:password@host:port |
ProxyFormat.URL |
requests, httpx, aiohttp, urllib and curl are tested in CI by sending a real request through a local proxy with each client's current release.
Supported Proxy Providers
| Provider | Class | Notes |
|---|---|---|
| Webshare | Webshare(api_key=...) |
10 free datacenter proxies to try it out |
| Bright Data | BrightData(api_key=..., zone=...) |
Super proxy or one proxy per IP in the zone |
| Any other vendor | ProxyList([...]) |
Your own list of proxy URLs or host:port:user:pass lines |
Getting credentials:
- Webshare: create an account here (affiliate link), then generate a token in the API tab.
- Bright Data: create an account here (affiliate link), then create an API token in the account settings.
Keep API tokens out of your code and repository; the examples below read them from environment variables. Each provider's options are in the provider documentation. Don't see your vendor? Add a provider or open an issue.
Quick Start Guide
List the proxies from a provider:
import os
from proxyproviders import Webshare
provider = Webshare(api_key=os.environ["WEBSHARE_API_KEY"])
proxies = provider.list_proxies()
print(proxies) # passwords are left out of the printed output
Or take one at a time. get_proxy() cycles through the list in order by default:
proxy = provider.get_proxy()
print(proxy.proxy_address, proxy.port, proxy.country_code)
Providers take their own options. Webshare passes search_params to the Webshare API, so this returns only proxies based in the US:
provider = Webshare(
api_key=os.environ["WEBSHARE_API_KEY"], search_params={"country_code_in": "US"}
)
Because every provider has the same interface, code written against ProxyProvider does not care which vendor is behind it:
import os
import requests
from proxyproviders import BrightData, ProxyFormat, ProxyProvider, Webshare
def fetch_ip(provider: ProxyProvider) -> str:
proxy = provider.get_proxy()
response = requests.get(
"https://httpbin.org/ip", proxies=proxy.format(ProxyFormat.REQUESTS)
)
return response.json()["origin"]
fetch_ip(Webshare(api_key=os.environ["WEBSHARE_API_KEY"]))
fetch_ip(BrightData(api_key=os.environ["BRIGHTDATA_API_KEY"], zone="my_zone"))
That is how other libraries, such as my TikTokAPI package, accept any proxy vendor without knowing about it.
Using Proxies With Your HTTP Client
Every example below starts with a provider, which can be any provider from the table above.
Python requests With Rotating Proxies
import requests
from proxyproviders import ProxyFormat
proxy = provider.get_proxy()
response = requests.get(
"https://httpbin.org/ip", proxies=proxy.format(ProxyFormat.REQUESTS)
)
Each get_proxy() call returns the next proxy, so calling it per request rotates through your proxies. To have rotation and retries done for you, use RotatingSession.
httpx Proxy
httpx (0.26 or newer) takes one proxy URL in its proxy= argument, for both the sync and async clients:
import httpx
from proxyproviders import ProxyFormat
with httpx.Client(proxy=provider.get_proxy().format(ProxyFormat.HTTPX)) as client:
response = client.get("https://httpbin.org/ip")
async with httpx.AsyncClient(
proxy=provider.get_proxy().format(ProxyFormat.HTTPX)
) as client:
response = await client.get("https://httpbin.org/ip")
aiohttp Proxy
import aiohttp
from proxyproviders import ProxyFormat
async with aiohttp.ClientSession() as session:
async with session.get(
"https://httpbin.org/ip", proxy=provider.get_proxy().format(ProxyFormat.AIOHTTP)
) as response:
print(await response.json())
Playwright Proxy in Python
from playwright.sync_api import sync_playwright
from proxyproviders import ProxyFormat
with sync_playwright() as p:
browser = p.chromium.launch()
context = browser.new_context(
proxy=provider.get_proxy().format(ProxyFormat.PLAYWRIGHT)
)
page = context.new_page()
page.goto("https://httpbin.org/ip")
Give each browser context its own get_proxy() call to spread contexts across proxies. The same dictionary works for p.chromium.launch(proxy=...).
Scrapy Proxy
Scrapy's built-in HttpProxyMiddleware reads the proxy from request.meta and takes the credentials out of the URL:
import scrapy
from proxyproviders import ProxyFormat
class IpSpider(scrapy.Spider):
name = "ip"
def start_requests(self):
yield scrapy.Request(
"https://httpbin.org/ip",
meta={"proxy": provider.get_proxy().format(ProxyFormat.SCRAPY)},
)
Crawlee for Python Proxy
Crawlee rotates proxies and manages sessions itself, so hand it the whole list:
from crawlee.proxy_configuration import ProxyConfiguration
proxy_configuration = ProxyConfiguration(
proxy_urls=[proxy.to_url() for proxy in provider.list_proxies()]
)
curl and Subprocesses
import os
import subprocess
from proxyproviders import ProxyFormat
proxy = provider.get_proxy()
# As curl arguments: ["-x", "http://user:pass@host:port"]
subprocess.run(["curl", *proxy.format(ProxyFormat.CURL), "https://httpbin.org/ip"])
# Or as HTTP_PROXY / HTTPS_PROXY for any program that reads them
subprocess.run(
["curl", "https://httpbin.org/ip"],
env={**os.environ, **proxy.format(ProxyFormat.ENV)},
)
Use Your Own Proxy List
If your vendor has no provider here, or you simply have a list of proxies, ProxyList gives them the same interface as everything else:
from proxyproviders import ProxyList
provider = ProxyList(
[
"http://user:pass@203.0.113.7:8080",
"203.0.113.8:8080:user:pass",
"socks5://user:pass@203.0.113.9:1080",
]
)
# Or one proxy per line from a file (blank lines and # comments are skipped)
provider = ProxyList.from_file("proxies.txt")
A single proxy can be parsed with Proxy.from_string(...), which accepts a URL, user:pass@host:port, host:port or host:port:user:pass.
Automatic Rotation and Retries
RotatingSession is a requests.Session that picks a proxy for every request. If a request cannot get through its proxy, that proxy is set aside for a while and the request is sent again through the next one:
from proxyproviders import RotatingSession
session = RotatingSession(provider, max_retries=3)
response = session.get("https://httpbin.org/ip", timeout=10)
print(response.json(), session.last_proxy)
It is opt-in: nothing changes for code that calls get_proxy() itself. What it treats as a failed proxy, and when it sends the request again:
- Could not connect to the proxy, or open a tunnel through it: retried for any HTTP method, because the request never reached the target.
- Any other connection error or timeout, such as a proxy that drops the connection or stops responding: retried for
GET,HEADandOPTIONSonly. APOSTmay already have been acted on, so the error is raised instead of sending it twice. - A
407 Proxy Authentication Requiredresponse, or any status you list inretry_statuses: retried for any method.
Ordinary error statuses such as 404 or 500 are returned as usual. File uploads and streamed bodies are sent once and never retried. Deciding that a response means your proxy is blocked is up to you, for example:
session = RotatingSession(provider, max_retries=3, retry_statuses={403, 429})
For httpx 0.26 or newer, sync or async, the same behaviour comes as a transport. Install it with pip install "proxyproviders[httpx]":
import httpx
from proxyproviders.httpx_transport import AsyncRotatingTransport, RotatingTransport
with httpx.Client(transport=RotatingTransport(provider, max_retries=3)) as client:
response = client.get("https://httpbin.org/ip")
async with httpx.AsyncClient(transport=AsyncRotatingTransport(provider)) as client:
response = await client.get("https://httpbin.org/ip")
Reporting Failed Proxies
When you make requests yourself, tell the provider about a proxy that failed and get_proxy() will avoid it for failure_cooldown seconds (60 by default):
import requests
from proxyproviders import ProxyFormat
proxy = provider.get_proxy()
try:
response = requests.get(url, proxies=proxy.format(ProxyFormat.REQUESTS), timeout=10)
except requests.exceptions.ProxyError:
provider.report_failure(proxy)
provider.report_success(proxy) puts it back early, and provider.available_proxies() lists the ones not currently set aside. If every proxy has failed, get_proxy() still returns one instead of raising. This works the same with every HTTP client, sync or async.
Choosing Which Proxy You Get
get_proxy() takes a selection algorithm. The default is round robin.
from proxyproviders.algorithms import First, Random, RoundRobin
proxy = provider.get_proxy() # RoundRobin: next proxy in order
proxy = provider.get_proxy(Random()) # a random proxy
proxy = provider.get_proxy(First()) # always the first proxy
# A RoundRobin you create keeps its own position, separate from the provider's default
round_robin = RoundRobin()
proxy1 = provider.get_proxy(round_robin)
proxy2 = provider.get_proxy(round_robin) # next in sequence
Write your own by subclassing Algorithm:
from typing import List
from proxyproviders import Proxy
from proxyproviders.algorithms import Algorithm
class USProxyAlgorithm(Algorithm):
"""Prefers US proxies, falls back to the first available."""
def select(self, proxies: List[Proxy]) -> Proxy:
for proxy in proxies:
if proxy.country_code == "US":
return proxy
return proxies[0]
proxy = provider.get_proxy(USProxyAlgorithm())
Configuration
Options shared by every provider live in ProxyConfig (full reference):
| Option | Default | Meaning |
|---|---|---|
refresh_interval |
600 | Seconds before the proxy list is fetched again. 0 fetches once and never refreshes. |
timeout |
30 | Seconds to wait on each call to the provider's API |
refresh_retry_delay |
30 | If a refresh fails, the previous list keeps being served and the API is tried again after this many seconds |
failure_cooldown |
60 | Seconds get_proxy() avoids a proxy after report_failure() |
import os
import time
from proxyproviders import ProxyConfig, Webshare
config = ProxyConfig(refresh_interval=3)
ws = Webshare(api_key=os.environ["WEBSHARE_API_KEY"], config=config)
ws.list_proxies() # calls the API
ws.list_proxies() # cached
time.sleep(5)
ws.list_proxies() # calls the API again, since more than 3s have passed
list_proxies() and get_proxy() are safe to call from several threads.
SOCKS Proxies
A proxy whose protocols only lists a SOCKS scheme, e.g. ["socks5"], is formatted with that scheme for every client:
proxy.format(ProxyFormat.REQUESTS)
# {'http': 'socks5://user:pass@host:1080', 'https': 'socks5://user:pass@host:1080'}
Your HTTP client needs SOCKS support for this to work. For requests, install it with:
python -m pip install "proxyproviders[socks]"
httpx needs httpx[socks]. aiohttp and Scrapy have no built-in SOCKS support.
Making Your Own Proxy Provider
Subclass ProxyProvider and return a list of Proxy objects from _fetch_proxies. Caching, refreshing, selection, formatting and failure reporting then work without any more code:
from typing import List, Optional
from proxyproviders import Proxy, ProxyConfig, ProxyProvider
class MyProxyProvider(ProxyProvider):
def __init__(self, config: Optional[ProxyConfig] = None):
super().__init__(config=config)
def _fetch_proxies(self) -> List[Proxy]:
# Call your vendor's API here and convert each entry
return [
Proxy(
id="1",
username="username",
password="password",
proxy_address="192.168.1.1",
port=80,
)
]
provider = MyProxyProvider()
proxy = provider.get_proxy()
If you write one for a vendor other people use, please contribute it back. CONTRIBUTING.md has the steps, and the repository includes an add-provider skill that walks a coding agent such as Claude Code through the whole process.
How to Support The Project
- Star the repo 😎
- Consider sponsoring me on GitHub
- Send me an email or a LinkedIn message about what you're doing with it :D
- Submit PRs for issues or any new providers/features :)
Note: If you want to learn how to web scrape websites check my free and open-source course for learning everything web scraping
Metadata
Release files for proxyproviders 0.3.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| proxyproviders-0.3.0.tar.gz | 25.6 kB | With provenance | Details for proxyproviders-0.3.0.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| proxyproviders-0.3.0-py3-none-any.whl | Python 3 | none | any | Details for proxyproviders-0.3.0-py3-none-any.whl |
Total release size: 58.4 kB
Release files / proxyproviders-0.3.0.tar.gz
| Download URL | proxyproviders-0.3.0.tar.gz |
|---|---|
| Size | 25.6 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
d28374c391db5c3fee8ba833bf8907c25579f6a5b8b008233aa7c0f37d4cc692
|
|
BLAKE2b-256 checksum How to use checksums |
a5699cb6a62f236cbc7c028608b1850c18f114df9674810265b52561e01351f7
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 13, 2026.
Transparency logRelease files / proxyproviders-0.3.0-py3-none-any.whl
| Download URL | proxyproviders-0.3.0-py3-none-any.whl |
|---|---|
| Size | 32.8 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
f0a721ae5729035550f2c1992b196639fa1b4cfef9a592716c6d6167e4e63147
|
|
BLAKE2b-256 checksum How to use checksums |
6cf9ba2c240efadc046d00f63cbaf1e75ef0a023d12e51efc8954da10fce614f
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 13, 2026.
Transparency logSecurity for proxyproviders 0.3.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
Full provenance
Both files in this release have been uploaded with provenance metadata.
Attestation for proxyproviders-0.3.0.tar.gz
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 13, 2026.
Transparency logA note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.