proxyrotation 0.1.0

automatic free proxy rotation

proxy-rotation

automatic free proxy rotation for web scraping with caching and filtering.

The proxy rotator API includes convenience features such as:

• specifying various filtering options, such as anonymity level, security, and alpha-2 country code;
• downloading proxy addresses from free public sources;
• managing the state of positive and negative proxy addresses using caching;
• automatically rotating proxy addresses when stale or exhausted.

installation

To install proxyrotation in stable release, you should use pip:

pip install proxyrotation

usage

Here are some examples to get started with the proxy rotator API:

configuration

from proxyrotation.modelling import Anonymity
from proxyrotation.rotator import ProxyRotator

rotator = ProxyRotator(
    anonymity=Anonymity.high,  # desired anonymity level
    cachedir="/path/to/cachedir",  # path to cachedir
    countrycodeset={"US", "CN"},  # alpha-2 country codes of interest
    livecheck=True,  # whether to check if proxy addresses are working while fetching
    maxshape=100,  # max number of proxy addresses to keep
    repository="sequential",  # repository type for downloading
    schedule=3600.0,  # automatic refresh interval in secs
    secure=True,  # whether to enforce secure connections (HTTPS)
)

rotating proxy addresses

rotator.rotate()

proxy = rotator.selected

print(f"selected proxy: {proxy.peername}")

checking crawledset

print(f"free proxy addresses: {rotator.crawledset}")

rotator shape

print(f"rotator shape: {len(rotator)}")

advanced usage

TBC