title: Title

summary: "summary"

Proxy Manager - Python

The package is intended to manage a set of proxies, rotating through at random, to prevent blocking. Proxies are currently dropped from the rotation if they fail twice in a row.

VERY EARLY STAGE

Installation

pip install ProxyManager

Basic usage

PROXY_LIST = [
         '70.45.80.236:4602',
         '70.45.80.236:4603',
         '70.45.80.236:4604'
        ]

URL = 'http://www.myurl.com'

from ProxyManager.core import ProxyManager, Proxy

proxies = [i for i in PROXY_LIST] # OR [Proxy(i) for i in PROXY_LIST]
PM = ProxyManager(proxies)
result = PM.make_request(URL)

# Pass a fail function into make_request method, to check for domain/website specific indicators that the request has failed
# i.e. tell the ProxyManager, that this request has failed
def fail_function(response):
    if '<something_wrong_about>' in response:
        return True
    return False

result = PM.make_request(URL, fail_func=fail_function) # or PM.fail_func = fail_function

Free Proxies

If you don't bring your own proxies, which is HIGHLY recommended, I have written a function that scrapes https://free-proxy-list.net/ to obtain a set of proxies.

from ProxyManager.free_proxies import get_free
FREE_PROXIES = get_free(10)
proxies = [Proxy('{}:{}'.format(i['ip'], i['port'])) for i in FREE_PROXIES]