Skip to main content

Python Proxy Utils

Project description

Python Proxy Utils

A set of python 3 utils to help with web scraping. Currently only a free proxy utility is implemented. It can be used as a proxy rotator or however you like.

Install

pip install python-proxy-utils

ProxyHelper Example Usage

When the class is initialised it scrapes a list of free proxies from a number of free proxy list websites concurrently using aiohttp.

>>> from proxy_utils import ProxyHelper
>>> p = ProxyHelper()
>>> p.get_count()
13224
>>> p.get_random()
'http://103.9.188.143:56368'
>>>
>>> import requests
>>> random_proxy = p.get_random()
>>> proxies = {'http': random_proxy, 'https': random_proxy}
>>> r = requests.get('http://example.com', proxies=proxies)

Free Proxy Lists Used

This class currently uses the below free lists. At the time of writing this these lists usually produced around 13,000 proxies.

  • proxy-list.download
  • free-proxy-list.net

Todo

  • Testing
  • Add ProxyHelper method that returns the proxies in the format requests wants it in
  • Add more free proxy lists
  • Add a clean method to the proxy helper to remove bad proxies
  • Add a UserAgentHelper

Author

Jeremy Storer storerjeremy@gmail.com

Project details


Release history Release notifications

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for python-proxy-utils, version 0.1.0
Filename, size File type Python version Upload date Hashes
Filename, size python-proxy-utils-0.1.0.tar.gz (3.5 kB) File type Source Python version None Upload date Hashes View hashes

Supported by

Elastic Elastic Search Pingdom Pingdom Monitoring Google Google BigQuery Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN DigiCert DigiCert EV certificate StatusPage StatusPage Status page