Skip to main content

Make HTTP requests with scraped proxies

Project description

Python Proxy Requests | make an http GET/POST with a proxy scraped from https://www.sslproxies.org/



NOTE: This repository has been archived and is no longer maintained after urllib3==1.26.2

The ProxyRequests class first scrapes proxies from the web. Then it recursively attempts to make a request if the initial request with a proxy is unsuccessful.

Either copy the code and put where you want it, or download via pip:

pip install proxy-requests (or pip3)
from proxy_requests import ProxyRequests

or if you need the Basic Auth subclass as well:
from proxy_requests import ProxyRequests, ProxyRequestsBasicAuth

If the above import statement is used, method calls will be identical to the ones shown below. Pass a fully qualified URL when initializing an instance.

System Requirements: Python 3 and the requests module.

Runs on Linux and Windows (and Mac probably) - It may take a moment to run depending on the current proxy.
Each request with a proxy is set with an 3 second timeout in the event that the request takes too long (before trying the next proxy socket in the queue).

Proxies are randomly popped from the queue.

The ProxyRequestBasicAuth subclass has the methods get(), get_with_headers(), post(), post_with_headers(), post_file(), and post_file_with_headers() that will override the Parent methods.

GET:

    
r = ProxyRequests('https://api.ipify.org')
r.get()
    

GET with headers:

    
h = {'User-Agent': 'NCSA Mosaic/3.0 (Windows 95)'}
r = ProxyRequests('url here')
r.set_headers(h)
r.get_with_headers()
    

POST:

    
r = ProxyRequests('url here')
r.post({'key1': 'value1', 'key2': 'value2'})
    

POST with headers:

    
r = ProxyRequests('url here')
r.set_headers({'name': 'rootVIII', 'secret_message': '7Yufs9KIfj33d'})
r.post_with_headers({'key1': 'value1', 'key2': 'value2'})
    

POST FILE:

    
r = ProxyRequests('url here')
r.set_file('test.txt')
r.post_file()
    

POST FILE with headers:

    
h = {'User-Agent': 'NCSA Mosaic/3.0 (Windows 95)'}
r = ProxyRequests('url here')
r.set_headers(h)
r.set_file('test.txt')
r.post_file_with_headers()
    

GET with Basic Authentication:

    
r = ProxyRequestsBasicAuth('url here', 'username', 'password')
r.get()
    

GET with headers & Basic Authentication:

    
h = {'User-Agent': 'NCSA Mosaic/3.0 (Windows 95)'}
r = ProxyRequestsBasicAuth('url here', 'username', 'password')
r.set_headers(h)
r.get_with_headers()
    

POST with Basic Authentication:

    
r = ProxyRequestsBasicAuth('url here', 'username', 'password')
r.post({'key1': 'value1', 'key2': 'value2'})
    

POST with headers & Basic Authentication:

    
r = ProxyRequestsBasicAuth('url here', 'username', 'password')
r.set_headers({'header_key': 'header_value'})
r.post_with_headers({'key1': 'value1', 'key2': 'value2'})
    

POST FILE with Basic Authentication:

    
r = ProxyRequestsBasicAuth('url here', 'username', 'password')
r.set_file('test.txt')
r.post_file()
    

POST FILE with headers & Basic Authentication:

    
h = {'User-Agent': 'NCSA Mosaic/3.0 (Windows 95)'}
r = ProxyRequestsBasicAuth('url here', 'username', 'password')
r.set_headers(h)
r.set_file('test.txt')
r.post_file_with_headers()
    



Response Methods

Returns a string:
print(r)
Or if you want the raw content as bytes:
r.get_raw()
Get the response as JSON (if valid JSON):
r.get_json()
Get the response headers:
print(r.get_headers())
Get the status code:
print(r.get_status_code())
Get the URL that was requested:
print(r.get_url())
Get the proxy that was used to make the request:
print(r.get_proxy_used())

To write raw data to a file (including an image):

    

url = 'https://www.restwords.com/static/ICON.png'
r = ProxyRequests(url)
r.get()
with open('out.png', 'wb') as f:
    f.write(r.get_raw())

    

Dump the response to a file as JSON:
    
import json
with open('test.txt', 'w') as f:
    json.dump(r.get_json(), f)
    


This was developed on Ubuntu 16.04.4/18.04 LTS.
Author: James Loye Colley 2018-2020

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

proxy_requests-0.5.2.tar.gz (4.8 kB view details)

Uploaded Source

File details

Details for the file proxy_requests-0.5.2.tar.gz.

File metadata

  • Download URL: proxy_requests-0.5.2.tar.gz
  • Upload date:
  • Size: 4.8 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.7.0 requests/2.25.1 setuptools/51.1.1 requests-toolbelt/0.9.1 tqdm/4.58.0 CPython/3.9.1

File hashes

Hashes for proxy_requests-0.5.2.tar.gz
Algorithm Hash digest
SHA256 2ee375e559eeb5fc698cd316074d4d10a9c77bb3e4902f010e2512ad032a3d67
MD5 eb467b8ede2f625b87f64119e2859562
BLAKE2b-256 bde9aa76c6139a8f1c90382a3667b427991846279b95bf71804e8cfdbf99ed6c

See more details on using hashes here.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Pingdom Monitoring Sentry Error logging StatusPage Status page