Requests With High Availability Onion Router. For the filthiest web scrapers that have no time for rate-limits.
Project description
requests-whaor [ri-kwests hawr]
Requests + Docker + HAproxy + Tor
Requests With High Availability Onion Router. For the filthiest web scrapers that have no time for rate-limits.
Overview
requests-whaor proxies GET requests through a local Docker network of TOR circuits. It takes care of starting and stopping a pool of TOR proxies behind an HAproxy load balancer, which acts as a round robin reverse proxy network. This will give each request a new IP address. If you start having issues with the initial pool of IPs, requests-whaor can gather a new pool of IP addresses by restarting all TOR containers.
Install with pip
pip install requests-whaor
Install with Poetry
poetry add requests-whaor
>> Quickstart / Docs <<
Projects to highlight.
Useful Docker commands.
If things get out of hand you may need these commands for debugging or killing containers.
docker ps -q --filter ancestor=osminogin/tor-simple | xargs -L 1 docker logs --follow
docker ps -q --filter ancestor=osminogin/haproxy | xargs -L 1 docker logs --follow
docker stop $(docker ps -q --filter ancestor=osminogin/tor-simple)
docker stop $(docker ps -q --filter ancestor=haproxy)
docker network rm $(docker network ls -q -f name=whaornet)
TODO
- Testing.
- More request methods if requested.
- Options for using different Tor containers.
- Options for different load balancer containers.
Contact Information
Telegram = Twitter = Tradingview = Discord = @dgnsrekt
Email = dgnsrekt@pm.me
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for requests_whaor-0.2.1-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 2982153a6125a61d3824f7d72f4bbc080800a9d228e7a01befdf80f7e66c0085 |
|
MD5 | e3aea4a12e992e1c2f29e6340ee09da0 |
|
BLAKE2b-256 | a3bb266ae8273263d912bc132dea10ae3a01dcd2950b1ead05cd6134506f6715 |