Skip to main content

Proxy collector

Project description

# proxy_py

proxy_py is a program which collects proxies, saves them in a database
and makes periodically checks.
It has a server for getting proxies with nice API(see below).

## Where is the documentation?

It's [here](

## How to build?

1 Clone this repository

`git clone`

2 Install requirements

cd proxy_py
pip3 install -r requirements.txt

3 Create settings file

`cp config_examples/ proxy_py/`

4 Install postgresql and change database configuration in file

5 (Optional) Configure alembic

6 Run your application


7 Enjoy!

## I'm too lazy. Can I just use it?

`TODO: update, old version!`

Yes, you can download virtualbox image
After downloading check that port forwarding is still working,
you need forwarding of 55555 host port to 55555 guest.

## How to get proxies?

proxy_py has a server, based on aiohttp, which is listening
(you can change it in the settings file) and provides proxies.
To get proxies you should send the following json request
on address ``
(or other domain if behind reverse proxy):

"model": "proxy",
"method": "get",
"order_by": "response_time, uptime"

Note: order_by makes the result sorted
by one or more fields(separated by comma).
You can skip it. The required fields are `model` and `method`.

It's gonna return you the json response like this:

"count": 1,
"data": [{
"address": "",
"auth_data": "",
"bad_proxy": false,
"domain": "",
"last_check_time": 1509466165,
"number_of_bad_checks": 0,
"port": 8080,
"protocol": "http",
"response_time": 461691,
"uptime": 1509460949
"has_more": false,
"status": "ok",
"status_code": 200

Note: All fields except *protocol*, *domain*, *port*, *auth_data*,
*checking_period* and *address* CAN be null

Or error if something went wrong:

"error_message": "You should specify \"model\"",
"status": "error",
"status_code": 400

Note: status_code is also duplicated in HTTP status code

Example using curl:

`curl -X POST -H "Content-Type: application/json" --data '{"model": "proxy", "method": "get"}'`

Example using httpie:

`http POST model=proxy method=get`

Example using python's `requests` library:

import requests
import json

def get_proxies():
result = []
json_data = {
"model": "proxy",
"method": "get",

response ='', json=json_data)
if response.status_code == 200:
response = json.loads(response.text)
for proxy in response['data']:
# check error here

return result
Example using aiohttp library:

import aiohttp

async def get_proxies():
result = []
json_data = {
"model": "proxy",
"method": "get",

async with aiohttp.ClientSession() as session:
async with'', json=json_data) as response:
if response.status == 200:
response = json.loads(await response.text())
for proxy in response['data']:
# check error here

return result

## How to interact with API?

Read more about API [here](

## How to contribute?

`TODO: write guide about it`

## How to test it?

If you made the changes to code and want to check that you didn't break
anything, go [here](

## How to deploy on production using supervisor, nginx and postgresql in 8 steps?

1 Install supervisor, nginx and postgresql

`root@server:~$ apt install supervisor nginx postgresql`

2 Create virtual environment and install requirements on it

3 Copy example:

`proxy_py@server:~/proxy_py$ cp config_examples/ proxy_py/`

4 create unprivileged user in postgresql database
and change database authentication data in

proxy_py@server:~/proxy_py$ vim proxy_py/

# number of simultaneous connections
# 'max_connections': 20,

5 Copy supervisor config example and change it for your case

root@server:~$ cp /home/proxy_py/proxy_py/config_examples/proxy_py.supervisor.conf /etc/supervisor/conf.d/proxy_py.conf
root@server:~$ vim /etc/supervisor/conf.d/proxy_py.conf

6 Copy nginx config example, enable it and change if you need

root@server:~$ cp /home/proxy_py/proxy_py/config_examples/proxy_py.nginx.conf /etc/nginx/sites-available/proxy_py
root@server:~$ ln -s /etc/nginx/sites-available/proxy_py /etc/nginx/sites-enabled/
root@server:~$ vim /etc/nginx/sites-available/proxy_py

7 Restart supervisor and Nginx

root@server:~$ supervisorctl reread
root@server:~$ supervisorctl update
root@server:~$ /etc/init.d/nginx configtest
root@server:~$ /etc/init.d/nginx restart

8 Enjoy using it on your server!

## What is it depend on?

See `requirements.txt`

Project details

Release history Release notifications | RSS feed

This version


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

proxypy-2.0.tar.gz (19.0 kB view hashes)

Uploaded Source

Built Distribution

proxypy-2.0-py3-none-any.whl (3.7 kB view hashes)

Uploaded Python 3

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page