Skip to main content

Proxy collector

Project description

# proxy_py

proxy_py is a program which collects proxies, saves them in a database
and makes periodically checks.
It has a server for getting proxies with nice API(see below).

## Where is the documentation?

It's [here](http://proxy-py.readthedocs.io)

## How to build?

1 Clone this repository

`git clone https://github.com/DevAlone/proxy_py.git`

2 Install requirements

```bash
cd proxy_py
pip3 install -r requirements.txt
```

3 Create settings file

`cp config_examples/settings.py proxy_py/settings.py`

4 Install postgresql and change database configuration in settings.py file

5 (Optional) Configure alembic

6 Run your application

`python3 main.py`

7 Enjoy!

## I'm too lazy. Can I just use it?

`TODO: update, old version!`

Yes, you can download virtualbox image
[here](https://drive.google.com/file/d/1oPf6xwOADRH95oZW0vkPr1Uu_iLDe9jc/view?usp=sharing).
After downloading check that port forwarding is still working,
you need forwarding of 55555 host port to 55555 guest.

## How to get proxies?

proxy_py has a server, based on aiohttp, which is listening 127.0.0.1:55555
(you can change it in the settings file) and provides proxies.
To get proxies you should send the following json request
on address `http://127.0.0.1:55555/api/v1/`
(or other domain if behind reverse proxy):

```json
{
"model": "proxy",
"method": "get",
"order_by": "response_time, uptime"
}
```

Note: order_by makes the result sorted
by one or more fields(separated by comma).
You can skip it. The required fields are `model` and `method`.

It's gonna return you the json response like this:

```json
{
"count": 1,
"data": [{
"address": "http://127.0.0.1:8080",
"auth_data": "",
"bad_proxy": false,
"domain": "127.0.0.1",
"last_check_time": 1509466165,
"number_of_bad_checks": 0,
"port": 8080,
"protocol": "http",
"response_time": 461691,
"uptime": 1509460949
}
],
"has_more": false,
"status": "ok",
"status_code": 200
}
```

Note: All fields except *protocol*, *domain*, *port*, *auth_data*,
*checking_period* and *address* CAN be null

Or error if something went wrong:

```json
{
"error_message": "You should specify \"model\"",
"status": "error",
"status_code": 400
}
```

Note: status_code is also duplicated in HTTP status code

Example using curl:

`curl -X POST http://127.0.0.1:55555/api/v1/ -H "Content-Type: application/json" --data '{"model": "proxy", "method": "get"}'`

Example using httpie:

`http POST http://127.0.0.1:55555/api/v1/ model=proxy method=get`

Example using python's `requests` library:

```python
import requests
import json


def get_proxies():
result = []
json_data = {
"model": "proxy",
"method": "get",
}

response = requests.post('http://127.0.0.1:55555/api/v1/', json=json_data)
if response.status_code == 200:
response = json.loads(response.text)
for proxy in response['data']:
result.append(proxy['address'])
else:
# check error here
pass

return result
```
Example using aiohttp library:

```python
import aiohttp


async def get_proxies():
result = []
json_data = {
"model": "proxy",
"method": "get",
}

async with aiohttp.ClientSession() as session:
async with session.post('http://127.0.0.1:55555/api/v1/', json=json_data) as response:
if response.status == 200:
response = json.loads(await response.text())
for proxy in response['data']:
result.append(proxy['address'])
else:
# check error here
pass

return result
```

## How to interact with API?

Read more about API [here](https://github.com/DevAlone/proxy_py/tree/master/docs/API.md)

## How to contribute?

`TODO: write guide about it`

## How to test it?

If you made the changes to code and want to check that you didn't break
anything, go [here](https://github.com/DevAlone/proxy_py/tree/master/docs/tests.md)

## How to deploy on production using supervisor, nginx and postgresql in 8 steps?

1 Install supervisor, nginx and postgresql

`root@server:~$ apt install supervisor nginx postgresql`

2 Create virtual environment and install requirements on it

3 Copy settings.py example:

`proxy_py@server:~/proxy_py$ cp config_examples/settings.py proxy_py/`

4 create unprivileged user in postgresql database
and change database authentication data in settings.py

```bash
proxy_py@server:~/proxy_py$ vim proxy_py/settings.py
```

```bash
DATABASE_CONNECTION_KWARGS = {
'database': 'YOUR_POSTGRES_DATABASE',
'user': 'YOUR_POSTGRES_USER',
'password': 'YOUR_POSTGRES_PASSWORD',
# number of simultaneous connections
# 'max_connections': 20,
}
```

5 Copy supervisor config example and change it for your case

```bash
root@server:~$ cp /home/proxy_py/proxy_py/config_examples/proxy_py.supervisor.conf /etc/supervisor/conf.d/proxy_py.conf
root@server:~$ vim /etc/supervisor/conf.d/proxy_py.conf
```

6 Copy nginx config example, enable it and change if you need

```bash
root@server:~$ cp /home/proxy_py/proxy_py/config_examples/proxy_py.nginx.conf /etc/nginx/sites-available/proxy_py
root@server:~$ ln -s /etc/nginx/sites-available/proxy_py /etc/nginx/sites-enabled/
root@server:~$ vim /etc/nginx/sites-available/proxy_py
```

7 Restart supervisor and Nginx

```bash
root@server:~$ supervisorctl reread
root@server:~$ supervisorctl update
root@server:~$ /etc/init.d/nginx configtest
root@server:~$ /etc/init.d/nginx restart
```

8 Enjoy using it on your server!

## What is it depend on?

See `requirements.txt`


Project details


Release history Release notifications | RSS feed

This version

2.0

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

proxypy-2.0.tar.gz (19.0 kB view details)

Uploaded Source

Built Distribution

proxypy-2.0-py3-none-any.whl (3.7 kB view details)

Uploaded Python 3

File details

Details for the file proxypy-2.0.tar.gz.

File metadata

  • Download URL: proxypy-2.0.tar.gz
  • Upload date:
  • Size: 19.0 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No

File hashes

Hashes for proxypy-2.0.tar.gz
Algorithm Hash digest
SHA256 fcc12ca7d8c94d42ad0c8802f7882ccfe73af38aa384eb54555df969d1ee1fe3
MD5 cbf68ef812a97b293a171018d78729ae
BLAKE2b-256 980cc3753d0fd11f9d675f3b6f5db4ed6c510d1facecffeeed2341e6f4ba9a3f

See more details on using hashes here.

File details

Details for the file proxypy-2.0-py3-none-any.whl.

File metadata

File hashes

Hashes for proxypy-2.0-py3-none-any.whl
Algorithm Hash digest
SHA256 0be482da4ab82f260a2857810ca3e39aeab1b9fad018dc3a3cad443255781157
MD5 ebb3e2c7567308795b1f451582a8b6ad
BLAKE2b-256 be9616b907b1b492374d6cdac88c2a4039f2799dbb8d1a7977149b23da7b1a09

See more details on using hashes here.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Pingdom Monitoring Sentry Error logging StatusPage Status page