Greendeck Proxy Grabber Package
Project description
greendeck-proxygrabber
This package is developed by Greendeck
Install from pip
https://pypi.org/project/greendeck-proxygrabber/
pip install greendeck-proxygrabber
What is proxy service?
Proxy service is a service that keeps and updates a Mongo Database with latest up and running proxies.
How to use ?
import the service class
from greendeck_proxygrabber import ProxyService
service = ProxyService(MONGO_URI = 'mongodb://127.0.0.1:27017',
update_time = 300,
pool_limit = 1000,
update_count = 200,
database_name = 'proxy_pool',
collection_name_http = 'http',
collection_name_https = 'https'
)
This creates a service object.
Args
- update_time = Time after which proxies will be updated (in seconds)
- pool_limit = Limit after which insertion will change to updation
- update_count = Number of proxies to request grabber at a time
- database_name = Mongo Database name to store proxies in
- collection_name_http = Collection name to store http proxies in
- collection_name_https = Collection name to store https proxies in
Starting the service
service.start()
Starting service gives the following output:
MONGO_URI: mongodb://127.0.0.1:27017
Database: proxy_pool
Collection names: http, https
Press Ctrl+C once to stop...
Running Proxy Service.../
This will run forever and will push/update proxies in mongodb after every {update_time
} seconds.
How to use ?
import the library
import greendeck_proxygrabber
import ProxyGrabber
class
from greendeck_proxygrabber import ProxyGrabber
initialize ProxyGrabber
object
grabber = ProxyGrabber(len_proxy_list, country_code, timeout)
Here default values of some arguments are,
len_proxy_list = 10
country_code = 'ALL'
timeout = 2
Currently the program only supports proxies of combined regions
Getting checked, running proxies
The grab_proxy grab_proxy()
function helps to fetch the proxies.
grabber.grab_proxy()
This returns a dictionary of the following structure:
{
'https': [< list of https proxies >],
'http': [< list of http proxies >],
'region': 'ALL' # default for now
}
Getting an unchecked list of proxies
The grab_proxy proxy_scraper()
method of ScrapeProxy
helps to fetch the proxies.
This returns a list of 200 proxies of both type http and https.
from greendeck_proxygrabber import ScrapeProxy
proxies_http, proxies_https = ScrapeProxy.proxy_scraper()
This returns list of proxies of type http proxies followed by https proxies.
http_proxies = [< list of http proxies >]
https_proxies = [< list of https proxies >]
Filtering invalid proxies from a list of proxies
The proxy_checker_https
and proxy_checker_http
methods from ProxyChecker
class helps to validate the proxies.
Given a list of proxies, it checks each of them to be valid or not, and returns a list of valid proxies from the proxies feeded to it.
from greendeck_proxygrabber import ProxyChecker
valid_proxies_http = ProxyChecker.proxy_checker_http(proxy_list = proxy_list_http, timeout = 2)
valid_proxies_https = ProxyChecker.proxy_checker_https(proxy_list = proxy_list_https, timeout = 2)
How to build your own pip package
- open an account here https://pypi.org/
In the parent directory
python setup.py sdist bdist_wheel
twine upload dist/*
references
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Hashes for greendeck-proxygrabber-0.2.0.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | 79dcce610c3c333054c1ae467bba5ce782f84ce1c70a68f44f1e2718811e971f |
|
MD5 | fb2f8ce482880804842df387daacda4c |
|
BLAKE2b-256 | e4e50a70a65010959e1e2645c1eaec388d4a52c78aaaf44421e469969e87e478 |