grey_harvest 0.1.5

Key Features

• Quickly and easily generate a list of reliable http/https proxies

• Usable as a comamnd line utility or a python module

• Can filter for proxies that support SSL

• Can filter for proxies locationed within specific countries

• Can exclude proxies located within specific countries

Installation

First, install the following dependencies:

# On Centos/RHEL/Fedora:
sudo yum install python-devel libxlt-devel libxml-devel

# On Debian/Ubuntu:
sudo apt-get install python-dev libxml2-dev libxslt1-dev

Then install grey_harvest using pip as follows:

pip install grey_harvest

Usage

We can generate a list of 10 viable proxies with the following command:

# use the -n flag to specify number of proxies to generate
grey_harvest -n 10

To select only proxies with SSL enabled, we do this:

# use the -H flag to select only https proxies
grey_harvest -n 10 -H

We can use the -a flag to filter for proxies located within a list of specific countries. For example, to choose proxies located within Ukraine, Hong Kong, and the United States, we’d use this:

# use the -a flag to filter by country
grey_harvest -a "United States" "Hong Kong" Ukraine -n 10

We can use the -p flag to filter for ports running on specific ports:

# the -p flag to only use proxies that run on port 80
grey_harvest -p 80 -n 10

We can deny proxies located within specific countries by using the -d flag. Proxies located within China are blocked by default as they are often located behind the Great Firewall, and as such tend to be unreliable. This can be changed within grey_harvest.py’s internal configs.:

# use the -d flag to deny proxies located within France and
# Germany
grey_harvest -d France Germany -n 10 -H

grey_harvest library - basic example

Before diving into the documentation for the grey_harvest library, check out how easily we can generate a list of 20 proxies:

import gray_harvest

''' spawn a harvester '''
harvester = grey_harvest.GreyHarvester()

''' harvest some proxies from teh interwebz '''
count = 0
for proxy in harvester.run():
        print proxy
        count += 1
        if count >= 20:
                break

That’s it. We now have 20 http/https proxies ready to go.

0.1.5 (2016-04-20)

• Fixed connections errors that occur when specifying custom test domain

0.1.4 (2016-04-19)

• Users can now filter for proxies running on specific ports

0.1.3 (2015-05-26)

• Added documentation

0.1.2 (2015-05-26)

• Corrected some build issues

0.1.0 (2015-05-26)

• Initial release

Contributors

Please add yourself here alphabetically when you submit your first pull request.