Skip to main content
Python Software Foundation 20th Year Anniversary Fundraiser  Donate today!

Proxy-list management application for Django

Project description


.. image::
:alt: Build Status
.. image::
.. image::
:alt: Current version on PyPi
.. image::
:alt: Downloads from PyPi

This application is useful for keep an updated list of proxy servers, it
contains everything you need to make periodic checks to verify the properties
of the proxies. Also you can periodically collect the proxy server
from the Internet, remove broken and slow proxies.

Installing the package

`django-proxylist-for-grab` can be easily installed using pip:

.. code-block:: bash

$ pip install django-proxylist-for-grab


After that you need to include `django-proxylist-for-grab` into your
*INSTALLED_APPS* list of your django settings file.

.. code-block:: python


Add `django-proxylist-for-grab` into ````

.. code-block:: python

urlpatterns = patterns(
url(r'', include('proxylist.urls')),

`django-proxylist-for-grab` has a list of variables that you can configure
throught django's settings file. You can see the entire list at
Advanced Configuration.

Database creation

You have two choices here:

Using south

We ancourage recommend you using `south` for your database migrations. If you
already use it you can migrate `django-proxylist-for-grab`:

.. code-block:: bash

$ python migrate proxylist

Using syncdb

If you don't want to use `south` you can make a plain *syncdb*:

.. code-block:: bash

$ python syncdb

Basic setup

At first, add a mirror. For working mirror, you need to install app on
server with external ip. This is in order to be able to verify the correctness
of data through proxy server. After adding mirror, you can add and test
your proxies.

Asynchronously checking
`django-proxylist-for-grab` has configured by default to non-async check.
You can change this behavior. Insert into your django settings
``PROXY_LIST_USE_CALLERY`` and change it to True.

After you need to install and configure django-celery and rabbit-mq.

For example on OS X
**Packages installation**

.. code-block:: bash

$ sudo pip install django-celery
$ sudo port install rabbitmq-server

Add the 'djcelery' application to 'INSTALLED_APPS' in settings

.. code-block:: python


**Sync database**

.. code-block:: bash

$ ./ syncdb

**Run rabbitmq and celery**

.. code-block:: bash

$ sudo rabbitmq-server -detached
$ nohup python celery worker >& /dev/null &

Command line reference


Add new proxies from a file.

.. code-block:: bash

$ python update_proxies [file1] <file2> <...>


Check proxies availability and anonymity.

.. code-block:: bash

$ python check_proxies


Search proxy list on internet

.. code-block:: bash

$ python grab_proxies


Remove broken proxies

.. code-block:: bash

$ python clean_proxies

GrabLib usage example:

.. code-block:: python

from proxylist import grabber

grab = grabber.Grab()

# Get your ip (You can do this a few times to see how the proxy will be changed)
if grab.response.code == 200:
print grab.response.body.strip()

# Get count of div on google page
if grab.response.code == 200:

GrabLib Spider example:

.. code-block:: python

# filename: apps/app/management/commands/
# usage: python spider
from import BaseCommand
from grab.spider.base import Task
from proxylist.grabber import Spider

class SimpleSpider(Spider):
initial_urls = ['']

def task_initial(self, grab, task):
grab.set_input('Search', 'linux')
yield Task('search', grab=grab)

def task_search(self, grab, task):
for elem in'//b/a/font/b/text()'):
print elem.text()

class Command(BaseCommand):
help = 'Simple Spider'

def handle(self, *args, **options):
bot = SimpleSpider()
print bot.render_stats()

* GitHub:

.. image::
:alt: Bitdeli badge

Project details

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for django-proxylist-for-grab, version 0.5.1
Filename, size File type Python version Upload date Hashes
Filename, size django_proxylist_for_grab-0.5.1-py2-none-any.whl (113.8 kB) File type Wheel Python version 2.7 Upload date Hashes View
Filename, size django-proxylist-for-grab-0.5.1.tar.gz (94.9 kB) File type Source Python version None Upload date Hashes View

Supported by

AWS AWS Cloud computing Datadog Datadog Monitoring DigiCert DigiCert EV certificate Facebook / Instagram Facebook / Instagram PSF Sponsor Fastly Fastly CDN Google Google Object Storage and Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Salesforce Salesforce PSF Sponsor Sentry Sentry Error logging StatusPage StatusPage Status page