Distributed Python web crawling framework
Project description
dragline
=======
dragline is a distributed Python web crawling framework.
Features include:
* Distributed, scalable and persistent crawling
* Efficient lightweight parallel execution based on gevent
* Redis backend for distributed and uninterrupted crawling
Requirements
============
* Python 2.7
* Works on Linux
* Redis
Install
=======
The quick way::
pip install dragline
Download the latest release from `Python Package Index`_ or clone `the repository`_.
.. _Python Package Index: http://pypi.python.org/pypi/dragline
.. _the repository: https://github.com/inzyte/dragline
=======
dragline is a distributed Python web crawling framework.
Features include:
* Distributed, scalable and persistent crawling
* Efficient lightweight parallel execution based on gevent
* Redis backend for distributed and uninterrupted crawling
Requirements
============
* Python 2.7
* Works on Linux
* Redis
Install
=======
The quick way::
pip install dragline
Download the latest release from `Python Package Index`_ or clone `the repository`_.
.. _Python Package Index: http://pypi.python.org/pypi/dragline
.. _the repository: https://github.com/inzyte/dragline
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Dragline-0.9.1b2.tar.gz
(12.6 kB
view hashes)