Skip to main content

Run Scrapy Distributed

Project description

.. image:: docs/images/logo_readme.jpg
======================================

Scrapy Eagle is a tool that allow us to run any Scrapy based project in a distributed fashion and monitor how it is going on and how many resources it is consuming on each server.

**This project is Under Development, don't use it yet**

Installation
------------

It could be easily made by running the code bellow,

.. code-block:: console

$ pip install scrapy-eagle

You should create one ``configparser`` configuration file (e.g. in /etc/scrapy-eagle.ini) containing:

.. code-block:: console

[redis]
host = 10.10.10.10
port = 6379
db = 0

Then you will be able to execute the `eagle_server` command like,

.. code-block:: console

eagle_server --config=/etc/scrapy-eagle.ini

Note: Scrapy Eagle use parts of https://github.com/rolando/scrapy-redis

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Filename, size & hash SHA256 hash help File type Python version Upload date
scrapy-eagle-0.0.4.tar.gz (5.4 kB) Copy SHA256 hash SHA256 Source None

Supported by

Elastic Elastic Search Pingdom Pingdom Monitoring Google Google BigQuery Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN SignalFx SignalFx Supporter DigiCert DigiCert EV certificate StatusPage StatusPage Status page