Skip to main content

Scrapy entrypoint for Scrapinghub job runner

Project description

Scrapy entrypoint for Scrapinghub job runner.

The package implements a base wrapper layer to extract job data from environment, parse/prepare it properly and execute job using Scrapy or custom executor.

Features

  • parsing job data from environment
  • processing job args and settings
  • running a job with Scrapy
  • collecting stats
  • advanced logging & error handling
  • full hubstorage support
  • custom scripts support

Requirements

  • Python 2/3
  • Scrapy, python-hubstorage (check requirements.txt for details)

Install

The quick way:

pip install scrapinghub-entrypoint-scrapy

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for scrapinghub-entrypoint-scrapy, version 0.6.1
Filename, size File type Python version Upload date Hashes
Filename, size scrapinghub_entrypoint_scrapy-0.6.1-py2-none-any.whl (12.8 kB) File type Wheel Python version py2 Upload date Hashes View hashes
Filename, size scrapinghub-entrypoint-scrapy-0.6.1.tar.gz (9.0 kB) File type Source Python version None Upload date Hashes View hashes

Supported by

Elastic Elastic Search Pingdom Pingdom Monitoring Google Google BigQuery Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN SignalFx SignalFx Supporter DigiCert DigiCert EV certificate StatusPage StatusPage Status page