Skip to main content

No project description provided

Project description

Scrapy SQS Pipeline

PyPI version

Each time this pipeline receives an item, it sends it to SQS.

Requirements

  • Python 3+
  • Scrapy 1.1+
  • boto3

Installation

pip3 install scrapy-sqspipeline

Configuration

  1. Add the following lines to your Scrapy project settings.

    ITEM_PIPELINES = {
       'sqspipeline.SQSPipeline': 100,
    }
    
    # Either `SQSPIPELINE_QUEUE_URL` or `SQSPIPELINE_QUEUE_NAME` is required.
    SQSPIPELINE_QUEUE_URL='https://sqs.ap-northeast-1.amazonaws.com/xxxxxxxxxx/scrapy-sqspipeline'
    # SQSPIPELINE_QUEUE_NAME=''
    
  2. Use AWS CLI's aws configure command to set up credentials. Alternatively, you can use Scrapy's settings AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

scrapy-sqspipeline-1.0.0.tar.gz (2.3 kB view hashes)

Uploaded Source

Built Distribution

scrapy_sqspipeline-1.0.0-py3-none-any.whl (3.6 kB view hashes)

Uploaded Python 3

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page