Skip to main content

No project description provided

Project description

Scrapy SQS Pipeline

PyPI version

Each time this pipeline receives an item, it sends it to SQS.

Requirements

  • Python 3+
  • Scrapy 1.1+
  • boto3

Installation

pip3 install scrapy-sqspipeline

Configuration

  1. Add the following lines to your Scrapy project settings.

    ITEM_PIPELINES = {
       'sqspipeline.SQSPipeline': 100,
    }
    
    # Either `SQSPIPELINE_QUEUE_URL` or `SQSPIPELINE_QUEUE_NAME` is required.
    SQSPIPELINE_QUEUE_URL='https://sqs.ap-northeast-1.amazonaws.com/xxxxxxxxxx/scrapy-sqspipeline'
    # SQSPIPELINE_QUEUE_NAME=''
    
  2. Use AWS CLI's aws configure command to set up credentials. Alternatively, you can use Scrapy's settings AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

scrapy-sqspipeline-1.0.0.tar.gz (2.3 kB view details)

Uploaded Source

Built Distribution

scrapy_sqspipeline-1.0.0-py3-none-any.whl (3.6 kB view details)

Uploaded Python 3

File details

Details for the file scrapy-sqspipeline-1.0.0.tar.gz.

File metadata

  • Download URL: scrapy-sqspipeline-1.0.0.tar.gz
  • Upload date:
  • Size: 2.3 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.1.1 pkginfo/1.5.0.1 requests/2.23.0 setuptools/46.1.3 requests-toolbelt/0.9.1 tqdm/4.43.0 CPython/3.7.7

File hashes

Hashes for scrapy-sqspipeline-1.0.0.tar.gz
Algorithm Hash digest
SHA256 fba85a2fb6c83c5aab884a9d92291815bb51e719ab075c4f015fe465835c601e
MD5 5a2de53cdf3ce1cc9680ec62fa3c251c
BLAKE2b-256 34a090b2ed05b1bac0b9a9f3dae2bfbc2d54df9d26a5c4a38a8e8d016fc4d041

See more details on using hashes here.

File details

Details for the file scrapy_sqspipeline-1.0.0-py3-none-any.whl.

File metadata

  • Download URL: scrapy_sqspipeline-1.0.0-py3-none-any.whl
  • Upload date:
  • Size: 3.6 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.1.1 pkginfo/1.5.0.1 requests/2.23.0 setuptools/46.1.3 requests-toolbelt/0.9.1 tqdm/4.43.0 CPython/3.7.7

File hashes

Hashes for scrapy_sqspipeline-1.0.0-py3-none-any.whl
Algorithm Hash digest
SHA256 ec8bfab822c39412b316be450d3773c1337574e2bc5a40a17198e36ce3969d8c
MD5 565fcea211d8897a4e96f261f5642036
BLAKE2b-256 1c4ba7cba4700b90bb12fc0d8508b9f9958cc7704e01e0be285e023e17d5d7c7

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page