No project description provided
Project description
Scrapy SQS Pipeline
Each time this pipeline receives an item, it sends it to SQS.
Requirements
- Python 3+
- Scrapy 1.1+
- boto3
Installation
pip3 install scrapy-sqspipeline
Configuration
-
Add the following lines to your Scrapy project settings.
ITEM_PIPELINES = { 'sqspipeline.SQSPipeline': 100, } # Either `SQSPIPELINE_QUEUE_URL` or `SQSPIPELINE_QUEUE_NAME` is required. SQSPIPELINE_QUEUE_URL='https://sqs.ap-northeast-1.amazonaws.com/xxxxxxxxxx/scrapy-sqspipeline' # SQSPIPELINE_QUEUE_NAME=''
-
Use AWS CLI's
aws configure
command to set up credentials. Alternatively, you can use Scrapy's settingsAWS_ACCESS_KEY_ID
andAWS_SECRET_ACCESS_KEY
.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Close
Hashes for scrapy_sqspipeline-1.0.0-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | ec8bfab822c39412b316be450d3773c1337574e2bc5a40a17198e36ce3969d8c |
|
MD5 | 565fcea211d8897a4e96f261f5642036 |
|
BLAKE2b-256 | 1c4ba7cba4700b90bb12fc0d8508b9f9958cc7704e01e0be285e023e17d5d7c7 |