No project description provided
Project description
Scrapy SQS Pipeline
Each time this pipeline receives an item, it sends it to SQS.
Requirements
- Python 3+
- Scrapy 1.1+
- boto3
Installation
pip3 install scrapy-sqspipeline
Configuration
-
Add the following lines to your Scrapy project settings.
ITEM_PIPELINES = { 'sqspipeline.SQSPipeline': 100, } # Either `SQSPIPELINE_QUEUE_URL` or `SQSPIPELINE_QUEUE_NAME` is required. SQSPIPELINE_QUEUE_URL='https://sqs.ap-northeast-1.amazonaws.com/xxxxxxxxxx/scrapy-sqspipeline' # SQSPIPELINE_QUEUE_NAME=''
-
Use AWS CLI's
aws configure
command to set up credentials. Alternatively, you can use Scrapy's settingsAWS_ACCESS_KEY_ID
andAWS_SECRET_ACCESS_KEY
.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
File details
Details for the file scrapy-sqspipeline-1.0.0.tar.gz
.
File metadata
- Download URL: scrapy-sqspipeline-1.0.0.tar.gz
- Upload date:
- Size: 2.3 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/3.1.1 pkginfo/1.5.0.1 requests/2.23.0 setuptools/46.1.3 requests-toolbelt/0.9.1 tqdm/4.43.0 CPython/3.7.7
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | fba85a2fb6c83c5aab884a9d92291815bb51e719ab075c4f015fe465835c601e |
|
MD5 | 5a2de53cdf3ce1cc9680ec62fa3c251c |
|
BLAKE2b-256 | 34a090b2ed05b1bac0b9a9f3dae2bfbc2d54df9d26a5c4a38a8e8d016fc4d041 |
File details
Details for the file scrapy_sqspipeline-1.0.0-py3-none-any.whl
.
File metadata
- Download URL: scrapy_sqspipeline-1.0.0-py3-none-any.whl
- Upload date:
- Size: 3.6 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/3.1.1 pkginfo/1.5.0.1 requests/2.23.0 setuptools/46.1.3 requests-toolbelt/0.9.1 tqdm/4.43.0 CPython/3.7.7
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | ec8bfab822c39412b316be450d3773c1337574e2bc5a40a17198e36ce3969d8c |
|
MD5 | 565fcea211d8897a4e96f261f5642036 |
|
BLAKE2b-256 | 1c4ba7cba4700b90bb12fc0d8508b9f9958cc7704e01e0be285e023e17d5d7c7 |