Scrapy SQS Pipeline
Each time this pipeline receives an item, it sends it to SQS.
Requirements
- Python 3+
- Scrapy 1.1+
- boto3
Installation
pip3 install scrapy-sqspipeline
Configuration
-
Add the following lines to your Scrapy project settings.
ITEM_PIPELINES = { 'sqspipeline.SQSPipeline': 100, } # Either `SQSPIPELINE_QUEUE_URL` or `SQSPIPELINE_QUEUE_NAME` is required. SQSPIPELINE_QUEUE_URL='https://sqs.ap-northeast-1.amazonaws.com/xxxxxxxxxx/scrapy-sqspipeline' # SQSPIPELINE_QUEUE_NAME=''
-
Use AWS CLI's
aws configurecommand to set up credentials. Alternatively, you can use Scrapy's settingsAWS_ACCESS_KEY_IDandAWS_SECRET_ACCESS_KEY.
1 maintainer
Release files for scrapy-sqspipeline 1.0.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| scrapy-sqspipeline-1.0.0.tar.gz | 2.3 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| scrapy_sqspipeline-1.0.0-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 5.9 kB
Release files / scrapy-sqspipeline-1.0.0.tar.gz
| Download URL | scrapy-sqspipeline-1.0.0.tar.gz |
|---|---|
| Size | 2.3 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
fba85a2fb6c83c5aab884a9d92291815bb51e719ab075c4f015fe465835c601e
|
|
BLAKE2b-256 checksum How to use checksums |
34a090b2ed05b1bac0b9a9f3dae2bfbc2d54df9d26a5c4a38a8e8d016fc4d041
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/3.1.1 pkginfo/1.5.0.1 requests/2.23.0 setuptools/46.1.3 requests-toolbelt/0.9.1 tqdm/4.43.0 CPython/3.7.7
|
Release files / scrapy_sqspipeline-1.0.0-py3-none-any.whl
| Download URL | scrapy_sqspipeline-1.0.0-py3-none-any.whl |
|---|---|
| Size | 3.6 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
ec8bfab822c39412b316be450d3773c1337574e2bc5a40a17198e36ce3969d8c
|
|
BLAKE2b-256 checksum How to use checksums |
1c4ba7cba4700b90bb12fc0d8508b9f9958cc7704e01e0be285e023e17d5d7c7
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/3.1.1 pkginfo/1.5.0.1 requests/2.23.0 setuptools/46.1.3 requests-toolbelt/0.9.1 tqdm/4.43.0 CPython/3.7.7
|
