scrapy-save-to-qiniu 0.2.0

Pipeline to save scrapy item field attachment to qiniu

Pipeline to save scrapy item field attachment to qiniu

Installation

Install scrapy-save-to-qiniu using pip::

$ pip install scrapy-save-to-qiniu

Configuration

1. Add the settings.py of your Scrapy project like this:

QINIU_AK = ''
QINIU_SK = ''
QINIU_BUCKET = ''
QINIU_DOMAIN = ''
QINIU_DEL_SRC = True
QINIU_FIELDS = [
    'pdf_url',
]

2. Enable the pipeline by adding it to ITEM_PIPELINES in your settings.py file and changing HttpCompressionMiddleware priority:

ITEM_PIPELINES = {
    'scrapy_save_to_qiniu.pipelines.SaveToQiniuPipeline': -1,
}

The order should before your persist pipeline such as save to database and after your preprocess pipeline.

Usage

no need to change your code

Getting help

Please use github issue

Contributing

PRs are always welcomed.

Changes

0.1.0 (2020-12-26)

Initial release