scrapy-mysql-pipeline 2017.10.7

Asynchronous mysql Scrapy item pipeline

Python 3.6

scrapy-mysql-pipeline

Asynchronous mysql Scrapy item pipeline

Installation

pip install scrapy-mysql-pipeline

Configuration

Add pipeline

ITEM_PIPELINES = {
    'scrapy_mysql_pipeline.MySQLPipeline': 300,
}

Default values:

MYSQL_HOST = 'localhost'
MYSQL_PORT = 3306
MYSQL_USER = None
MYSQL_PASSWORD = ''
MYSQL_DB = None
MYSQL_TABLE = None
MYSQL_UPSERT = False
MYSQL_RETRIES = 3
MYSQL_CLOSE_ON_ERROR = True

MYSQL_TABLE variable must be set in settings.py