Scrapy-Pipelines 0.2

Overview

These pipelines enable Scrapy to save items into various backends, including:

• MongoDB

And also these pipelines provide multiple ways to save or update the items, and return id created by backends

Requirements

• Python 3.6+

• Works on Linux, Windows, Mac OSX

Installation

The quick way:

pip install scrapy-pipelines

For more details see the installation section in the documentation: https://scrapy-pipelines.readthedocs.io/en/latest/intro/installation.html

Documentation

Documentation is available online at https://scrapy-pipelines.readthedocs.io/en/latest/ and in the docs directory.

Community (blog, twitter, mail list, IRC)

Keeping this section same as Scrapy is intending to benefit back to Scrapy.

See https://scrapy.org/community/

Contributing

Keeping this section same as Scrapy is intending to be easier when this repo merge back to Scrapy.

See https://doc.scrapy.org/en/master/contributing.html

Companies using Scrapy

Keeping this section same as Scrapy is intending to benefit back to Scrapy.

See https://scrapy.org/companies/

Commercial Support

Keeping this section same as Scrapy is intending to benefit back to Scrapy.

See https://scrapy.org/support/

TODO

• [X] Add indexes creation in open_spider()

• [X] Add item_completed method

• [X] Add signals for MongoDB document’s id return

• [ ] Add MongoDB document update

• [ ] Add Percona Server for MongoDB docker support

• [ ] Add Redis support

• [ ] Add InfluxDB support

• [ ] Add LevelDB support