Skip to main content

No project description provided

Project description

PyPI Version Build Status Wheel Status Coverage report


Scrapy is a great framework for web crawling. This package provides two pipelines of saving items into MongoDB in both async and sync ways for scrapy. Also it provides a highly customized way to interact with MongoDB in both async and sync ways:

  • Save an item and get Object ID with this pipeline
  • Update an item and get Object ID with this pipeline


  • Txmongo, a async MongoDB driver with Twisted
  • Tests on Python 3.5
  • Tests on Linux, but it’s a pure python module, should work on other platforms with official python and Twisted supported


The quick way:

pip install scrapy-pipeline-mongodb

Or put this middleware just beside the scrapy project.


Set Block Inspector in ITEMPIPELINES in, for example:

from txmongo.filter import ASCENDING
from txmongo.filter import DESCENDING

# -----------------------------------------------------------------------------
# -----------------------------------------------------------------------------

    'scrapy_pipeline_mongodb.pipelines.mongodb_async.PipelineMongoDBAsync': 500,

MONGODB_HOST = 'localhost'
MONGODB_DATABASE = 'test_mongodb_async_db'
MONGODB_COLLECTION = 'test_mongodb_async_coll'


MONGODB_INDEXES = [('field_0', ASCENDING, {'unique': True}),
                   (('field_0', 'field_1'), ASCENDING),
                   (('field_0', ASCENDING), ('field_0', DESCENDING))]

MONGODB_PROCESS_ITEM = 'scrapy_pipeline_mongodb.utils.process_item.process_item'

Settings Reference


A string of the username of the database.


A string of the password of the database.


A string of the ip address or the domain of the database.


A int of the port of the database.


A string of the name of the database.


A list of the indexes to create on the collection.


Options can be attached when the pipeline start to connect to MongoBD.

If any options are needed, the option can be set with the prefix MONGODB_OPTIONS_, the pipeline will parse it.

For example:

option name in
authMechanism MONGODB_OPTIONS_authMechanism

For more options, please refer to the page:

Connection String URI Format — MongoDB Manual 3.4


A list of the indexes defined in this setting will be created when the spider is open.

If the index has already existed, the warning or error will be suspended.


This pipeline provides a setting to define the function process_item, which can help to customize the way to interact with MongoDB. With this package there is one default function provided: calling the method insert_one of collection to save the item into MongoDB, then return the item.

If a customize method is provided to replace the default one, please note the behavior should follow the requirement which is clearly written in the scrapy documents:

Item Pipeline — Scrapy 1.4.0 documentation

Built-in Functions For Processing Item


This is a built-in function to call the method insert_one of collection, and return the item.

To use this function, in

MONGODB_PROCESS_ITEM = 'scrapy_pipeline_mongodb.utils.process_item.process_item'


The database drivers may have different api for the same operation, this pipeline adopts txmongo as the async driver for MongoDB. Please read the relative documents to make sure the customized method can run fluently in this pipeline.


  • Add a unit test for the index created function
  • Add a sync pipeline

Project details

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

scrapy-pipeline-mongodb-0.0.7.tar.gz (24.0 kB view hashes)

Uploaded source

Built Distribution

Supported by

AWS AWS Cloud computing Datadog Datadog Monitoring Facebook / Instagram Facebook / Instagram PSF Sponsor Fastly Fastly CDN Google Google Object Storage and Download Analytics Huawei Huawei PSF Sponsor Microsoft Microsoft PSF Sponsor NVIDIA NVIDIA PSF Sponsor Pingdom Pingdom Monitoring Salesforce Salesforce PSF Sponsor Sentry Sentry Error logging StatusPage StatusPage Status page