scrapy-pipeline-mongodb 0.0.7

Overview

Scrapy is a great framework for web crawling. This package provides two pipelines of saving items into MongoDB in both async and sync ways for scrapy. Also it provides a highly customized way to interact with MongoDB in both async and sync ways:

• Save an item and get Object ID with this pipeline

• Update an item and get Object ID with this pipeline

Requirements

• Txmongo, a async MongoDB driver with Twisted

• Tests on Python 3.5

• Tests on Linux, but it’s a pure python module, should work on other platforms with official python and Twisted supported

Installation

The quick way:

pip install scrapy-pipeline-mongodb

Or put this middleware just beside the scrapy project.

Documentation

Set Block Inspector in ITEMPIPELINES in settings.py, for example:

from txmongo.filter import ASCENDING
from txmongo.filter import DESCENDING

# -----------------------------------------------------------------------------
# PIPELINE MONGODB ASYNC
# -----------------------------------------------------------------------------

ITEM_PIPELINES.update({
    'scrapy_pipeline_mongodb.pipelines.mongodb_async.PipelineMongoDBAsync': 500,
})

MONGODB_USERNAME = 'user'
MONGODB_PASSWORD = 'password'
MONGODB_HOST = 'localhost'
MONGODB_PORT = 27017
MONGODB_DATABASE = 'test_mongodb_async_db'
MONGODB_COLLECTION = 'test_mongodb_async_coll'

# MONGODB_OPTIONS_ = 'MONGODB_OPTIONS_'

MONGODB_INDEXES = [('field_0', ASCENDING, {'unique': True}),
                   (('field_0', 'field_1'), ASCENDING),
                   (('field_0', ASCENDING), ('field_0', DESCENDING))]

MONGODB_PROCESS_ITEM = 'scrapy_pipeline_mongodb.utils.process_item.process_item'

Settings Reference

Built-in Functions For Processing Item

scrapy_pipeline_mongodb.utils.process_item.process_item

This is a built-in function to call the method insert_one of collection, and return the item.

To use this function, in settings.py:

MONGODB_PROCESS_ITEM = 'scrapy_pipeline_mongodb.utils.process_item.process_item'

NOTE

The database drivers may have different api for the same operation, this pipeline adopts txmongo as the async driver for MongoDB. Please read the relative documents to make sure the customized method can run fluently in this pipeline.

• Welcome to TxMongo’s documentation!

• pymongo – Python driver for MongoDB

TODO

• Add a unit test for the index created function

• Add a sync pipeline