MongoDB plugins for Scrapy

Installation

pip install scrapy-mongo

Pipeline

This pipeline stores scraped items into a MongoDB collection.

Each item must have a unique id field to avoid duplicates. This field is automatically mapped to MongoDB’s _id field.

Each item must include a collection field that specifies the name of the target MongoDB collection.

Items are upserted in batches of 100 by default. The batch size can be adjusted using the PIPELINE_MONGO_BATCH_SIZE setting.

To enable the pipeline, include the following lines in settings.py:

ITEM_PIPELINES = {
    'scrapy_mongo.MongoPipeline': 300,
}
PIPELINE_MONGO_URL = "mongodb://localhost:27017"
PIPELINE_MONGO_DATABASE = "mycollection"

Note: Update PIPELINE_MONGO_URL and PIPELINE_MONGO_DATABASE with the appropriate values for the specific environment.

Cache

The cache component stores scraped responses in a MongoDB collection to avoid downloading the same pages multiple times. It leverages Scrapy’s fingerprinting mechanism to identify responses.

It uses Scrapy's fingerprint mechanism to identify the responses.

To enable caching, include the following lines in settings.py:

HTTPCACHE_ENABLED = True
HTTPCACHE_STORAGE = 'scrapy_mongo.MongoCacheStorage'
HTTPCACHE_MONGO_URL = "mongodb://localhost:27017"
HTTPCACHE_MONGO_DATABASE = "scraping"
HTTPCACHE_EXPIRATION_SECS = 604800  # Default is 1 week

Note: Update HTTPCACHE_MONGO_URL and HTTPCACHE_MONGO_DATABASE with the appropriate values for the specific environment.

The default expiration time is set to 1 week (604800 seconds). This value can be modified via HTTPCACHE_EXPIRATION_SECS.

Cache policy

An advanced cache policy mechanism with whitelist support is available. This feature allows for the definition of specific HTTP response codes to be cached, using both explicit lists and regular expressions.

To enable the cache policy, add the following lines to settings.py:

HTTPCACHE_ENABLED = True
HTTPCACHE_POLICY = 'scrapy_mongo.CacheOnlyPolicy'
HTTPCACHE_ACCEPT_HTTP_CODES = [302]
HTTPCACHE_ACCEPT_HTTP_CODES_REGEX = r'2\d\d'

This configuration will accept all 2XX HTTP codes and 302 redirects.

Error

The error component stores error logs in a MongoDB collection. It catches error from the Downloader pipeline and the Spider pipeline.

To enable error logging, include the following lines in settings.py:

DOWNLOADER_MIDDLEWARES = {
    'scrapy_mongo.TraceErrorDownloaderMiddleware': 1000,
}

SPIDER_MIDDLEWARES = {
    'scrapy_mongo.TraceErrorSpiderMiddleware': 1000,
}

ERROR_MONGO_URL = "mongodb://localhost:27017"
ERROR_MONGO_DATABASE = 'scraping'
ERROR_MONGO_COLLECTION = 'errors'

Note: Update ERROR_MONGO_URL, ERROR_MONGO_DATABASE and ERROR_MONGO_COLLECTION with the appropriate values for the specific environment.

It is possible to use the same MongoDB connection for both the pipeline and cache by replacing PIPELINE_MONGO_URL, HTTPCACHE_MONGO_URL and ERROR_MONGO_URL with a unified MONGO_URL setting.

Build for publish

Install dependencies:

pip install build twine

Build the package:

python -m build --outdir dist

And publish to PyPi:

python -m twine upload dist/*