scrapy-pagestorage

A scrapy extension to store requests and responses information in storage service.

Installation

You can install scrapy-pagestorage using pip:

pip install scrapy-pagestorage

You can then enable the middleware in your settings.py:

SPIDER_MIDDLEWARES = {
    ...
    'scrapy_pagestorage.PageStorageMiddleware': 900
}

How to use it

Enable extension through settings.py:

PAGE_STORAGE_ENABLED = True
PAGE_STORAGE_ON_ERROR_ENABLED = True

Configure the exension through settings.py:

PAGE_STORAGE_MODE = "VERSIONED_CACHE"
PAGE_STORAGE_LIMIT = 100
PAGE_STORAGE_ON_ERROR_LIMIT = 100
PAGE_STORAGE_TRIM_HTML = True

The extension is auto-enabled for Portia spiders (SHUB_SPIDER_TYPE=portia).

Settings

PAGE_STORAGE_MODE

Default: None

A string which specifies if the extension will store information using cache store or versioned cache store (set PAGE_STORAGE_MODE=”VERSIONED_CACHE” to use versioned one).

PAGE_STORAGE_LIMIT

An integer to set a limit of visited pages amount to store.

PAGE_STORAGE_ON_ERROR_LIMIT

An integer to set a limit for page errors amount to store.

PAGE_STORAGE_TRIM_HTML

Default: False

Remove whitespace from the start and end of the HTML to reduce file size.

Metadata

Release files for scrapy-pagestorage 0.4.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for scrapy-pagestorage 0.4.0
File	Size	Uploaded
scrapy-pagestorage-0.4.0.tar.gz	4.7 kB	Mar 11, 2022	Details

Built distribution (wheel)

Table of built distributions (wheels) for scrapy-pagestorage 0.4.0
File	Interpreter	ABI	Platform	Reset
scrapy_pagestorage-0.4.0-py2.py3-none-any.whl 4.9 kB Mar 11, 2022	Python 2, Python 3	none	any	Details

Total release size: 9.6 kB

Release files / scrapy-pagestorage-0.4.0.tar.gz

Download URL	scrapy-pagestorage-0.4.0.tar.gz
Size	4.7 kB
Tags	Source
SHA-256 checksum How to use checksums	`344b906ddc2e5ec1dcddbaf202c00534a0d17b58d66343dda41b0a171554ba78`
BLAKE2b-256 checksum How to use checksums	`d4a1bb6d774bdd5d5eb911deb2ed33fa29dd4dad4ae3efdab8991dfe6eaf0d14`
Upload date	Mar 11, 2022
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/3.8.0 pkginfo/1.8.2 readme-renderer/33.0 requests/2.27.1 requests-toolbelt/0.9.1 urllib3/1.26.8 tqdm/4.63.0 importlib-metadata/4.11.2 keyring/23.5.0 rfc3986/2.0.0 colorama/0.4.4 CPython/3.9.10`

Release files / scrapy_pagestorage-0.4.0-py2.py3-none-any.whl

Download URL	scrapy_pagestorage-0.4.0-py2.py3-none-any.whl
Size	4.9 kB
Tags	Python 2 Python 3
SHA-256 checksum How to use checksums	`3e5d1472bbc4623ee8c985b6968ede64c03dd18b15abad08c0a146cd00947c73`
BLAKE2b-256 checksum How to use checksums	`93236abe5290e9451234c14accfd1086453a591178f00197e03cf07afd1c3fb3`
Upload date	Mar 11, 2022
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/3.8.0 pkginfo/1.8.2 readme-renderer/33.0 requests/2.27.1 requests-toolbelt/0.9.1 urllib3/1.26.8 tqdm/4.63.0 importlib-metadata/4.11.2 keyring/23.5.0 rfc3986/2.0.0 colorama/0.4.4 CPython/3.9.10`