This is a Scrapy spider middleware to ignore requests to pages seen in previous crawls of the same spider, thus producing a “delta crawl” containing only new requests.
This also speeds up the crawl, by reducing the number of requests that need to be crawled, and processed (typically, item requests are the most CPU intensive).
DeltaFetch middleware uses Python’s dbm package to store requests fingerprints.
Installation
Install scrapy-deltafetch using pip:
$ pip install scrapy-deltafetch
Configuration
Add DeltaFetch middleware by including it in SPIDER_MIDDLEWARES in your settings.py file:
SPIDER_MIDDLEWARES = { 'scrapy_deltafetch.DeltaFetch': 100, }Here, priority 100 is just an example. Set its value depending on other middlewares you may have enabled already.
Enable the middleware using DELTAFETCH_ENABLED in your settings.py:
DELTAFETCH_ENABLED = True
Usage
Following are the different options to control DeltaFetch middleware behavior.
Supported Scrapy settings
DELTAFETCH_ENABLED — to enable (or disable) this extension
DELTAFETCH_DIR — directory where to store state
DELTAFETCH_RESET — reset the state, clearing out all seen requests
These usually go in your Scrapy project’s settings.py.
Supported Scrapy spider arguments
deltafetch_reset — same effect as DELTAFETCH_RESET setting
Example:
$ scrapy crawl example -a deltafetch_reset=1
Supported Scrapy request meta keys
deltafetch_key — used to define the lookup key for that request. by default it’s Scrapy’s default Request fingerprint function, but it can be changed to contain an item id, for example. This requires support from the spider, but makes the extension more efficient for sites that many URLs for the same item.
deltafetch_enabled - if set to False it will disable deltafetch for some specific request
4 maintainers
Metadata
4 maintainers
Release files for scrapy-deltafetch 2.1.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| scrapy_deltafetch-2.1.0.tar.gz | 7.6 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| scrapy_deltafetch-2.1.0-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 11.7 kB
Release files / scrapy_deltafetch-2.1.0.tar.gz
| Download URL | scrapy_deltafetch-2.1.0.tar.gz |
|---|---|
| Size | 7.6 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
66dbc1d7483a8ff24d729c8b6494b847ef030bb4d883431640cbccd2771a2713
|
|
BLAKE2b-256 checksum How to use checksums |
33827674b8c5ab5f40bfbbd0f910ac1b00910590e38170adde773c1a3d62c846
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/6.1.0 CPython/3.12.9
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Feb 26, 2025.
Transparency logRelease files / scrapy_deltafetch-2.1.0-py3-none-any.whl
| Download URL | scrapy_deltafetch-2.1.0-py3-none-any.whl |
|---|---|
| Size | 4.1 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
1505cc5846ba1c8534b0cc66176c18eed04012780dbc335baecdc6680672a7cb
|
|
BLAKE2b-256 checksum How to use checksums |
4d751c47c74c995bd711ce72149672340bb599acf7fa0e03cdf003c4e7853d5f
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/6.1.0 CPython/3.12.9
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Feb 26, 2025.
Transparency log
