Handy tools to maintain persistent meta values between requests in Scrapy spiders
Project description
Handy tools to maintain persistent meta values between requests in Scrapy spiders. Available as spider middleware and spider callback decorators.
Installation
pip install stickymeta
Usage
As spider middleware
Add middleware to settings.py:
SPIDER_MIDDLEWARES = { ... 'stickymeta.StickyMetaMiddleware': 543, ... }
and sticky_meta attribute containing persistent meta keys to spider:
class TheSpider(scrapy.Spider): name = 'thespider' sticky_meta = ('cookiejar', 'foo', 'bar')
All values for the corresponding keys will be kept persistent between all the requests and responses.
As decorators
@stick_meta
from stickymeta import stick_meta
Keep persistent values for keys passed as decorator parameters:
@stick_meta('a', 'b', 'c') def parse(self, response): ... yield scrapy.Request(url)
is equivalent to:
def parse(self, response) ... meta = { 'a': response.meta['a'], 'b': response.meta['b'], 'c': response.meta['c'], } yield scrapy.Request(url, meta=meta}
@stick_cj
from stickymeta import stick_cj
Shortcut for stick_meta handling cookiejar as default argument value, so
@stick_cj('a', 'b', 'c') def parse(self,response): ...
is equivalent to
@stick_meta('cookiejar', 'a', 'b', 'c') def parse(self,response): ...
Spider attribute sticky_meta also affects to decorators, next two pieces of code will handle meta in the same way:
class TheSpider(scrapy.Spider): name = 'thespider' sticky_meta = ('a', 'b', 'c') @stick_meta() def parse(self, response): ... yield Request(url)
vs
class TheSpider(scrapy.Spider): name = 'thespider' @stick_meta('a', 'b', 'c') def parse(self, response): ... yield Request(url)
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for stickymeta-0.0.5-py2-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 5edda9545bf4b15ad93d1879b4c3ebd194d504810df2c9e132a428b69e2133cc |
|
MD5 | 91efb3401fba630bd107fa169c214f11 |
|
BLAKE2b-256 | 6fad5294c09bebba39ce581920d2b7a2b3fd589abe148274191a4b054fe7dde4 |