Skip to main content
This is a pre-production deployment of Warehouse. Changes made here affect the production instance of PyPI (pypi.python.org).
Help us improve Python packaging - Donate today!

Handy tools to maintain persistent meta values between requests in Scrapy spiders

Project Description

Handy tools to maintain persistent meta values between requests in Scrapy spiders. Available as spider middleware and spider callback decorators.

Installation

pip install stickymeta

Usage

As spider middleware

Add middleware to settings.py:

SPIDER_MIDDLEWARES = {
    ...
    'stickymeta.StickyMetaMiddleware': 543,
    ...
}

and sticky_meta attribute containing persistent meta keys to spider:

class TheSpider(scrapy.Spider):
    name = 'thespider'
    sticky_meta = ('cookiejar', 'foo', 'bar')

All values for the corresponding keys will be kept persistent between all the requests and responses.

As decorators

@stick_meta

from stickymeta import stick_meta

Keep persistent values for keys passed as decorator parameters:

@stick_meta('a', 'b', 'c')
def parse(self, response):
    ...
    yield scrapy.Request(url)

is equivalent to:

def parse(self, response)
    ...
    meta = {
        'a': response.meta['a'],
        'b': response.meta['b'],
        'c': response.meta['c'],
    }
    yield scrapy.Request(url, meta=meta}

@stick_cj

from stickymeta import stick_cj

Shortcut for stick_meta handling cookiejar as default argument value, so

@stick_cj('a', 'b', 'c')
def parse(self,response):
    ...

is equivalent to

@stick_meta('cookiejar', 'a', 'b', 'c')
def parse(self,response):
    ...

Spider attribute sticky_meta also affects to decorators, next two pieces of code will handle meta in the same way:

class TheSpider(scrapy.Spider):
    name = 'thespider'
    sticky_meta = ('a', 'b', 'c')

    @stick_meta()
    def parse(self, response):
        ...
        yield Request(url)

vs

class TheSpider(scrapy.Spider):
    name = 'thespider'

    @stick_meta('a', 'b', 'c')
    def parse(self, response):
        ...
        yield Request(url)
Release History

Release History

This version
History Node

0.0.5

History Node

0.0.4

History Node

0.0.3

Download Files

Download Files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

File Name & Checksum SHA256 Checksum Help Version File Type Upload Date
stickymeta-0.0.5-py2-none-any.whl (6.0 kB) Copy SHA256 Checksum SHA256 2.7 Wheel Jan 29, 2017
stickymeta-0.0.5.tar.gz (2.8 kB) Copy SHA256 Checksum SHA256 Source Jan 29, 2017

Supported By

WebFaction WebFaction Technical Writing Elastic Elastic Search Pingdom Pingdom Monitoring Dyn Dyn DNS Sentry Sentry Error Logging CloudAMQP CloudAMQP RabbitMQ Heroku Heroku PaaS Kabu Creative Kabu Creative UX & Design Fastly Fastly CDN DigiCert DigiCert EV Certificate Rackspace Rackspace Cloud Servers DreamHost DreamHost Log Hosting