scrapy-inline-requests 0.3.1

A decorator for writing coroutine-like spider callbacks.

Scrapy Inline Requests

A decorator for writing coroutine-like spider callbacks.

• Free software: MIT license

• Documentation: https://scrapy-inline-requests.readthedocs.org.

• Python versions: 2.7, 3.4+

Quickstart

The spider below shows a simple use case of scraping a page and following a few links:

from inline_requests import inline_requests
from scrapy import Spider, Request

class MySpider(Spider):
    name = 'myspider'
    start_urls = ['http://httpbin.org/html']

    @inline_requests
    def parse(self, response):
        urls = [response.url]
        for i in range(10):
            next_url = response.urljoin('?page=%d' % i)
            try:
                next_resp = yield Request(next_url)
                urls.append(next_resp.url)
            except Exception:
                self.logger.info("Failed request %s", i, exc_info=True)

        yield {'urls': urls}

See the examples/ directory for a more complex spider.

Warning

The generator resumes its execution when a request’s response is processed, this means the generator won’t be resume after yielding an item or a request with it’s own callback.

Known Issues

• Middlewares can drop or ignore non-200 status responses causing the callback to not continue its execution. This can be overcome by using the flag handle_httpstatus_all. See the httperror middleware documentation.

• High concurrency and large responses can cause higher memory usage.

• This decorator assumes your method have the following signature (self, response).

• Wrapped requests may not be able to be serialized by persistent backends.

• Unless you know what you are doing, the decorated method must be a spider method and return a generator instance.

History

0.3.1 (2016-07-04)

• Added deprecation about decorating non-spider functions.

• Warn if the callback returns requests with callback or errback set. This reverts the compability with requests with callbacks.

0.3.0 (2016-06-24)

• ~~Backward incompatible change: Added more restrictions to the request object (no callback/errback).~~

• Cleanup callback/errback attributes before sending back the request to the generator. This fixes an edge case when using request.replace().

• Simplified example spider.

0.2.0 (2016-06-23)

• Python 3 support.

0.1.2 (2016-05-22)

• Scrapy API and documentation updates.

0.1.1 (2013-02-03)

• Minor tweaks and fixes.

0.1.0 (2012-02-03)

• First release on PyPI.