Skip to main content

Scrapy Item Record Extension

Project description

os-scrapy-record

Build Status codecov PyPI - Python Version PyPI

This project provide extensions to process Response/Failure, generate standard Item.

Install

pip install os-scrapy-record

You can run example spider directly in the project root path

scrapy crawl example

APIs

  • os_scrapy_record.ResponseCallback

    • the callback method of this extension will replace the default Request.callback, process Response and generate FetchRecord
    • the callback method will not work when the request already set callback function
    • the callback method will override the parse method of spider
    • enable extension in the project settings.py file:
    EXTENSIONS = {
        "os_scrapy_record.ResponseCallback": 1,
    }
    
  • os_scrapy_record.ResponseErrback

    • the errback method of this extension will replace the default Request.errback, process Failure and generate FetchRecord
    • the errback method will not work when the request already set errback function
    • enable extension in the project settings.py file:
    EXTENSIONS = {
        "os_scrapy_record.ResponseErrback": 1,
    }
    
  • os_scrapy_record.FetchRecord

    This class is subclass of Item

    the mumbers of this class are:

    • request: os_scrapy_record.items.RequestItem, members: url, method, headers, body
    • meta: dict, request.meta, it is better to use lower case and '_' as separator as key
    • response: os_scrapy_record.items.ResponseItem,members: headers, body, status, ip_address(Scrapy 2.1.0+), failure
  • os_scrapy_record.fetch_status.FetchStatus

    A mumber of ResponseItem, include HTTP, DNS, Network and user defined status. It is a two-tuple object: group and code. e.g, HTTP:200, DNS:-2, SERVER:111, RULE:16

Unit Tests

sh scripts/test.sh

License

MIT licensed.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

os-scrapy-record-0.0.15.tar.gz (10.4 kB view hashes)

Uploaded source

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page