os-scrapy-record

This project provide extensions to process Response/Failure, generate standard Item.

Install

pip install os-scrapy-record

You can run example spider directly in the project root path

scrapy crawl example

APIs

os_scrapy_record.ResponseCallback
- the callback method of this extension will replace the default Request.callback, process Response and generate FetchRecord
- the callback method will not work when the request already set callback function
- the callback method will override the parse method of spider
- enable extension in the project settings.py file:
```
EXTENSIONS = {
    "os_scrapy_record.ResponseCallback": 1,
}
```
os_scrapy_record.ResponseErrback
- the errback method of this extension will replace the default Request.errback, process Failure and generate FetchRecord
- the errback method will not work when the request already set errback function
- enable extension in the project settings.py file:
```
EXTENSIONS = {
    "os_scrapy_record.ResponseErrback": 1,
}
```
os_scrapy_record.FetchRecord

This class is subclass of Item

the mumbers of this class are:
- request: os_scrapy_record.items.RequestItem, members: url, method, headers, body
- meta: dict, request.meta, it is better to use lower case and '_' as separator as key
- response: os_scrapy_record.items.ResponseItem，members: headers, body, status, ip_address(Scrapy 2.1.0+), failure
os_scrapy_record.fetch_status.FetchStatus

A mumber of ResponseItem, include HTTP, DNS, Network and user defined status. It is a two-tuple object: group and code. e.g, HTTP:200, DNS:-2, SERVER:111, RULE:16

Unit Tests

sh scripts/test.sh

License

MIT licensed.

Metadata

Release files for os-scrapy-record 0.0.15

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for os-scrapy-record 0.0.15
File	Size	Uploaded	Provenance
os-scrapy-record-0.0.15.tar.gz	10.4 kB	Dec 24, 2020	Without provenance	Details

Release files / os-scrapy-record-0.0.15.tar.gz

Download URL	os-scrapy-record-0.0.15.tar.gz
Size	10.4 kB
Tags	Source
SHA-256 checksum How to use checksums	`bb82c668c620c524fa50bb8397bd5724ca27bd2bccc60289d9da26ba0e7863f5`
BLAKE2b-256 checksum How to use checksums	`4d67504dac092a1fa7d1a5cf57f0d608928232db6a65d2c13a5d6438805d0e24`
Upload date	Dec 24, 2020
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/3.3.0 pkginfo/1.6.1 requests/2.25.1 setuptools/51.1.0 requests-toolbelt/0.9.1 tqdm/4.54.1 CPython/3.6.7`

Security for os-scrapy-record 0.0.15

Report project as malware

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.