Scrapy-QOS
QOS components for Scrapy
Usage
Active the QosDownloaderMiddleware in settings.py
DOWNLOADER_MIDDLEWARES = {
"scrapy_qos.QosDownloaderMiddleware": 543
}
Config following option in settings.py
- QOS_IOPS_ENABLED
- default
False - set
Trueto enable IOPS limiter
- default
- QOS_IOPS_CAPACITY
- default
1 - burst IO count per seconds
- default
- QOS_IOPS_LIMIT
- default
1/ s - how many requests sent per seconds
- default
- QOS_BPS_ENABLED
- default
False - set
Trueto enable BPS limiter
- default
- QOS_BPS_CAPACITY
- default
1048576Bytes - burst IO Bytes per seconds
- default
- QOS_BPS_LIMIT
- default
1048576Bytes / s - how many response Bytes receive per seconds
- default
- QOS_SMALL_RESPONSE_SIZE
- default
1048576Bytes - guess next response size filter response less than this value
- default
Requirements
- Python 3.7+
- Scrapy >= 2.0
- asyncio
Installation
From pip
pip install scrapy-qos
From Gitee
git clone https://gitee.com/hgdsdq/scrapy_qos.git
cd scrapy_qos
python setup.py install
Implementation
- Basic implement QOS with Token Bucket Algorithm
- For scrapy, QosDownloaderMiddleware will guess next response body size that used for BPS limiter
α = 0.8
guess_response_size = (1 - α) * guess_response_size + α * guess_response_size
1 maintainer
Metadata
1 maintainer
Release files for scrapy-qos 0.0.2
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | |
|---|---|---|---|---|
| scrapy-qos-0.0.2.tar.gz | 3.5 kB | Without provenance | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| scrapy_qos-0.0.2-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 7.2 kB
Release files / scrapy-qos-0.0.2.tar.gz
| Download URL | scrapy-qos-0.0.2.tar.gz |
|---|---|
| Size | 3.5 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
b2dc10b98f12ad64bb6754061845854f2e07283aa695a4b4f2f9b63734b1a17d
|
|
BLAKE2b-256 checksum How to use checksums |
984a35c02a605db99de581645acacb4b5708468bc3ec5ca3e43b429bb5b5052c
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/4.0.2 CPython/3.10.6
|
Release files / scrapy_qos-0.0.2-py3-none-any.whl
| Download URL | scrapy_qos-0.0.2-py3-none-any.whl |
|---|---|
| Size | 3.7 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
97051473db34c37e2b38ec4fb1e8d3da3a2b1390f735cd4dfed71acd0cfe0c43
|
|
BLAKE2b-256 checksum How to use checksums |
0feb7c3dd1a218abb9463e058aadba796881fea99f554ff56a656f567192cb4f
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/4.0.2 CPython/3.10.6
|
Security for scrapy-qos 0.0.2
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
