os-rq-scrapy
A framework for Scrapy working with os-rq-pod and os-rq-hub to build "broad crawls" system.
As you know, Scrapy is a very popular python crawler framework. It is suit for "focused crawl", start from several URLs of specific sites, fetch html, extract and save "structured data" also with patternd links to crawl recursively. But for large scale, long time crawling especially "broad crawls", scrapy is incompetent. Basically, you have to decouple the whole crawling system into several sub-systems, high-performance full-featured distributed fetcher, task scheduler, html extractor, link database, data storage, proxy and a lot of auxiliary equipments. It will be more complex when your system is for multi-tenancy.
The os-rq-scrapy and os-rq-pod project are basic components to build "broad crawls" system. The core conceptions are very simple, os-rq-pod is multi-sites request queue have http api to recieve requests. os-rq-scrapy is fetcher, getting requests from os-rq-pod and crawl multi-sites concurrently. os-rq-hub can also be used to connect multi pod and scrapy instances to work simultaneously.
Requirements
- Python 3.6+ (pypy3.6+)
- Scrapy 2.0
extra requirements:
- ujson, for json performance
Install
pip install os-rq-scrapy
Usage
Command line
rq-scrapy command enhance the basic scrapy command. When RQ_API is configured, the crawl subcommand will run on rq mode, get requests from rq.
Unit Tests
tox
License
MIT licensed.
1 maintainer
Metadata
1 maintainer
Release files for os-rq-scrapy 0.0.7
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | |
|---|---|---|---|---|
| os-rq-scrapy-0.0.7.tar.gz | 13.9 kB | Without provenance | Details |
Release files / os-rq-scrapy-0.0.7.tar.gz
| Download URL | os-rq-scrapy-0.0.7.tar.gz |
|---|---|
| Size | 13.9 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
3ada9612a96b6e11e1f887fa5e7b189ac31f1e39347cae46c327750fd6812034
|
|
BLAKE2b-256 checksum How to use checksums |
00ca974fdc645ff6521677b6629fcf0726fe5df1e9fa79b1461fb720c455c67f
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/3.1.1 pkginfo/1.5.0.1 requests/2.23.0 setuptools/46.1.3 requests-toolbelt/0.9.1 tqdm/4.45.0 CPython/3.6.7
|
Security for os-rq-scrapy 0.0.7
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.
