爬虫系统业务层封装
Project description
爬虫系统公共类封装
目标
- redis db 划分
- url 记录池封装
- 下载页面池封装
- 解析结果池封装
- 网站信息类封装
redis db 划分(0-15)
- 0 -> 常用的队列与有序集或集合
其中的hash表定义
- website -> 网站信息
- url_record -> url 记录池
- url_page -> 下载页面池封装
- parse_result -> 解析结果池封装
- 10-15 -> 监控自行处理
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
qg_spider_sdk-5.2.9.tar.gz
(21.4 kB
view hashes)
Built Distribution
Close
Hashes for qg_spider_sdk-5.2.9-py3-none-any.whl
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | 53a2d834e69af97d51da90c0ab7b40d1def97d6a803630a9c38117df4fdbd8d8 |
|
| MD5 | c79c20e6539b525c55f4d8b13c8b4568 |
|
| BLAKE2b-256 | 18d9e474a91092cd9c151db0f4a13f21bc66d7a37b28c92eca2e9c1e11066c4b |
