一个简单快速的异步爬虫框架
Project description
HSSP 爬虫框架
一个基于python asyncio开发的爬虫框架 (开发中)
作者
特性
- 使用scrapy框架的选择器 parsel 作为内置网页选择器
- 基于tenacity的自动异常重试
- 基于fake-useragent的可选随机UA
- 可选的多种下载器: httpx、aiohttp、requests、curl-cffi、requests-go
- 请求前、响应后、重试后监听
路线
- 增加其他解析器
- 在情求过程中临时更换下载器：比如net初始化时使用的是httpx下载器，其中一个情求要临时切换至
DrissionPage, 其他的依旧是httpx - 支持
DrissionPage、playwright浏览器渲染的下载器 - 下载器支持更多配置项及自定义项
- 编写详细使用文档
安装
使用 pip 安装 hssp
pip install hssp
使用 uv 安装 hssp
uv add hssp
支持
如需支持，请发送电子邮件至 xhrtxh@gmail.com。
开发测试
项目使用uv管理依赖，需先安装uv
rye sync
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
hssp-0.4.17.tar.gz
(15.9 kB
view details)
Built Distribution
hssp-0.4.17-py3-none-any.whl
(22.2 kB
view details)
File details
Details for the file hssp-0.4.17.tar.gz.
File metadata
- Download URL: hssp-0.4.17.tar.gz
- Upload date:
- Size: 15.9 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/6.1.0 CPython/3.12.8
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
bbe0d7a5a2c83ad4793e174677a167f117bba76037124a42dc73c1ef640ba76f
|
|
| MD5 |
641e26d3b5dcb1105f1b2d56d36e99da
|
|
| BLAKE2b-256 |
cc0ed77b314dbba35db2d354315288810aa5ed696c060fa89d440165beca9481
|
File details
Details for the file hssp-0.4.17-py3-none-any.whl.
File metadata
- Download URL: hssp-0.4.17-py3-none-any.whl
- Upload date:
- Size: 22.2 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/6.1.0 CPython/3.12.8
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
7ecabfca1f9024fa3dc5d21ea34a51557341630c9efa960b71383ae58131f33a
|
|
| MD5 |
db80dad6d9ee31c5ae146ff1b81db8fe
|
|
| BLAKE2b-256 |
33f6392fcf14f337464102798ea97737cdc73912de56d7bc2ae341eecfaf1d70
|
