hoopa
简介
hoopa 是一个轻量、快速的异步分布式爬虫框架
- 支持内存、redis 的优先级队列
- 支持 aiohttp、 httpx、requests 等 HTTP 库
- 支持断点续传
兼容同步和异步代码，不习惯异步的，可以使用同步写，但是要注意的是不能在异步方法里面进行阻塞的操作
自用框架，不保证稳定性，请勿用于生产环境
文档地址：https://fishtn.github.io/hoopa/
环境要求：
- Python 3.7.0+
- Works on Linux, Windows, macOS
安装
# For Linux & Mac
pip install -U hoopa[uvloop]
# For Windows
pip install -U hoopa
开始
创建爬虫
hoopa create -s first_spider
然后添加 url：http://httpbin.org/get
import hoopa
class FirstSpider(hoopa.Spider):
name = "first"
start_urls = ["http://httpbin.org/get"]
def parse(self, request, response):
print(response)
if __name__ == "__main__":
FirstSpider.start()
todo
- 监控平台
- 远程部署
- 任务调度
感谢
1 maintainer
Release files for hoopa 0.1.18
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| hoopa-0.1.18.tar.gz | 39.9 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| hoopa-0.1.18-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 92.6 kB
Release files / hoopa-0.1.18.tar.gz
| Download URL | hoopa-0.1.18.tar.gz |
|---|---|
| Size | 39.9 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
9f87016f76aff5e2ef4f8dca17a0e0296acf4afc1d1aabd7d2e49cc8b9e747f3
|
|
BLAKE2b-256 checksum How to use checksums |
4d6e8946524bf90281fa96fb85e3f19c6ab2e1152273528544a7858d2265783d
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.1.0 CPython/3.12.9
|
Release files / hoopa-0.1.18-py3-none-any.whl
| Download URL | hoopa-0.1.18-py3-none-any.whl |
|---|---|
| Size | 52.7 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
9c86ea135c98881e3b05fffca851fbceb1819b19a73231ccb93784714ecdf6a5
|
|
BLAKE2b-256 checksum How to use checksums |
d8f46f1eb64e617fc7eb364be87345b25adab181b08fea7ca26d73380bc99fb1
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.1.0 CPython/3.12.9
|
