MineWorker
一个上手简单、结构清晰的 Python 爬虫框架，对标 feapder：
你只写 start_requests 和 parse，框架负责调度、下载、重试、去重、批量落库。
0.15.0 —— 单机（
AirSpider）到分布式（Spider/TaskSpider/BatchSpider）全部可用， 并支持浏览器 TLS 指纹伪装。 变更见 CHANGELOG，后续规划见 Roadmap。
安装
pip install mineworker # 核心
pip install "mineworker[all]" # 含渲染 / 各类存储 / Redis / CLI / 指标 / 指纹伪装
也可以按需装单项：render · mongo · mysql · postgres · elasticsearch · kafka ·
redis · cli · metrics · curl。
快速开始
pip install "mineworker[cli]"
mineworker create -p news_crawler
cd news_crawler && python main.py
或直接写：
import mineworker as mw
class NewsSpider(mw.AirSpider):
def start_requests(self):
yield mw.Request("https://news.ycombinator.com/", callback=self.parse)
def parse(self, request, response):
for a in response.css("span.titleline > a"):
yield {"title": a.css("::text").get(), "url": a.css("::attr(href)").get()}
NewsSpider().start()
示例
examples/ 里有可以直接跑的完整例子：
python examples/books_toscrape.py # 两级抓取：列表页翻页 → 详情页
文档
完整文档：https://apersonw.github.io/mineworker/
（本地预览：pip install "mineworker[docs]" && mkdocs serve）
设计与实施计划见 IMPLEMENTATION_PLAN.md。
能力一览
| 运行时 | 单进程多线程、优先级队列、结束检测、Ctrl-C 优雅排空、崩溃 dump + retry 回放 |
| 网络 | Request/Response（httpx + parsel）、自动重试、validate/failed_request 钩子、随机 UA、会话复用 |
| 礼貌性 | robots.txt（含 Crawl-delay）、per-domain 限速 + 单域并发上限、跨节点全局限速（Redis）、429/503 读 Retry-After 整域降速 |
| 稳健性 | 状态码策略（错误页不当数据入库）、指数退避 + 抖动、同域连续失败熔断、运行时长上限、响应体大小上限 + Content-Type 白名单、SIGINT/SIGTERM 优雅停止、任务租约 + 落库后才销账（节点被硬杀，任务能回收、数据不丢） |
| 分布式 | Spider：Redis 队列 + 布隆去重 + 断点续爬 + 种子一次性锁 + 多节点心跳结束检测 |
| 任务驱动 | TaskSpider 从任务源持续消费；BatchSpider 周期批次采集（任务表状态机 + 进度追踪 + 防丢） |
| 数据 | Item/UpdateItem、Pipeline（Console/CSV/MongoDB/MySQL/PostgreSQL/Elasticsearch/Kafka）、请求级 + Item 级去重（布隆/精确） |
| 渲染 | Request(render=True) —— Playwright 渲染池、wait_for/render_time/render_script |
| 反爬 | TLS / HTTP2 指纹伪装（impersonate 真实浏览器）、Cloudflare / Akamai 挑战页识别 |
| 扩展 | 下载中间件链、代理池 / 账号池（耗尽时不会悄悄降级成直连或匿名）、掉登录自动换号 |
| 观测 | Prometheus exporter、卡死/失败率告警（飞书 / 钉钉 / 企业微信 / 邮件，发送失败会记录）、debug=True |
| 开发体验 | 响应缓存（重跑读本地文件，不再反复打目标站） |
| 工具 | mineworker create/shell/retry/cache，create -i --table 读表结构反射生成 Item |
开发
conda env create -f environment.yml && conda activate mineworker
pre-commit install
pytest && ruff check . && mypy
CI（GitHub Actions）：ruff + mypy(strict) + pytest（Python 3.10–3.13）+ mkdocs build。
License
1 maintainer
Release files for mineworker 0.18.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| mineworker-0.18.0.tar.gz | 366.4 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| mineworker-0.18.0-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 542.6 kB
Release files / mineworker-0.18.0.tar.gz
| Download URL | mineworker-0.18.0.tar.gz |
|---|---|
| Size | 366.4 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
fe595e861850d2375827039d9a13a284acac26aac0ce282b263137ca664c509b
|
|
BLAKE2b-256 checksum How to use checksums |
f5317600174f45fcde3d190d1a964841608ef7d8e4bfe5655400398e3daa671d
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Sep 9, 2026.
Transparency logRelease files / mineworker-0.18.0-py3-none-any.whl
| Download URL | mineworker-0.18.0-py3-none-any.whl |
|---|---|
| Size | 176.3 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
5d1f5962578e6c492d316d037f76f683a7be47e5c465ad204954120deeb98406
|
|
BLAKE2b-256 checksum How to use checksums |
a38c77c3855cd5fda5a7e7feed802c6806c3b615cddc6d6323b2f2345292fbb2
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Sep 9, 2026.
Transparency log
