CrawlerKeeper SDK for Python
CrawlerKeeper Python SDK 提供了爬虫与 Agent 通信的封装,支持日志记录、结果写入、参数获取、代理管理和并发爬取。
安装
pip install crawlerkeeper-sdk
快速开始
from crawlerkeeper_sdk import params, info, result, StealthSession
# 获取任务参数
config = params()
info(f"收到参数: {config}")
# 使用 StealthSession 发送请求
session = StealthSession()
resp = session.get("https://example.com/api/data")
data = resp.json()
# 写入结果
result({"data": data, "source": "example.com"})
info("爬取完成")
session.close()
核心功能
日志记录
from crawlerkeeper_sdk import info, warning, error
info("信息日志")
warning("警告日志")
error("错误日志")
结果写入
from crawlerkeeper_sdk import result
result({"count": 100, "items": [...]})
任务参数
from crawlerkeeper_sdk import params
# python spider.py --params '{"keyword": "python", "page": 1}'
config = params() # {"keyword": "python", "page": 1}
StealthSession
StealthSession 提供隐式浏览器指纹的 HTTP 请求能力,自动管理 TLS 指纹和代理。
from crawlerkeeper_sdk import StealthSession, result
session = StealthSession()
# GET 请求
resp = session.get("https://example.com/api/data")
print(resp.status_code)
print(resp.json())
# POST 请求
resp = session.post("https://example.com/api/submit", json={"key": "value"})
session.close()
主要方法
| 方法 | 说明 |
|---|---|
get(url, **kwargs) |
发送 GET 请求 |
post(url, **kwargs) |
发送 POST 请求 |
close() |
关闭会话 |
ThreadPool 并发爬取
ThreadPool 提供并发爬取能力,内部管理 StealthSession 的创建和关闭。
推荐用法:使用上下文管理器
from crawlerkeeper_sdk import ThreadPool, StealthSession, result
def crawl(session: StealthSession, keyword: str, page: int):
"""爬虫函数:session 会自动传入"""
resp = session.get(f"https://example.com/search?kw={keyword}&page={page}")
return resp.json()
# 使用 with 语句自动管理生命周期,不会遗漏 close()
with ThreadPool(max_workers=5) as pool:
pool.submit(crawl, keyword="python", page=1)
pool.submit(crawl, keyword="java", page=1)
pool.submit(crawl, keyword="go", page=1)
# 阻塞等待所有任务完成
results = pool.wait()
# 处理结果:聚合后一次性写入(result 每次调用会整体覆写数据文件,
# 循环内逐条调用只会保留最后一条,必须先聚合再单次调用)
collected = [data for data in results if data is not None]
result(collected)
# with 块结束时自动调用 wait() 和 close()
传统用法
pool = ThreadPool(max_workers=5)
pool.submit(crawl, keyword="python", page=1)
pool.submit(crawl, keyword="java", page=1)
results = pool.wait()
pool.close() # 需要手动调用
环境变量配置并发数
export CONCURRENT_NUM=10
# 不指定 max_workers 时使用 CONCURRENT_NUM 环境变量
with ThreadPool() as pool:
pool.submit(crawl, keyword="python", page=1)
ThreadPool 方法
| 方法 | 说明 |
|---|---|
submit(crawl_fn, **kwargs) |
提交任务,worker 全满时阻塞等待 |
wait() |
阻塞等待所有任务完成,返回结果列表 |
close() |
等待所有任务完成后关闭线程池 |
异常处理
爬虫函数抛出的异常会被捕获并通过 output.error() 记录,任务结果记为 None,不中断其他任务。
def failing_crawl(session):
raise ValueError("some error")
pool = ThreadPool(max_workers=2)
pool.submit(failing_crawl)
results = pool.wait() # results[0] is None
pool.close()
代理配置
StealthSession 的代理行为由环境变量控制(通常由 Agent 下发),不支持按请求传入代理参数:
| 环境变量 | 说明 |
|---|---|
PROXY_ENABLED |
on 开启代理,其他值直连 |
PROXY_MODE |
tunnel 静态代理 / private 管理端动态分配 |
PROXY_URL |
tunnel 模式的静态代理 URL |
PROXY_VENDOR |
private 模式的供应商标识 |
PROXY_TARGET_SITE |
private 模式的目标站点标识 |
ADMIN_SERVER_URL |
private 模式下管理端地址 |
AGENT_API_KEY |
private 模式下调用管理端的第三方 Token(td_ 前缀;代理分配/上报接口仅对第三方 Token 开放,节点 ck_ Key 无权调用) |
# tunnel 模式示例
export PROXY_ENABLED=on
export PROXY_MODE=tunnel
export PROXY_URL=http://proxy:8080
private 模式下代理用尽或持续被目标站封禁时,get/post 抛出 ProxyExhaustedError。
运行模式
SDK 自动检测运行模式:
- Agent 模式: 当
LOG_FILE_PATH和DATA_FILE_PATH环境变量存在时,日志和结果写入文件 - 开发模式: 环境变量不存在时,输出到控制台
# 开发模式
python spider.py
# Agent 模式
export LOG_FILE_PATH=/tmp/test.log
export DATA_FILE_PATH=/tmp/test.json
python spider.py
完整示例
#!/usr/bin/env python3
from crawlerkeeper_sdk import params, info, result, StealthSession, ThreadPool
def crawl(session: StealthSession, keyword: str, page: int):
"""爬虫函数"""
info(f"爬取 {keyword} 第 {page} 页")
resp = session.get(f"https://example.com/api?kw={keyword}&page={page}")
return resp.json()
def main():
config = params()
keywords = config.get("keywords", ["python"])
max_workers = config.get("max_workers", 3)
with ThreadPool(max_workers=max_workers) as pool:
# 提交所有任务
for keyword in keywords:
for page in range(1, 4):
pool.submit(crawl, keyword=keyword, page=page)
# 阻塞等待所有任务完成
results = pool.wait()
# 输出结果:聚合后一次性写入(result 每次调用会整体覆写数据文件,
# 循环内逐条调用只会保留最后一条,必须先聚合再单次调用)
collected = [data for data in results if data is not None]
result(collected)
info("爬取完成")
if __name__ == "__main__":
main()
许可证
MIT License
Metadata
Release files for crawlerkeeper-sdk 0.3.1
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| crawlerkeeper_sdk-0.3.1.tar.gz | 33.3 kB | Without provenance | Details for crawlerkeeper_sdk-0.3.1.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| crawlerkeeper_sdk-0.3.1-py3-none-any.whl | Python 3 | none | any | Details for crawlerkeeper_sdk-0.3.1-py3-none-any.whl |
Total release size: 61.9 kB
Release files / crawlerkeeper_sdk-0.3.1.tar.gz
| Download URL | crawlerkeeper_sdk-0.3.1.tar.gz |
|---|---|
| Size | 33.3 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
577becea65b76c961613ae5ee8d0ba4c5cc7030b46fd42daf35dd8812466bdf3
|
|
BLAKE2b-256 checksum How to use checksums |
2f5d3615ae6a4e9e2538396a79c63ad4e6c67825f76e38bd6ec0e4df318eca74
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
uv/0.13.7 {"installer":{"name":"uv","version":"0.13.7","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"macOS","version":null,"id":null,"libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}
|
Release files / crawlerkeeper_sdk-0.3.1-py3-none-any.whl
| Download URL | crawlerkeeper_sdk-0.3.1-py3-none-any.whl |
|---|---|
| Size | 28.6 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
0003458426320316b6026f76ba3eef16a25b562d11812e471592b97fd94fcf6b
|
|
BLAKE2b-256 checksum How to use checksums |
c997dda273beda151239154b09d50ad22ffb7e4b33127a981cf50b04e9adefc5
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
uv/0.13.7 {"installer":{"name":"uv","version":"0.13.7","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"macOS","version":null,"id":null,"libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}
|
Security for crawlerkeeper-sdk 0.3.1
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.