Skip to main content

CrawlerKeeper SDK for Python

CrawlerKeeper Python SDK 提供了爬虫与 Agent 通信的封装,支持日志记录、结果写入、参数获取、代理管理和并发爬取。

安装

pip install crawlerkeeper-sdk

快速开始

from crawlerkeeper_sdk import params, info, result, StealthSession

# 获取任务参数
config = params()
info(f"收到参数: {config}")

# 使用 StealthSession 发送请求
session = StealthSession()
resp = session.get("https://example.com/api/data")
data = resp.json()

# 写入结果
result({"data": data, "source": "example.com"})

info("爬取完成")
session.close()

核心功能

日志记录

from crawlerkeeper_sdk import info, warning, error

info("信息日志")
warning("警告日志")
error("错误日志")

结果写入

from crawlerkeeper_sdk import result

result({"count": 100, "items": [...]})

任务参数

from crawlerkeeper_sdk import params

# python spider.py --params '{"keyword": "python", "page": 1}'
config = params()  # {"keyword": "python", "page": 1}

StealthSession

StealthSession 提供隐式浏览器指纹的 HTTP 请求能力,自动管理 TLS 指纹和代理。

from crawlerkeeper_sdk import StealthSession, result

session = StealthSession()

# GET 请求
resp = session.get("https://example.com/api/data")
print(resp.status_code)
print(resp.json())

# POST 请求
resp = session.post("https://example.com/api/submit", json={"key": "value"})

session.close()

主要方法

方法 说明
get(url, **kwargs) 发送 GET 请求
post(url, **kwargs) 发送 POST 请求
close() 关闭会话

ThreadPool 并发爬取

ThreadPool 提供并发爬取能力,内部管理 StealthSession 的创建和关闭。

推荐用法:使用上下文管理器

from crawlerkeeper_sdk import ThreadPool, StealthSession, result

def crawl(session: StealthSession, keyword: str, page: int):
    """爬虫函数:session 会自动传入"""
    resp = session.get(f"https://example.com/search?kw={keyword}&page={page}")
    return resp.json()

# 使用 with 语句自动管理生命周期,不会遗漏 close()
with ThreadPool(max_workers=5) as pool:
    pool.submit(crawl, keyword="python", page=1)
    pool.submit(crawl, keyword="java", page=1)
    pool.submit(crawl, keyword="go", page=1)

    # 阻塞等待所有任务完成
    results = pool.wait()

    # 处理结果
    for data in results:
        if data is not None:
            result(data)
# with 块结束时自动调用 wait() 和 close()

传统用法

pool = ThreadPool(max_workers=5)
pool.submit(crawl, keyword="python", page=1)
pool.submit(crawl, keyword="java", page=1)

results = pool.wait()
pool.close()  # 需要手动调用

环境变量配置并发数

export CONCURRENT_NUM=10
# 不指定 max_workers 时使用 CONCURRENT_NUM 环境变量
with ThreadPool() as pool:
    pool.submit(crawl, keyword="python", page=1)

ThreadPool 方法

方法 说明
submit(crawl_fn, **kwargs) 提交任务,worker 全满时阻塞等待
wait() 阻塞等待所有任务完成,返回结果列表
close() 等待所有任务完成后关闭线程池

异常处理

爬虫函数抛出的异常会被捕获并通过 output.error() 记录,任务结果记为 None,不中断其他任务。

def failing_crawl(session):
    raise ValueError("some error")

pool = ThreadPool(max_workers=2)
pool.submit(failing_crawl)
results = pool.wait()  # results[0] is None
pool.close()

代理配置

StealthSession 的代理行为由环境变量控制(通常由 Agent 下发),不支持按请求传入代理参数:

环境变量 说明
PROXY_ENABLED on 开启代理,其他值直连
PROXY_MODE tunnel 静态代理 / private 管理端动态分配
PROXY_URL tunnel 模式的静态代理 URL
PROXY_VENDOR private 模式的供应商标识
PROXY_TARGET_SITE private 模式的目标站点标识
ADMIN_SERVER_URL private 模式下管理端地址
AGENT_API_KEY private 模式下调用管理端的第三方 Token(td_ 前缀;代理分配/上报接口仅对第三方 Token 开放,节点 ck_ Key 无权调用)
# tunnel 模式示例
export PROXY_ENABLED=on
export PROXY_MODE=tunnel
export PROXY_URL=http://proxy:8080

private 模式下代理用尽或持续被目标站封禁时,get/post 抛出 ProxyExhaustedError。

运行模式

SDK 自动检测运行模式:

  • Agent 模式: 当 LOG_FILE_PATH 和 DATA_FILE_PATH 环境变量存在时,日志和结果写入文件
  • 开发模式: 环境变量不存在时,输出到控制台
# 开发模式
python spider.py

# Agent 模式
export LOG_FILE_PATH=/tmp/test.log
export DATA_FILE_PATH=/tmp/test.json
python spider.py

完整示例

#!/usr/bin/env python3
from crawlerkeeper_sdk import params, info, result, StealthSession, ThreadPool

def crawl(session: StealthSession, keyword: str, page: int):
    """爬虫函数"""
    info(f"爬取 {keyword} 第 {page} 页")
    resp = session.get(f"https://example.com/api?kw={keyword}&page={page}")
    return resp.json()

def main():
    config = params()
    keywords = config.get("keywords", ["python"])
    max_workers = config.get("max_workers", 3)

    with ThreadPool(max_workers=max_workers) as pool:
        # 提交所有任务
        for keyword in keywords:
            for page in range(1, 4):
                pool.submit(crawl, keyword=keyword, page=page)

        # 阻塞等待所有任务完成
        results = pool.wait()

        # 输出结果
        for data in results:
            if data is not None:
                result(data)

    info("爬取完成")

if __name__ == "__main__":
    main()

许可证

MIT License

Metadata

Release files for crawlerkeeper-sdk 0.2.2

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for crawlerkeeper-sdk 0.2.2
File Size Uploaded Provenance Actions
crawlerkeeper_sdk-0.2.2.tar.gz 12.9 kB Without provenance Details for crawlerkeeper_sdk-0.2.2.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for crawlerkeeper-sdk 0.2.2
File Interpreter ABI Platform Actions
crawlerkeeper_sdk-0.2.2-py3-none-any.whl Python 3 none any Details for crawlerkeeper_sdk-0.2.2-py3-none-any.whl

Total release size: 29.4 kB

Release files / crawlerkeeper_sdk-0.2.2.tar.gz

Download URL crawlerkeeper_sdk-0.2.2.tar.gz
Size 12.9 kB
Tags Source
SHA-256 checksum
How to use checksums
1179a63e34cf949709e856b2c3fd6fed0effa8449f955bae52f96386b6633659
BLAKE2b-256 checksum
How to use checksums
45f0e883566bfb8fafeea617ad0d9d7921f7fbdfd042d35b2d367bb7bcfaf295
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via uv/0.13.6 {"installer":{"name":"uv","version":"0.13.6","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"macOS","version":null,"id":null,"libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}

Release files / crawlerkeeper_sdk-0.2.2-py3-none-any.whl

Download URL crawlerkeeper_sdk-0.2.2-py3-none-any.whl
Size 16.5 kB
Tags Python 3
SHA-256 checksum
How to use checksums
adb21ec1d86a32a93385e1bd152ac06b41282ca9fd627c8c0c7e5ae473bb5bdd
BLAKE2b-256 checksum
How to use checksums
434492ef9ce3d0a052e6f2bb2035cc1adaf49dec9e0e593f380ae29f7a02ec30
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via uv/0.13.6 {"installer":{"name":"uv","version":"0.13.6","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"macOS","version":null,"id":null,"libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}

Security for crawlerkeeper-sdk 0.2.2

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page