crawlerkit-core 0.2.0

Browserless crawler base: curl_cffi transport, TLS/AIA, identity, proxy, captcha, BaseCrawler/BaseParser.

crawlerkit-core

A standalone, browserless crawler base (crawlerkit.core): fingerprinted curl_cffi transport, per-host TLS with AIA repair + .pfx client certs, browserforge identity (UA snapped to the impersonate target), proxy providers, a pluggable captcha registry, an error taxonomy with retry+rotation, and the BaseCrawler.flow() / BaseParser.parse() hooks. Zero non-PyPI dependencies — parse() returns your own type, not one the library dictates.

Install

pip install crawlerkit-core

Use

from crawlerkit.core import BaseCrawler, BaseParser, RawResponse, Transport, Profile
from crawlerkit.core.captcha import default_registry, McaptchaPowSolver, mcaptcha_hint
from crawlerkit.core.proxy import StaticProxyProvider, BrightDataProxyProvider
from crawlerkit.core.errors import BlockedError, TransientError, raise_for_block

HTTP is curl_cffi only — requests is never used. Deps: curl_cffi, browserforge, cryptography, certifi, selectolax, lxml, beautifulsoup4, weasyprint, structlog, tenacity.

Logging

Logging is opt-in and off by default — crawlerkit emits nothing unless you ask. Set enable_logs = True on your crawler or parser to turn on structlog events:

class MyCrawler(BaseCrawler):
    enable_logs = True   # default is False

Build a crawler: GETTING_STARTED.md. Run the demos: examples/ (quotes.py — a full crawl+parse; fingerprint_demo.py — identity proof). Reference: docs/ (identity, transport-tls, proxy, captcha, cracking-govbr-turnstile, errors, api). License: MIT.