anti-bot-detector

Framework-agnostic anti-bot detection for web crawlers — HTTP header fingerprinting, JS challenge identification, and proxy auto-rotation in one dependency-free core. Derived from crawl4ai's Anti-Bot Detection subsystem (Apache-2.0); zero crawl4ai imports, zero runtime dependencies.

Why

Every crawler eventually hits a WAF: Cloudflare challenges, DataDome blocks, rate-limit pages that look like real content. anti-bot-detector classifies responses before your parser wastes time on them, and tells you what to do next — proceed, rotate proxy, solve challenge, or back off.

• Zero dependencies — the detection core is pure standard library
• Framework-agnostic — adapters for httpx, scrapy, playwright and crawl4ai (all optional extras, lazily imported)
• 3-layer detection — header fingerprints → JS challenge bodies → proxy auto-upgrade state machine
• Apache-2.0 compliant — derivation scope documented in DESIGN.md, new code MIT-licensed

Installation

pip install anti-bot-detector

# with framework adapters (optional extras):
pip install anti-bot-detector[httpx]      # httpx integration
pip install anti-bot-detector[scrapy]     # scrapy downloader middleware
pip install anti-bot-detector[playwright] # playwright integration
pip install anti-bot-detector[dev]        # pytest + build + twine

Requires Python ≥ 3.9. Installing the base package does not install crawl4ai, httpx, scrapy or playwright.

Quick start

Classify a response you already have (no network needed):

from anti_bot_detector import AntiBotEngine, FetchResponse

engine = AntiBotEngine()
result = engine.detect_response(
    FetchResponse(
        url="https://example.com",
        status_code=503,
        headers={"cf-mitigated": "challenge", "server": "cloudflare"},
        body="<html>Just a moment...</html>",
    )
)
print(result.summary())
# https://example.com -> level=blocked challenge=cloudflare
#   confidence=0.90 action=rotate_proxy

Fetch + detect + auto-upgrade in one call:

from anti_bot_detector import AntiBotEngine

def fetch(url: str, headers: dict):
    import httpx  # any HTTP client works
    resp = httpx.get(url, headers=headers, timeout=10, follow_redirects=True)
    from anti_bot_detector import FetchResponse
    return FetchResponse(
        url=str(resp.url),
        status_code=resp.status_code,
        headers=dict(resp.headers),
        body=resp.text,
    )

engine = AntiBotEngine(
    fetcher=fetch,
    proxies=["http://proxy1:8080", "http://proxy2:8080"],
    max_retries=3,
)
result = engine.detect("https://target.example.com")
if result.recommended_action.value == "proceed":
    print("safe to crawl", result.url)

API reference

Core (anti_bot_detector)

Key methods:

engine.detect_response(response: FetchResponse) -> DetectionResult  # layers 1+2
engine.detect(url: str) -> DetectionResult                          # fetch + layers 1-3
engine.fetch(url: str) -> DetectionResult                           # with auto-upgrade

rotator.current()                                  # active proxy or None (exhausted)
rotator.rotate(reason="soft-block")                # upgrade to next healthy proxy
rotator.mark_failed(proxy, reason="timeout")       # cooldown + rotate away

Adapters (anti_bot_detector.adapters)

Adapters are lazily imported — importing the package never pulls a framework.

Integration examples

httpx client

import httpx
from anti_bot_detector import BotLevel
from anti_bot_detector.adapters import HttpxAdapter

def detect_url(url: str):
    with httpx.Client() as client:
        response = client.get(url, follow_redirects=True)
    verdict = HttpxAdapter().detect(response)
    if verdict.bot_level is BotLevel.BLOCKED:
        raise RuntimeError(f"WAF block: {verdict.summary()}")
    return verdict

scrapy downloader middleware

# settings.py
DOWNLOADER_MIDDLEWARES = {
    "anti_bot_detector.adapters.scrapy_middleware.AntiBotMiddleware": 560,
}
ANTIBOT_DROP_CHALLENGES = True  # also drop "challenge" pages, not just blocks

Blocked responses raise before they reach your spider callback, so parsers never see WAF error pages.

More runnable scripts in examples/: standalone_detect.py (pure Python, stdlib only) and httpx_integration.py (httpx client integration).

License

New code: MIT (see LICENSE). Derived from crawl4ai's Anti-Bot Detection subsystem, Apache-2.0 — see NOTICE and LICENSES/ for the upstream license and the derivation scope.