bot-guard-py

bot-guard-py 0.7.0 is a zero-runtime-dependency Python 3.9+ HTTP bot detection and application-layer flood mitigation package. It includes bounded per-client/global token buckets and an in-flight request cap, as well as configurable bot scoring and policy actions.

Quick start

from bot_guard import BotGuard

guard = BotGuard(threshold=0.6)
result = guard.inspect(request.headers, client_ip="203.0.113.10")
if result.action == "BLOCK":
    return Response("Forbidden", status=403)

For direct enforcement:

from bot_guard import BotDetected, BotGuard
try:
    result = BotGuard().enforce(request.headers, client_ip="203.0.113.10")
except BotDetected as blocked:
    return Response("Forbidden", status=403)

Policy controls

Default actions are ALLOW below 0.4, CHALLENGE from 0.4, RATE_LIMIT from 0.5, and BLOCK from the configured bot threshold (0.6 by default). BotResult.explain() gives a readable decision report, while matched_rules gives stable rule identifiers.

Detection layers

The core inspects CLI clients, headless browsers, Puppeteer, Playwright, Selenium, PhantomJS, scrapers, missing browser headers, proxy/tunnel headers, browser-header consistency, IP/CIDR allowlists and denylists, and optional reputation/behavior providers. The opt-in signed browser probe adds client-reported WebDriver and automation-global signals. The core performs no network calls and never stores raw headers.

The included InMemoryReputation is a short-lived, thread-safe process-local score store. For Redis, ASN, VPN, Tor, datacenter, or behavioral systems, implement the tiny provider interface:

class MyProvider:
    def score(self, headers, client_ip=None):
        return 0.0  # normalized 0..1

For short-lived correlation, fingerprint() hashes selected normalized headers and never stores raw values. For production fleets, replace InMemoryRateLimiter with a shared Redis or gateway limiter; the local implementation is useful for single-process deployments and tests.

ChallengeManager creates an expiring HMAC-signed browser challenge. Its rendered page solves a small SHA-256 proof-of-work using Web Crypto; a valid proof is accepted once per manager process, then the application can issue a short-lived signed browser attestation. Multi-process deployments need a shared replay store if they require global one-time use. This proves only that a browser-like client executed JavaScript; it is not hardware-backed attestation and must not be treated as identity proof. BehaviorTracker adds short-lived burst, path-scan, and 404/5xx error-history signals; unknown status 0 is not treated as an error.

Middleware

FastAPI/Starlette blocks actions by default. Supply on_action to implement a real challenge or application-specific rate limit; otherwise CHALLENGE returns 403 and RATE_LIMIT returns 429:

from bot_guard.middleware.fastapi import BotGuardMiddleware
app.add_middleware(BotGuardMiddleware, guard=BotGuard())

JavaScript automation probe

For browser pages you control, add the same-origin script to the shared HTML layout. It solves a short SHA-256 nonce challenge, reports navigator.webdriver, known Selenium/Playwright globals, and a small interaction count, then receives a signed HttpOnly; SameSite=Lax cookie. A positive WebDriver signal is a strong risk signal and is blocked on the next request by default. Missing JavaScript or missing interactions is neutral; the interaction count is not proof that a user is human.

import os
from bot_guard import (BotGuard, BrowserSignalManager, BehaviorTracker,
                       ChallengeManager, DDoSProtector)
from bot_guard.middleware import (BotGuardASGI, BrowserSignalASGI,
                                  DDoSProtectionASGI)

# `verified_crawlers` is built from the refreshed official feeds above.
challenge = ChallengeManager(os.environ["BOT_GUARD_SECRET"],
                             challenge_ttl=60, attestation_ttl=300)
signals = BrowserSignalManager(challenge)
guard = BotGuard(trusted_crawler_verifier=verified_crawlers,
                 browser_signal_provider=signals,
                 behavior_provider=BehaviorTracker())
protector = DDoSProtector(trusted_crawler_verifier=verified_crawlers)

# Wrap your FastAPI/Starlette `app` at its ASGI boundary.
app = BotGuardASGI(
    DDoSProtectionASGI(BrowserSignalASGI(app, signals), protector),
    guard=guard,
)

Add <script src="/__bot_guard/probe.js" defer></script> to the shared page template. For WSGI, use the same order with BotGuardWSGI(DDoSProtectionWSGI(BrowserSignalWSGI(app, signals), protector), guard). The probe runs only where you add it; a first page request can complete before telemetry returns, so still protect sensitive actions with authentication, authorization, CSRF defenses, and rate limits. The server-issued token and cookie are signed and peer-IP-bound, but browser telemetry is still client-reported and can be hidden by stealth tools. Verified search-crawler CIDRs are checked before this signal, so verified crawlers do not need JavaScript. Behind a reverse proxy, configure all layers to use the same trusted client-IP resolver; never trust arbitrary forwarded headers.

HTTP flood mitigation (application layer)

Use the separate rate/concurrency middleware before the application does expensive work:

from bot_guard import DDoSProtector
from bot_guard.middleware.fastapi import DDoSProtectionMiddleware

protector = DDoSProtector(
    per_client_rate=20, per_client_burst=60,
    global_rate=1000, global_burst=2000,
    max_concurrent=500,
)
app.add_middleware(DDoSProtectionMiddleware, protector=protector)

For Flask, wrap the WSGI boundary: from bot_guard.middleware.flask import init_ddos_protection; init_ddos_protection(app, protector). The generic equivalents are DDoSProtectionASGI and DDoSProtectionWSGI; Django provides bot_guard.middleware.django.DDoSProtectionMiddleware. DDoSProtector defaults to 20 requests/second per client (burst 60), 1,000 requests/second globally (burst 2,000), and at most 500 concurrent application requests. Tune these values to the site's real capacity; NAT/shared-IP users can share one per-client bucket. Rate rejections return 429; concurrency shedding returns 503.

Search crawler handling

Do not allow Googlebot, bingbot, DuckDuckBot, or DuckAssistBot from the User-Agent string alone; any client can spoof it. Use VerifiedCrawlerAllowlist.from_json_feeds() with cached, current official feeds, then pass the same verifier to bot detection and flood control:

import json
from pathlib import Path
from bot_guard import BotGuard, DDoSProtector, VerifiedCrawlerAllowlist
from bot_guard.middleware.fastapi import BotGuardMiddleware, DDoSProtectionMiddleware

# Download and validate these files outside the request path; refresh periodically.
# python examples/update_crawler_feeds.py --directory ./crawler-feeds
def read_feed(filename):
    return json.loads(Path(filename).read_text(encoding="utf-8"))

# Download/cache these feeds outside the request path; refresh as vendors update them.
verified_crawlers = VerifiedCrawlerAllowlist.from_json_feeds({
    "googlebot": read_feed("common-crawlers.json"),
    "bingbot": read_feed("bingbot.json"),
    "duckduckbot": read_feed("duckduckbot.json"),
    "duckassistbot": read_feed("duckassistbot.json"),
})
guard = BotGuard(trusted_crawler_verifier=verified_crawlers)
protector = DDoSProtector(trusted_crawler_verifier=verified_crawlers)

# `app` is your FastAPI application. Add flood control last so it is outermost.
app.add_middleware(BotGuardMiddleware, guard=guard)
app.add_middleware(DDoSProtectionMiddleware, protector=protector)

Verified crawlers skip heuristic bot blocking and the per-client rate bucket, but remain under the global rate/concurrency caps. Maintain feeds from Google's official crawler ranges, Bing's verification guidance/IP list, DuckDuckBot's official IP JSON, and DuckAssistBot's official IP JSON. DuckDuckGo publishes each bot's User-Agent and IP list in its DuckDuckBot and DuckAssistBot documentation. Refresh feeds outside the request path and do not ship a stale baked-in list. DuckDuckBot respects robots.txt; DuckDuckGo documents opting out of DuckAssistBot via its robots.txt User-Agent. Do not use broad allow_user_agents=["googlebot"] (or other crawler names) for public traffic.

Flask and Django adapters are available in bot_guard.middleware.flask and bot_guard.middleware.django. For framework portability, use BotGuardASGI or BotGuardWSGI from bot_guard.middleware.generic. All adapters pass the socket peer IP to the detector and preserve the result on the request scope/object where the framework supports it. They do not trust forwarded headers; configure trusted-proxy handling in the application/server separately. Flask retains the on_bot callback name for compatibility and also accepts on_action.

CLI

Inspect a JSON header object from a file or stdin:

echo '{"User-Agent":"curl/8.0"}' | bot-guard inspect -
echo '{"User-Agent":"curl/8.0"}' | bot-guard explain -

The CLI exits with status 1 for a blocking decision and status 0 otherwise, making it useful in gateways and smoke tests.

Operations and security

Use mode="shadow" before enabling bot blocking on unfamiliar traffic. Send event_sink events to your logger or telemetry system without recording raw header values; Metrics.snapshot() exposes counters for export to Prometheus or OpenTelemetry adapters. Configure the reverse proxy to strip user-supplied forwarding headers and pass only trusted client IP values. The built-in token buckets are process-local: they do not coordinate across workers, containers, or regions. Supply shared client_limiter and global_limiter objects implementing consume(key) -> (allowed, retry_after_seconds) for distributed rate limiting, or enforce limits at a gateway/CDN.

This package can shed excess HTTP requests after they reach the application server. It cannot stop network/volumetric DDoS or protect an already saturated link/origin; keep CDN, WAF, hosting-provider DDoS protection, origin firewalling, and upstream rate limits in front. Header signals remain spoofable. Use authentication, quotas, and application authorization for protected resources. See SECURITY.md and CHANGELOG.md.

Development

python -m unittest discover -s tests -v
python -m compileall -q bot_guard
python -m pip wheel . --no-deps
# Optional real GUI Chromium/Selenium flow (requires Xvfb, Selenium, Chromium)
BOT_GUARD_RUN_GUI_TESTS=1 xvfb-run -a python -m pytest -q

CI covers Python 3.9 through 3.13, compilation, tests, and wheel building. MIT licensed.