is-crawler 1.5.1

Fast, regex-free crawler detection from user agents. Zero deps, ReDoS-safe heuristics, ~100× faster than alternatives.

is-crawler

Fast, regex-free crawler detection from user agents. Zero deps, ReDoS-safe heuristics, ~100× faster than alternatives. Includes FCrDNS IP verification for 100+ known crawlers.

Why regex-free?

Regex is a frequent source of ReDoS vulnerabilities, one un-anchored .* or nested quantifier against a hostile UA can spike CPU to seconds. Crawler detection runs on every request, so a catastrophic pattern is a denial-of-service primitive. is-crawler implements all heuristics with str.find + char scans. No regex engine, no backtracking, no ReDoS surface. crawler_info uses re only to match against curated DB patterns (monperrus/crawler-user-agents) which are simple literals (e.g. Googlebot\/, bingbot, AdsBot-Google([^-]|$), [wW]get), no nested quantifiers, no catastrophic backtracking paths.

Install

pip install is-crawler

Usage

from is_crawler import (
    is_crawler, crawler_signals, crawler_info, crawler_has_tag,
    crawler_name, crawler_version, crawler_url, CrawlerInfo,
)
from is_crawler.ip import (
    verify_crawler_ip,
    reverse_dns,
    forward_confirmed_rdns,
    ip_in_range,
    known_crawler_ip,
    known_crawler_rdns,
)

ua = "Googlebot/2.1 (+http://www.google.com/bot.html)"
ip = "66.249.66.1"

is_crawler(ua)                                  # True
crawler_signals(ua)                             # ['bot_signal', 'no_browser_signature', 'url_in_ua']
crawler_name(ua)                                # 'Googlebot'
crawler_version(ua)                             # '2.1'
crawler_url(ua)                                 # 'http://www.google.com/bot.html'
verify_crawler_ip(ua, ip)                       # True - FCrDNS validation
reverse_dns(ip)                                 # 'crawl-66-249-66-1.googlebot.com'
forward_confirmed_rdns(ip, (".googlebot.com",)) # hostname or None
ip_in_range(ip)                                 # True - in known crawler CIDRs
known_crawler_ip(ip)                            # alias for ip_in_range
known_crawler_rdns(ip)                          # True - known crawler via FCrDNS/rDNS

info = crawler_info(ua)                         # CrawlerInfo(...)
if info is not None:
    info.url                                    # 'http://www.google.com/bot.html'
    info.description                            # "Google's main web crawling bot..."
    info.tags                                   # ('search-engine',)

crawler_has_tag(ua, "search-engine")            # True
crawler_has_tag(ua, ["ai-crawler", "seo"])      # False

API

is_crawler(ua: str) -> bool

Heuristic detection. Returns True if the UA is a crawler. No DB lookup, no regex.

Three short-circuit rules:

1. Positive signal: bot keywords (bot, crawl, spider, scrape, headless, slurp, archiv, preview, ...), known tools (playwright, selenium, wget, lighthouse, sqlmap, nikto, nmap, httrack, pingdom, google-safety, ...), or a URL/email embedded in the UA.
2. No browser signature: missing Mozilla/, WebKit, Gecko, Trident, Presto, KHTML, Links, Lynx, Opera, or an OS token like (Windows, (Linux, (X11, (Macintosh.
3. Bare (compatible; ...): classic bot block without OS/browser tokens inside.

crawler_signals(ua: str) -> list[str]

Which individual rules fired. Subset of: bot_signal, no_browser_signature, bare_compatible, known_tool, url_in_ua. Useful for diagnostics and logging. is_crawler does not call this.

crawler_name(ua: str) -> str | None

Product name extracted from the UA.

• Googlebot/2.1 ... → 'Googlebot'
• Mozilla/5.0 (compatible; bingbot/2.0; ...) → 'bingbot'
• Mozilla/5.0 ... Speedy Spider (...) → 'Speedy Spider'
• Chrome/Firefox/Safari → None

crawler_version(ua: str) -> str | None

Version token extracted from the UA. Returns None if no non-browser version is detectable.

• curl/7.64.1 → '7.64.1'
• Mozilla/5.0 (compatible; Miniflux/2.0.10; ...) → '2.0.10'
• Googlebot/2.1 ... → '2.1'

crawler_url(ua: str) -> str | None

URL embedded in the UA (after +, ;, or -).

• Googlebot/2.1 (+http://www.google.com/bot.html) → 'http://www.google.com/bot.html'
• UA with no embedded URL → None

crawler_info(ua: str) -> CrawlerInfo | None

DB lookup against 1200 known crawler patterns. Returns None for browsers (short-circuits via is_crawler).

class CrawlerInfo(NamedTuple):
    url: str                # crawler's info/docs URL (may be '')
    description: str        # human-readable description
    tags: tuple[str, ...]   # classification tags, e.g. ('search-engine',)

crawler_has_tag(ua: str, tags: str | Iterable[str]) -> bool

True if the crawler has any of the given tags. tags accepts a single string or a list.

Available tags: search-engine, ai-crawler, seo, social-preview, advertising, archiver, feed-reader, monitoring, scanner, academic, http-library, browser-automation.

Category shortcuts

One-tag wrappers over crawler_has_tag:

is_search_engine(ua)       # 'search-engine'
is_ai_crawler(ua)          # 'ai-crawler'
is_seo(ua)                 # 'seo'
is_social_preview(ua)      # 'social-preview'
is_advertising(ua)         # 'advertising'
is_archiver(ua)            # 'archiver'
is_feed_reader(ua)         # 'feed-reader'
is_monitoring(ua)          # 'monitoring'
is_scanner(ua)             # 'scanner'
is_academic(ua)            # 'academic'
is_http_library(ua)        # 'http-library'
is_browser_automation(ua)  # 'browser-automation'

is_good_crawler(ua) / is_bad_crawler(ua)

Opinionated groupings for quick allow/deny gates.

• Good (indexing, previews, archives, feeds, research): search-engine, social-preview, feed-reader, archiver, academic.
• Bad (scraping, scanning, unattributed traffic): ai-crawler, scanner, http-library, browser-automation, seo.

advertising and monitoring are intentionally neither: policy-dependent.

Middleware

from is_crawler.contrib import WSGICrawlerMiddleware

app = WSGICrawlerMiddleware(app)

# Flask
request.environ["is_crawler"].is_crawler

# Django
request.META["is_crawler"].name

from is_crawler.contrib import ASGICrawlerMiddleware

app = ASGICrawlerMiddleware(app, block=True, block_tags="ai-crawler")

# FastAPI / Starlette
request.scope["is_crawler"].is_crawler
request.state.crawler.verified

Both middlewares are zero-dep. They attach CrawlerMiddlewareResult with user_agent, ip, is_crawler, name, and verified.

• WSGICrawlerMiddleware: Flask, Django, any WSGI app
• ASGICrawlerMiddleware: FastAPI, Starlette, any ASGI app

Optional flags: block=True, block_tags=..., verify_ip=True, trust_forwarded=True.

With trust_forwarded=True, middleware uses the first IP from X-Forwarded-For, then X-Real-IP, before the direct client address.

robots.txt helpers

Generate directives from DB tags. Names extracted from DB patterns (slash/URL-only entries skipped).

from is_crawler import build_robots_txt, robots_agents_for_tags, iter_crawlers

robots_agents_for_tags("ai-crawler")
# ['AI2Bot', 'Applebot-Extended', 'Bytespider', 'CCBot', 'ChatGPT-User', 'Claude-Web', 'GPTBot', ...]

print(build_robots_txt(disallow=["ai-crawler", "scanner"]))
# User-agent: GPTBot
# Disallow: /
#
# User-agent: Nikto
# Disallow: /
# ...

build_robots_txt(allow="search-engine", path="/public")
# User-agent: Googlebot
# Allow: /public
# ...

for info, name in iter_crawlers():      # (CrawlerInfo, robots-name) per DB entry
    ...

IP verification (is_crawler.ip)

Two complementary strategies — use either or both.

FCrDNS (forward-confirmed reverse DNS)

rDNS → suffix check → forward lookup → IP match. Catches UA spoofing. socket only, no deps.

from is_crawler.ip import verify_crawler_ip, forward_confirmed_rdns, reverse_dns

verify_crawler_ip("Googlebot/2.1 (+http://www.google.com/bot.html)", "66.249.66.1")
# True → rDNS ends in .googlebot.com AND forward lookup returns same IP

verify_crawler_ip("Googlebot/2.1", "8.8.8.8")               # False (spoof)
reverse_dns("8.8.8.8")                                       # 'dns.google'
forward_confirmed_rdns("66.249.66.1", (".googlebot.com",))   # hostname or None

Built-in suffixes: Googlebot, Bingbot, Applebot, DuckDuckBot, YandexBot, Baiduspider, FacebookBot, and 80+ more. Crawler name taken from crawler_name(ua).

IP range lookup

Check whether an IP belongs to any known crawler's published CIDR range. Requires building the range database first (see Tools below).

from is_crawler.ip import ip_in_range, known_crawler_ip, known_crawler_rdns

ip_in_range("66.249.66.1")    # True  — in Googlebot ranges
ip_in_range("8.8.8.8")        # False — not a known crawler range
known_crawler_ip("66.249.66.1")  # alias for ip_in_range
known_crawler_rdns("66.249.66.1")  # True — reverse DNS matches a known crawler domain

Results are LRU-cached. The file is optional — if absent, ip_in_range returns False rather than raising.

Tools

Scripts in tools/ build the data files shipped inside the package.

build_user_agents.py

Compiles is_crawler/crawler-user-agents.json from the upstream monperrus/crawler-user-agents source plus local extras.

python3 tools/build_user_agents.py
python3 tools/build_user_agents.py --input crawler-user-agents.json --output is_crawler/crawler-user-agents.json

build_ip_ranges.py

Fetches live IP range data from 39 official sources (Google, Bing, DuckDuckGo, Apple, OpenAI, Anthropic, Perplexity, Common Crawl, Cloudflare, Fastly, AWS, Azure, Oracle Cloud, GitHub, Telegram, Ahrefs, Yandex, Facebook, Kagi, Amazon, UptimeRobot, Pingdom, Stripe, and more) and writes a flat is_crawler/crawler-ip-ranges.json mapping each source name to its CIDR list.

python3 tools/build_ip_ranges.py
python3 tools/build_ip_ranges.py --timeout 30 --skip-errors

Source definitions live in tools/crawler-ip-ranges.json (name → {url, pattern}) and can be extended independently of the build script.

CLI

python -m is_crawler "Googlebot/2.1 (+http://www.google.com/bot.html)"
tail -f access.log | awk -F'"' '{print $6}' | python -m is_crawler

One JSON object per UA (arg or stdin line) with is_crawler, name, version, url, signals, info.

Caching

Every public function has a 32k-entry LRU cache. Repeat UAs hit in ~40 ns.

Benchmarks

Python 3.14, Linux x86_64. Corpus: 1,231 crawler UAs, 15,812 browser UAs. cua = crawler-user-agents v1.44.

Hot-path (warm cache)

Function	is_crawler	cua	speedup
is_crawler (mixed)	0.05 µs	158.9 µs	3000×
crawler_info	0.60 µs	732.0 µs	1220×
crawler_signals	1.13 µs	-	-
crawler_name	0.33 µs	-	-
crawler_version	0.32 µs	-	-
crawler_url	0.09 µs	-	-
crawler_has_tag	0.10 µs	-	-

Cold-cache (per-call, no LRU hits)

Function	Test Case	is_crawler	cua	speedup
is_crawler	crawlers	1.94 µs	64.35 µs	33×
is_crawler	browsers	1.85 µs	183.76 µs	99×
is_crawler	mixed	1.85 µs	176.94 µs	96×
crawler_info	-	2.07 µs	733.4 µs	354×
crawler_name	-	1.36 µs	-	-
crawler_version	-	1.37 µs	-	-
crawler_url	-	0.29 µs	-	-

Cold-start

Module	Cold-start
is_crawler	1.29 ms
crawleruseragents	0.80 ms

DB patterns compile lazily per 48-entry chunk on first match.

Formatting

pip install black isort
isort . && black .
npx prtfm

Contributing

Bug reports, feature requests, and pull requests are welcome. See CONTRIBUTING.md to get started.

Security

Report vulnerabilities via GitHub private security advisory, do not open a public issue. See SECURITY.md.

Code of Conduct

See CODE_OF_CONDUCT.md.

License

Apache-2.0