is-crawler 1.1.0

Tiny, zero-dependency crawler detection via regex.

is-crawler

Tiny, zero-dependency Python library that detects bots and crawlers from user-agent strings. Fast, lightweight, and ready to drop into any web app or API.

Docs & live demo: is-crawler.tn3w.dev

Install

pip install is-crawler

For faster regex matching, optionally install google-re2. It will be used automatically when available:

pip install is-crawler google-re2

Usage

from is_crawler import is_crawler, crawler_info, crawler_has_tag

ua = "Googlebot/2.1 (+http://www.google.com/bot.html)"

is_crawler(ua)                              # True
crawler_has_tag(ua, "search-engine")        # True
crawler_has_tag(ua, ["ai-crawler", "seo"])  # False

info = crawler_info(ua)
# CrawlerInfo(url='http://www.google.com/bot.html',
#             description="Google's main web crawling bot for search indexing",
#             tags=['search-engine'])
info.url          # 'http://www.google.com/bot.html'
info.description  # "Google's main web crawling bot for search indexing"
info.tags         # ['search-engine']

The module is also callable directly, no named import required:

import is_crawler
is_crawler("Googlebot/2.1 (+http://www.google.com/bot.html)")  # True

API reference

Function	Returns	Description
is_crawler(ua)	bool	Heuristic detection: fast, no DB
crawler_signals(ua)	list[str]	Which heuristic rules matched
crawler_info(ua)	CrawlerInfo | None	url, description, tags: DB lookup for 646 known crawlers
crawler_has_tag(ua, tag)	bool	tag can be str or list[str] (matches any)
crawler_name(ua)	str | None	Product name extracted from the UA string
crawler_version(ua)	str | None	Version extracted from the UA string
crawler_url(ua)	str | None	URL embedded in the UA string

crawler_signals returns a subset of: bot_signal, no_browser_signature, bare_compatible, known_tool, url_in_ua.

crawler_info tags: search-engine, ai-crawler, seo, social-preview, advertising, archiver, feed-reader, monitoring, scanner, academic, http-library, browser-automation.

Middleware example

from is_crawler import is_crawler, crawler_has_tag

@app.before_request
def gate():
    ua = request.headers.get("User-Agent", "")
    if crawler_has_tag(ua, "ai-crawler"):
        abort(403)        # block AI scrapers
    if is_crawler(ua):
        log_crawler(ua)   # track other bots without blocking

How it works

is_crawler / crawler_signals: five heuristic regex checks, no lookup:

1. Bot signals: common keywords (bot, crawl, spider, scrape, ...), URL/email patterns, headless
2. Missing browser signature: real browsers always include engine tokens like WebKit, Gecko, or Trident
3. Bare (compatible; ...) block: classic bot pattern without OS tokens
4. Known tools: playwright, selenium, wget, lighthouse, sqlmap, and more
5. URL in UA: an embedded http:// or https:// URL, a near-universal bot convention

crawler_info / crawler_has_tag: pattern database loaded lazily on first call, built from monperrus/crawler-user-agents with supplemental tags. Returns url, description, and tags for 646 known crawlers.

Formatting

pip install black isort
isort . && black .
npx prtfm

License

Apache-2.0