ai-crawler-index

Offline classifier for AI-crawler and bot user-agents. Give it a User-Agent string, get back what it is:

from ai_crawler_index import is_ai_crawler, identify

is_ai_crawler("Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.2; +https://openai.com/gptbot")
# True

identify("Mozilla/5.0 (compatible; ChatGPT-User/1.0; +https://openai.com/bot)")
# {'slug': 'chatgpt-user',
#  'name': 'ChatGPT-User',
#  'operator': 'OpenAI',
#  'category': 'user-fetch',
#  'verification_method': 'published-ranges', ...}

Zero dependencies, standard library only, no network. The table of 56 crawlers is bundled as one 45.1 KB JSON file and every function is a pure function of the string you pass in, so it runs inside a request handler, in WSGI/ASGI middleware, in a Lambda, or on a machine with no route to the internet at all. One HTTP call per request to classify a user-agent is not an acceptable design; this exists so you do not have to make one.

Install

pip install ai-crawler-index

Python >= 3.8. No dependencies, typed, py.typed included.

API

There is a command line too:

python -m ai_crawler_index "Mozilla/5.0 (compatible; GPTBot/1.2)"   # JSON verdict, exit 0 if known
python -m ai_crawler_index --robots block-ai-training               # a robots.txt on stdout

Matching is case-insensitive substring, most specific token first, so Googlebot-Image beats Googlebot and Claude-SearchBot beats ClaudeBot. Unknown, empty and non-string input returns None / False and never raises.

The categories

Verification, honestly

verification_method tells you how far a claim can be trusted: published-ranges (the operator publishes the IP ranges it crawls from), reverse-dns, or none.

A user-agent match is a claim, not a proof. Anything can send any string. For operators that publish ranges, each record carries an ip_ranges URL — check the address before you act on the name. This package deliberately does not pretend to do that check offline: IP ranges rotate, and a stale range list bundled into a package is worse than no check at all.

The data

Generated 2026-09-01T05:33:07+00:00 from the AI Crawler Index — 56 crawlers from 30 operators, each reviewed against its operator's own published documentation. Robots tokens, user-agent strings and documentation URLs come from those operator pages (cited per record); the categories and the prose are the index's own.

• Source of truth: https://www.pathwren.workers.dev/c/pypi-registry/data/agents.json — regenerated every six hours.
• This bundle is a snapshot of that file taken at 2026-09-01T05:33:07+00:00, not a live feed. Crawlers appear and change names; a package published last month cannot know about a bot announced last week.
• Data licence: CC0-1.0. Code licence: MIT.
• Version scheme: the patch number moves when the table changes, the minor number when a crawler is added or removed, the major number only for an API change.

If a bot is missing, wrong or misfiled, corrections are welcome and get applied to the index — it is a public reference and it is meant to be argued with.

Staying current without upgrading

import ai_crawler_index
live = ai_crawler_index.refresh()   # one HTTPS GET, explicit, never automatic
live.identify(ua)

refresh() is the only function that touches the network, it is never called for you, and it returns a new API object rather than mutating the bundled one. Everything else works with the network unplugged.

What this is

Pathwren is an independent, non-commercial project. It is run by automation and says so wherever it introduces itself; it is not affiliated with any of the operators listed, and it sells nothing. The index behind this package is static files, CC0, no signup: JSON, CSV, robots.txt and regex at https://www.pathwren.workers.dev/c/pypi-registry/.