ai-crawler-index

Offline classifier for AI-crawler and bot user-agents. Give it a User-Agent string, get back what it is:

from ai_crawler_index import is_ai_crawler, identify

is_ai_crawler("Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.2; +https://openai.com/gptbot")
# True

identify("Mozilla/5.0 (compatible; ChatGPT-User/1.0; +https://openai.com/bot)")
# {'slug': 'chatgpt-user',
#  'name': 'ChatGPT-User',
#  'operator': 'OpenAI',
#  'category': 'user-fetch',
#  'verification_method': 'published-ranges', ...}

Zero dependencies, standard library only, no network. The table of 150 crawlers is bundled as one 123.6 KB JSON file and every function is a pure function of the string you pass in, so it runs inside a request handler, in WSGI/ASGI middleware, in a Lambda, or on a machine with no route to the internet at all. One HTTP call per request to classify a user-agent is not an acceptable design; this exists so you do not have to make one.

Install

pip install ai-crawler-index

Python >= 3.8. No dependencies, typed, py.typed included.

API

There is a command line too — installed as ai-crawler-index, and also runnable as a module if you would rather not put a script on your PATH:

ai-crawler-index "Mozilla/5.0 (compatible; GPTBot/1.2)"             # JSON verdict, exit 0 if known
ai-crawler-index --robots block-ai-training                         # a robots.txt on stdout
python -m ai_crawler_index "Mozilla/5.0 (compatible; GPTBot/1.2)"   # the same thing, no entry point

Matching is case-insensitive substring, most specific token first, so Googlebot-Image beats Googlebot and Claude-SearchBot beats ClaudeBot. Unknown, empty and non-string input returns None / False and never raises.

A user-agent is a claim, not evidence

This package tells you what a client says it is. Whether the claim is true is a question about its IP address, and it has a separate answer: ai-crawler-verify checks the address against the operator's own published ranges or forward-confirmed reverse DNS, and says None rather than guessing when the operator publishes no check at all.

pip install ai-crawler-verify
ai-crawler-verify 20.171.206.10 --ua "GPTBot/1.2"

The categories

Verification, honestly

verification_method tells you how far a claim can be trusted: published-ranges (the operator publishes the IP ranges it crawls from), reverse-dns, or none.

A user-agent match is a claim, not a proof. Anything can send any string. For operators that publish ranges, each record carries an ip_ranges URL — check the address before you act on the name. This package deliberately does not pretend to do that check offline: IP ranges rotate, and a stale range list bundled into a package is worse than no check at all.

The data

Generated 2026-09-02T14:41:10+00:00 from the AI Crawler Index — 150 crawlers from 74 operators, each reviewed against its operator's own published documentation. Robots tokens, user-agent strings and documentation URLs come from those operator pages (cited per record); the categories and the prose are the index's own.

• Source of truth: https://www.pathwren.workers.dev/c/pypi-registry/data/agents.json — regenerated every six hours.
• This bundle is a snapshot of that file taken at 2026-09-02T14:41:10+00:00, not a live feed. Crawlers appear and change names; a package published last month cannot know about a bot announced last week.
• Data licence: CC0-1.0. Code licence: MIT.
• Version scheme: the patch number moves when the table changes, the minor number when a crawler is added or removed, the major number only for an API change.

If a bot is missing, wrong or misfiled, corrections are welcome and get applied to the index — it is a public reference and it is meant to be argued with.

Staying current without upgrading

import ai_crawler_index
live = ai_crawler_index.refresh()   # one HTTPS GET, explicit, never automatic
live.identify(ua)

refresh() is the only function that touches the network, it is never called for you, and it returns a new API object rather than mutating the bundled one. Everything else works with the network unplugged.

What this is

Pathwren is an independent, non-commercial project. It is run by automation and says so wherever it introduces itself; it is not affiliated with any of the operators listed, and it sells nothing. The index behind this package is static files, CC0, no signup: JSON, CSV, robots.txt and regex at https://www.pathwren.workers.dev/c/pypi-registry/.