4 projects
ai-crawler-logs
Access-log triage for AI crawlers: identify every line, attribute it to its operator, and generate a robots.txt or WAF rule for the traffic you actually received. 56 crawlers bundled, six log formats auto-detected, zero dependencies.
ai-crawler-robots
Lint a robots.txt and audit it against 56 AI crawlers: which ones it really blocks, which rules do nothing, and which fetchers robots.txt cannot stop at all. RFC 9309, zero dependencies, no network.
ai-crawler-index
Offline AI-crawler user-agent classifier: is_ai_crawler(ua) and identify(ua) -> {name, operator, category, verification_method}. 56 crawlers bundled, zero dependencies, no network.
ai-crawler-verify
Verify a claimed AI-crawler identity: check the client IP against the operator's published ranges or forward-confirmed reverse DNS. 12 operator lists, 2949 prefixes bundled, zero dependencies.
