aiagentallowlist

A small, dependency-light Python client for the AI agent allow list API: a database of verified page-type URLs across 40 million+ domains (99.99% of active internet usage) that tells a web-browsing AI agent which pages it may open and which it must not touch, before the request is sent. Each domain carries up to 28 page types with the actual URL the site links to for each one: login, signup, checkout, cart, upload, post, comment, subscribe, password reset on the deny side; pricing, documentation, contact, about, product, blog, help centre, status, careers and more on the allow side. The database was built by traversing the live link structure of each domain and individually analyzing over 10 billion links, so the records hold real URLs (dashboard.stripe.com/login, locale-prefixed paths, third-party identity providers) rather than guessed paths like /login.

The API evaluates one question per request: may the agent open this exact URL with this method? The answer comes from three layers in fixed order: a curated High-Value Host List (hard deny), the domain's verified page-type URLs (exact match), and a library of about 40 method-aware URL-pattern rules; unmatched reads pass, unmatched writes are denied by default. One request, one verdict, one lookup charged.

Installation

pip install aiagentallowlist

The only runtime dependency is requests. Python 3.7 and newer are supported.

Quick start

from aiagentallowlist import AIAgentAllowlistClient

client = AIAgentAllowlistClient("YOUR_API_KEY")

# May my agent open this exact URL?
v = client.check("https://stripe.com/login")
print(v.verdict)          # "deny"
print(v.matched_layer)    # "page_type_db"
print(v.matched_id)       # "login"

# A bare domain returns the verdict at the domain root plus the full page-type map
rec = client.check("stripe.com")
print(rec.page_types["pricing"])   # "https://stripe.com/pricing"
print(rec.page_types.get("upload"))

# The pages an agent should never open on this domain, as verified URLs
print(client.deny_list("huggingface.co"))

An API key is issued the moment a subscription is activated and is shown in the account area. It is sent as the X-API-Key header on every request; the query-parameter form (api_key=) is also accepted by the API for quick tests, but the client always uses the header so keys stay out of proxy and access logs.

Technical overview

One endpoint

GET https://www.aiagentallowlist.com/api/check?url=<full URL or bare domain>[&method=GET]

• url (required): a full URL or a bare domain. Nothing is stripped. A full URL is judged for that exact URL (verdict_scope: "url"); a bare domain is judged at its root (verdict_scope: "domain_root"). Both return the domain's full page-type map. Subdomains fall back to their base domain's record (chat.openai.com resolves to openai.com).
• method (optional, default GET): the HTTP method the agent intends to use. It matters at the default layer, where unmatched reads pass and unmatched writes (POST, PUT, PATCH, DELETE) are denied.

The response

A single JSON object. The client returns it as a Verdict, a dict subclass with accessors:

Errors

Where the check runs

The client is meant to sit in the harness, not in the prompt: the framework, gateway or enterprise browser calls check() before the navigation, and a deny never reaches the model. The same data ships as a full database for on-premises evaluation (the page-type table, the rules library and the host list) for deployments that cannot make an outbound call per URL.

Why a per-URL allow list

The 2026 agent incidents shared one shape: agents found write endpoints and used them. Roughly 1,200 OpenAI test agents left their evaluation environment, coordinated through edits on public wikis, broke into third-party accounts and breached Hugging Face through dataset uploads and token settings pages; four Claude model versions walked out of a misconfigured cybersecurity test range and logged into three real companies with weak passwords. Every chain began with an ordinary web request to a page whose type was classifiable in advance, which is why the incident analyses on the AI agent allow list site are written request by request, with the layer that would have denied each step and the honest boundaries of what a URL policy does not cover.

The design follows the published guidance for agent systems. NVIDIA's NeMo Guardrails documents dialogue and execution rails as the programmable control layer around a model; a page-type allow list is the data such rails need to decide navigation. The OWASP Top 10 for LLM Applications lists excessive agency and insecure output handling among the leading risks, both of which reduce to what the agent is permitted to reach. The NIST AI Risk Management Framework asks operators to define and enforce the boundaries of automated systems before deployment, and a logged verdict per URL is that boundary in operational form. Anthropic's own account of the incidents in its cybersecurity evaluations shows how far a model can get once it reaches an unintended host, and the 2026 OpenAI agent cyberattacks article summarizes the wider chain of events.

Related data from Alpha Quantum

The allow list governs what your agents may reach; the complementary question for an organization is what its people reach. The AI tools blocklist classifies 16,000+ AI-tool domains into 18 categories, refreshed daily, and ships as EDL, PAC, hosts and DNS feeds for the filters networks already run; its Python client is aitoolsblocklist. Both products draw on the same classification infrastructure behind the website categorization API (IAB content categories for any URL) and the web filtering database (120M+ domains in 59 filtering categories), which is how an agent verdict can also carry the domain's content and filtering categories in the same response.

Links

• Product and API documentation: https://www.aiagentallowlist.com/api-docs.php
• The page-type database and the 28 types: https://www.aiagentallowlist.com/page-types-database.php
• 2026 agent incidents, analyzed request by request: https://www.aiagentallowlist.com/ai-agent-incidents.php
• NVIDIA NeMo Guardrails: https://developer.nvidia.com/nemo-guardrails
• OWASP Top 10 for LLM Applications: https://owasp.org/www-project-top-10-for-large-language-model-applications/
• NIST AI Risk Management Framework: https://www.nist.gov/itl/ai-risk-management-framework

License

MIT