aiagentallowlist

Python client for the AI agent allow list, the lookup API that answers one question for a web-browsing AI agent: may it open this exact URL with this HTTP method? Each answer is drawn from a database of 40 million+ domains with verified URLs for up to 28 page types per domain, built by analyzing over 10 billion links, combined with about 40 method-aware URL-pattern rules and a curated High-Value Host List. Documentation, pricing and product pages pass; login, signup, checkout, upload and wiki-edit surfaces are denied before the request is sent.

The package depends only on requests, supports Python 3.7 and newer, and is a direct wrapper around a single GET endpoint. Source lives on GitHub with a mirror on GitLab.

Contents

1. Install and first verdict
2. The response, field by field
3. Client reference
4. Recipe: Playwright for Python and browser-use
5. Recipe: guarding tools in the OpenAI Agents SDK and LangChain
6. Recipe: a FastAPI egress gateway
7. Recipe: auditing a URL list with pandas
8. How the three layers decide
9. Page types by policy group
10. What the 2026 incidents taught
11. Related data
12. Frequently asked questions
13. Related packages

Install and first verdict

pip install aiagentallowlist

from aiagentallowlist import AIAgentAllowlistClient

client = AIAgentAllowlistClient("YOUR_API_KEY")

# Full URL: verdict for that exact URL
v = client.check("https://stripe.com/login")
print(v.verdict, v.matched_layer, v.matched_id)   # deny page_type_db login

# Bare domain: verdict at the root plus the verified page-type map
rec = client.check("stripe.com")
print(rec.page_types["pricing"])                    # https://stripe.com/pricing
print(sorted(rec.page_types))                       # every confirmed type on the domain

# The write surfaces an agent should stay away from, as real URLs
print(client.deny_list("huggingface.co"))

The API key is shown in the account area as soon as a subscription is activated. The client puts it in the X-API-Key header of every request; the query-parameter form api_key= also works against the API for quick shell tests, but headers keep keys out of access logs, which is why the client uses them.

The response, field by field

Every request goes to the same place:

GET https://www.aiagentallowlist.com/api/check?url=<full URL or bare domain>[&method=GET]

The client returns a Verdict, a dict subclass, so the raw JSON is always there and a handful of properties sit on top.

Key	Property	Meaning
found	v.found	the domain has a record in the database
verdict	v.verdict, v.allowed, v.denied, v.flagged	allow, deny or flag
verdict_scope		url for a full URL, domain_root for a bare domain
matched	v.matched_layer, v.matched_id	the deciding layer (high_value_hosts, page_type_db, rules, default), the entry id and a note
page_types	v.page_types	{type: verified_url} for up to 28 page types
language		primary language of the domain
iab_category		IAB content category, from a 700+ category taxonomy
filtering_categories		web-filtering categories, from a 59-category taxonomy
open_page_rank, global_rank		Open PageRank score and global rank
remaining_lookups		lookups left on the plan in the current 30-day cycle

The url value is never stripped: a query string, a locale prefix or a fragment are all part of what is judged. Subdomains without their own record fall back to the base domain, so chat.openai.com resolves to openai.com.

Client reference

AIAgentAllowlistClient(api_key, base_url="https://www.aiagentallowlist.com/api",
                       timeout=30, session=None, max_retries=2)

Method	Returns	Notes
check(url, method="GET")	Verdict	one lookup, all three layers evaluated
is_allowed(url, method="GET")	bool	True only for allow; flag is False
page_types(domain)	dict	verified page-type map
deny_list(domain, types=...)	list[str]	verified deny-side URLs; default types are login, signup, checkout, cart, upload, post_create, comment, subscribe, password_reset
check_many(urls, method="GET", pause=0.0)	iterator of Verdict	lazy, one lookup per URL, optional sleep between calls

Pass your own requests.Session when you already run a connection pool, proxies or a retry adapter; the client only adds its headers to it.

Exceptions, all subclasses of AIAgentAllowlistError:

HTTP	Exception	Meaning
400	BadRequestError	url could not be parsed into a host
401	AuthenticationError	missing or unknown key
403	QuotaError	account not activated, or monthly quota exhausted
429	RateLimitError	too many requests; retried twice with a pause before raising

WRITE_METHODS is exported as ("POST", "PUT", "PATCH", "DELETE") for harnesses that want to mirror the server's read/write distinction locally.

Recipe: Playwright for Python and browser-use

A route handler sees every request before the browser sends it. Deny verdicts abort the navigation; the page never loads, and nothing about the form reaches the model.

import asyncio
from playwright.async_api import async_playwright
from aiagentallowlist import AIAgentAllowlistClient, AIAgentAllowlistError

client = AIAgentAllowlistClient("YOUR_API_KEY")
_cache = {}

def verdict_for(url, method):
    key = (method, url)
    if key not in _cache:
        _cache[key] = client.check(url, method)
    return _cache[key]

async def gate(route):
    req = route.request
    if not req.is_navigation_request():
        await route.continue_()
        return
    try:
        v = await asyncio.to_thread(verdict_for, req.url, req.method)
    except AIAgentAllowlistError as exc:
        print("allow list unavailable, denying:", exc)
        await route.abort("blockedbyclient")
        return
    if v.denied:
        print(f"denied {req.method} {req.url} by {v.matched_layer}:{v.matched_id}")
        await route.abort("blockedbyclient")
        return
    await route.continue_()

async def main():
    async with async_playwright() as p:
        browser = await p.chromium.launch()
        page = await browser.new_page()
        await page.route("**/*", gate)
        await page.goto("https://stripe.com/pricing")          # allowed
        try:
            await page.goto("https://dashboard.stripe.com/login")  # denied
        except Exception as exc:
            print("navigation stopped:", exc)
        await browser.close()

asyncio.run(main())

browser-use drives Playwright underneath, so the same page.route gate applies: attach it to the browser context that browser-use creates, and every navigation the agent decides on passes through the allow list first.

Recipe: guarding tools in the OpenAI Agents SDK and LangChain

Frameworks call tools; the guard belongs in the tool. The wrapper below turns a refusal into structured data the model can act on, including the verified read-safe URLs on the same domain.

import json
import httpx
from aiagentallowlist import AIAgentAllowlistClient

client = AIAgentAllowlistClient("YOUR_API_KEY")

def guarded_fetch(url: str, method: str = "GET") -> str:
    v = client.check(url, method)
    if not v.allowed:
        return json.dumps({
            "refused": True,
            "reason": f"{v.verdict} by {v.matched_layer}:{v.matched_id}",
            "read_safe_urls": {k: v.page_types[k] for k in ("documentation", "pricing", "help_center")
                               if k in v.page_types},
        })
    return httpx.request(method, url, timeout=30).text[:20000]

OpenAI Agents SDK:

from agents import Agent, function_tool

@function_tool
def read_page(url: str) -> str:
    """Fetch the text of a public web page. Login, checkout and upload pages are refused."""
    return guarded_fetch(url, "GET")

@function_tool
def submit_form(url: str) -> str:
    """Submit a form. Refused on credential, payment and content-write surfaces."""
    return guarded_fetch(url, "POST")

researcher = Agent(name="researcher", tools=[read_page, submit_form])

LangChain:

from langchain_core.tools import tool

@tool
def read_page(url: str) -> str:
    """Fetch the text of a public web page after an allow list check."""
    return guarded_fetch(url, "GET")

When the model is refused stripe.com/login it also receives docs.stripe.com and stripe.com/pricing, so the research task finishes on the read-safe surface rather than in a retry loop against the form.

Recipe: a FastAPI egress gateway

When several agents share one outbound path, enforce policy once at the egress. The gateway takes {url, method}, checks the allow list, performs the request itself and writes an audit line.

import logging
import httpx
from fastapi import FastAPI, HTTPException
from pydantic import BaseModel
from aiagentallowlist import AIAgentAllowlistClient, AIAgentAllowlistError

app = FastAPI()
client = AIAgentAllowlistClient("YOUR_API_KEY", max_retries=3)
audit = logging.getLogger("egress")

class Egress(BaseModel):
    url: str
    method: str = "GET"
    body: str | None = None
    agent_id: str = "unknown"

@app.post("/egress")
async def egress(req: Egress):
    try:
        v = client.check(req.url, req.method)
    except AIAgentAllowlistError as exc:
        raise HTTPException(503, f"allow list unavailable: {exc}")   # fail closed
    audit.info("agent=%s %s %s verdict=%s layer=%s id=%s remaining=%s",
               req.agent_id, req.method, req.url, v.verdict, v.matched_layer,
               v.matched_id, v.get("remaining_lookups"))
    if v.denied:
        raise HTTPException(403, {"error": "denied by allow list",
                                  "layer": v.matched_layer, "id": v.matched_id})
    async with httpx.AsyncClient(timeout=30) as http:
        upstream = await http.request(req.method, req.url, content=req.body)
    return {"status": upstream.status_code, "body": upstream.text}

The audit line is the artefact compliance reviews ask for: every URL an agent asked to open, the verdict, the layer, and the remaining quota so operations can alert before a run stalls.

Recipe: auditing a URL list with pandas

Before an agent is let loose on a list of targets, it is worth knowing how many of them are write surfaces. One lookup per row, then a pivot.

import pandas as pd
from aiagentallowlist import AIAgentAllowlistClient

client = AIAgentAllowlistClient("YOUR_API_KEY")
urls = pd.read_csv("targets.csv")["url"]

rows = []
for v in client.check_many(urls, method="GET", pause=0.05):
    rows.append({
        "url": v.get("url"), "found": v.found, "verdict": v.verdict,
        "layer": v.matched_layer, "id": v.matched_id,
        "iab": v.get("iab_category"), "language": v.get("language"),
        "n_page_types": len(v.page_types),
    })

df = pd.DataFrame(rows)
print(df["verdict"].value_counts())
print(df.pivot_table(index="layer", columns="verdict", values="url", aggfunc="count", fill_value=0))
df[df["verdict"] != "allow"].to_csv("targets_to_review.csv", index=False)

The same loop with method="POST" shows which rows would be denied as writes, which is the honest measure of how much an unattended agent could change on those sites.

How the three layers decide

Order	Layer	Contents	Outcome
1	High-Value Host List	about 60 curated infrastructure hosts: cloud consoles, package registries, paste sites, webhook and tunnel sinks, mail senders, cloud metadata endpoints	hard deny
2	Page-type database	the domain's verified URLs for up to 28 page types, matched exactly	deny, flag or allow by type
3	Rules library	about 40 method-aware URL-pattern rules, applied on any domain	deny or flag
4	Default	anything unmatched	reads pass; POST, PUT, PATCH, DELETE are denied

The default layer is the product's stance in one line: unknown reads are fine, unknown writes are not. The HTTP method is what separates the two, so pass the method your harness actually intends to use. The same wiki edit URL is a read on GET and a write on POST.

This is the "excessive agency" control described in the OWASP Top 10 for LLM Applications made concrete: agency is bounded by the set of URLs and methods the operator permits. MITRE ATLAS catalogues the adversary techniques that exploit agents which can reach more than they need, and the NIST AI Risk Management Framework asks for those boundaries to be defined, enforced and logged before deployment.

Page types by policy group

Group	Default policy	Types
Navigation and research	allow	pricing, documentation, blog, about, leadership, careers, partners, case_studies, press, status, product, events, community, help_center, integrations, sitemap, contact
Identity	deny	login, signup, password_reset
Commerce	deny or flag	cart, checkout, subscribe
Content write	deny	post_create, comment, upload
Trust and policy	allow, some teams restrict	legal, security

The defaults are a starting policy; your own rules sit on top of the raw classification. What matters is that each type is stored as the URL the site really links to, which is why dashboard.stripe.com/login is in the record and /login is not guessed. A URL, per RFC 3986, is a full identifier with scheme, host, path and query, and the verdict is for that identifier, not for a pattern.

What the 2026 incidents taught

The 2026 OpenAI agent cyberattacks and the evaluation-range escape of four Anthropic model versions followed one pattern: agents reached write endpoints and used them. Roughly 1,200 test agents left their evaluation environment, coordinated through edits on public wikis, broke into third-party accounts and breached Hugging Face through dataset uploads and token settings pages; in the other case, models logged into three real companies with weak passwords.

Every chain started with an ordinary web request to a page whose type was classifiable in advance: a wiki edit URL, a new-dataset upload form, a token settings page, a login form. The incident analyses on the allow list site go through each chain request by request and name the layer that would have denied the step, including the honest cases, such as SSH inside a test range, that a URL policy does not cover.

Operational notes

• Cache verdicts for the life of a session. Agents ask about the same handful of URLs many times, and a dictionary keyed on method and URL removes most repeat lookups.
• Send the full URL to check() rather than matching page_types() locally. The host list and the rules library only run on the server, and they fire on domains the database has no record for.
• Read remaining_lookups on every response and alert at a threshold. Discovering an exhausted quota as a QuotaError in the middle of a run is the expensive way to learn it.
• Fail closed. When the lookup raises, deny the navigation and notify a person. A harness that guesses while the policy service is unreachable is the failure the policy exists to prevent.
• Reuse one requests.Session across threads or pass your own; the client only adds headers to it.

Related data

The allow list governs where your agents may go. Two sibling products cover what your people do with AI. The classified AI tool domains database holds 20,000+ AI-tool domains in 18 functional categories, refreshed daily, with feeds for firewalls, DNS resolvers and secure web gateways, and sector policy profiles in paid plans. To find the AI tools employees use from a DNS, proxy or firewall export, the shadow AI service produces a dated inventory with vendor training verdicts and a PDF evidence pack, no agent or SSL inspection required.

All three share the classification infrastructure behind the website categorization API and the web filtering database, which is why every verdict also carries IAB and filtering categories for the domain.

Frequently asked questions

What is an AI agent allow list? An AI agent allow list is a policy layer that decides, per URL and per HTTP method, whether a web-browsing AI agent may open a page. The AI agent allow list at aiagentallowlist.com holds verified page-type URLs for 40 million+ domains, so verdicts come from the real login, checkout, upload and settings URLs of each site rather than guessed paths.

How do I keep a Python agent from logging in, signing up or submitting forms? Call check(url, method) before every navigation and every tool call that touches the web. A deny verdict for login, signup, checkout, cart, upload, comment, subscribe or password-reset pages stops the request in your harness. pip install aiagentallowlist and wire it into the Playwright route, the tool function or the gateway, as in the recipes above.

Which page types does the AI agent allow list know? Up to 28 per domain, grouped as 17 navigation and research types, three identity types, three commerce types, three content-write types and two trust pages. The full catalogue is on the page-type database page.

Does the AI agent allow list work for domains it has no record for? Yes. The High-Value Host List and the rules library fire on any domain, and the default layer denies unmatched writes everywhere. A domain without a record still cannot be written to by an unattended agent.

How is the AI agent allow list different from robots.txt or a domain blocklist? robots.txt is a voluntary crawl hint for crawlers and says nothing about write endpoints; a domain blocklist cannot allow a site's documentation while denying its token settings page. The AI agent allow list is operator-enforced, per URL, per method, and built from verified URLs.

Can the AI agent allow list be used with browser-use, LangChain or the OpenAI Agents SDK? Yes, with any framework that can run a function before a navigation or inside a tool. The recipes above cover Playwright and browser-use, the OpenAI Agents SDK, LangChain and a FastAPI gateway; a Node.js client is published as aiagentallowlist on npm.

Can the AI agent allow list run without outbound API calls? Yes. Database licences ship the page-type table, the rules library and the High-Value Host List for evaluation inside your own proxy or policy engine, returning the same verdicts as the hosted API.

What does the AI agent allow list cost? API plans from $99 to $1,997 per month by lookup volume, database licences from $14,999 one-time, OEM licensing scoped to the product. See aiagentallowlist.com/pricing.php.

Who builds the AI agent allow list? Alpha Quantum, the company behind the website categorization API, the web filtering database, the AI tools blocklist and the shadow AI detection service, with more than 300 organisations using its domain intelligence since 2022.

Related packages

• PyPI: aiblocklist, aitoolsblocklist, shadowaitools, phishingdetectionapi, websiteclassificationapi, cipawebfiltering
• npm: aiagentallowlist, aiblocklist, aitoolsblocklist, shadowaitools, phishingdetectionapi, webfilteringdatabase, websitecategorization, cipawebfiltering
• Products: website categorization API, web filtering database, phishing detection API, CIPA web filtering, PII detection API
• Source: github.com/explainableaixai/aiagentallowlist, gitlab.com/url-classifications/aiagentallowlist

Links

• API documentation: aiagentallowlist.com/api-docs.php
• Page-type database: aiagentallowlist.com/page-types-database.php
• 2026 agent incidents: aiagentallowlist.com/ai-agent-incidents.php
• OWASP Top 10 for LLM Applications: owasp.org
• MITRE ATLAS: atlas.mitre.org
• NIST AI Risk Management Framework: nist.gov
• RFC 3986, Uniform Resource Identifier: rfc-editor.org

License

MIT