langchain-agentreader

LangChain document loader for AgentReader: turn web pages, JavaScript-rendered pages, RSS/Atom feeds and PDF URLs into clean, compact Markdown for LLMs and RAG.

• Works without an API key on the free tier (5 reads/day per IP).
• For more, buy prepaid credits at https://agentreader.xyz/buy ($3 for 2,000 reads, $10 for 7,000) and set AGENTREADER_API_KEY.
• Failed reads (bot checks, paywalls, upstream errors) return an error and are not charged.

Install

pip install langchain-agentreader

Usage

from langchain_agentreader import AgentReaderLoader

loader = AgentReaderLoader([
    "https://react.dev/learn",
    "https://arxiv.org/pdf/1706.03762",   # PDF URLs work too
])
docs = loader.load()
print(docs[0].metadata["title"])
print(docs[0].page_content[:500])

With a key (env var, recommended):

export AGENTREADER_API_KEY=ar_...

or AgentReaderLoader(urls, api_key="ar_...").

Options

Metadata

Each Document has source (final URL after redirects), requested_url, and when available title, author, date, siteName, description, lang, contentType, pageType, wordCount, pages (PDFs), rendered_with_browser, fields.

Errors

AgentReaderError carries status, code and reason (for example 402 when the free tier is used up, 502 with bot_challenge when a site blocks automated reads).

Links

• API docs: https://agentreader.xyz/llms.txt
• Remote MCP server: https://agentreader.xyz/mcp
• LlamaIndex reader: llama-index-readers-agentreader

MIT licensed.