llamaindex-webmetadata-extractor

LlamaIndex integration for the Web Metadata, OpenGraph & Contact Extractor API — a Reader that ingests any URL as a clean-Markdown Document for RAG, plus agent tools for contact discovery and SEO auditing.

pip install llamaindex-webmetadata-extractor

Quick start

Get a free API key on RapidAPI (1,000 requests/month, no credit card required).

export WEBMETADATA_API_KEY=your-rapidapi-key

As a Reader (RAG ingestion)

from llamaindex_webmetadata_extractor import WebMarkdownReader
from llama_index.core import VectorStoreIndex

reader = WebMarkdownReader()  # reads WEBMETADATA_API_KEY, or pass api_key=...
documents = reader.load_data(["https://example.com", "https://another-site.com"])

index = VectorStoreIndex.from_documents(documents)

Each page's boilerplate (nav, ads, cookie banners) is stripped server-side before it reaches you, so Document.text is close to ready for chunking/embedding as-is. Document.metadata also carries the page's title, word count, reading time, and a short summary snippet. A URL that fails to fetch is skipped by default (skip_errors=True); pass skip_errors=False to raise instead.

As agent tools

from llamaindex_webmetadata_extractor import get_tools
from llama_index.core.agent import ReActAgent

tools = get_tools(api_key="YOUR_RAPIDAPI_KEY")
agent = ReActAgent.from_tools(tools, llm=llm)

Each tool returns a JSON string; API errors come back as {"error": true, "status_code": ..., "detail": ...} instead of raising, so an agent loop can react to them without crashing. For content ingestion (as opposed to a one-off agent tool call), use WebMarkdownReader above instead.

Configuration

Pass api_key explicitly to the reader/tool factories, or set it once via the WEBMETADATA_API_KEY environment variable and omit it everywhere.

Links

• Python SDK this wraps (webmetadata-extractor)
• Also available for LangChain and CrewAI
• Full API documentation
• Live interactive demo — no signup needed

License

MIT