LlamaIndex Readers Integration: Scrappey

Use Scrappey to load web pages into LlamaIndex as clean Markdown, bypassing anti-bot protections (Cloudflare, DataDome, PerimeterX, etc.).

Scrappey fetches the page through its anti-bot proxy and returns server-side Markdown directly (via the markdown: true flag). This reader reads that Markdown straight into a LlamaIndex Document. If a response ever lacks server-side Markdown, the reader falls back to local conversion with markdownify.

Installation

pip install llama-index-readers-scrappey

Setup

1. Sign up at scrappey.com and grab your API key.
2. Pass it to the reader (or set SCRAPPEY_API_KEY and read from env):

Quickstart

import os
from llama_index.core import VectorStoreIndex
from llama_index.readers.scrappey import ScrappeyReader

reader = ScrappeyReader(api_key=os.environ["SCRAPPEY_API_KEY"])

documents = reader.load_data(
    [
        "https://example.com",
        "https://en.wikipedia.org/wiki/Web_scraping",
    ]
)

index = VectorStoreIndex.from_documents(documents)
query_engine = index.as_query_engine()
print(query_engine.query("Summarize web scraping."))

Async

import asyncio

async def main():
    reader = ScrappeyReader(api_key="...")
    docs = await reader.aload_data(["https://example.com"])
    print(docs[0].text[:200])

asyncio.run(main())

Raw HTML instead of Markdown

reader = ScrappeyReader(api_key="...", as_markdown=False)
docs = reader.load_data(["https://example.com"])
# docs[0].text is the raw HTML returned by Scrappey

Document schema

Each URL produces one Document:

Constructor options

Roadmap

v0.1 is intentionally minimal (URLs → Markdown Documents). Planned for later releases:

• Session reuse (sessions.create / sessions.destroy) for cheaper crawls and consistent fingerprinting
• proxyCountry, premiumProxy, browser configuration
• browserActions, customHeaders, cookies, postData passthrough for JS-heavy and auth-gated pages
• POST-body scraping via cmd: "request.post"

Links

• Scrappey homepage
• Scrappey wiki