xdive

Python client for XDive: scrape pages to Markdown, crawl and map sites, search the web, and answer questions with verified citations. Synchronous and asynchronous clients, typed errors, automatic retries, and LangChain and LlamaIndex integrations.

Full documentation: https://crawler.mehdisadeghian.com/docs/sdk-python

Install

pip install xdive
pip install "xdive[langchain]"     # LangChain tools and document loader
pip install "xdive[llama-index]"   # LlamaIndex reader

Requires Python 3.9 or later. The only runtime dependency is httpx.

Quickstart

from xdive import XDive, page_markdown

with XDive() as xdive:  # reads XDIVE_API_KEY
    page = xdive.scrape("https://example.com", formats=["markdown", "links"])
    print(page_markdown(page))
    site = xdive.crawl("https://docs.example.com", max_pages=50)  # waits for the crawl
    answer = xdive.research("What does PEP 639 change about license metadata?")
    print(answer["answer"])
    for citation in answer["citations"]:
        print(citation["number"], citation["url"])

Create an API key under API Keys in the dashboard. A key belongs to one organization; the client looks the organization up once through GET /whoami, so the key is all you configure.

Configuration

A missing key raises ValueError when the client is created.

Methods

XDive and AsyncXDive have the same methods; on AsyncXDive each is a coroutine. Keyword options are passed through as the API's JSON fields.

XDive.org_id is a property; on AsyncXDive it is a coroutine, await client.org_id().

formats takes the API's format names (markdown, text, html, rawHtml, links, images, json, summary, question, highlights, branding, audio, screenshot, product, menu) or dicts such as {"type": "json", "schema": {...}}. page_markdown(page) returns a page's Markdown whichever field carries it (artifacts.markdown.content or markdown).

Async

import asyncio
from xdive import AsyncXDive

async def main():
    async with AsyncXDive() as client:
        pages = await asyncio.gather(*(client.scrape(u) for u in ["https://a.example", "https://b.example"]))

asyncio.run(main())

Close clients you do not use as context managers: xdive.close() or await client.aclose().

Errors and retries

Only overload is retried: a 429 (organization rate or concurrency limit) and a 503 whose JSON body has the code SERVER_BUSY (server at capacity). The client waits the number of seconds in the response's Retry-After header (1 second when absent) before each retry. Every other error, including any other 503 (for example a feature the server has not configured), raises XDiveError at once with the API's status, message and code. Network errors and timeouts surface as httpx exceptions.

Crawl polling sends If-None-Match, so an unchanged status comes back as an empty 304, and waits the interval the server sets in Retry-After, falling back to poll_interval.

Integrations

from xdive.langchain import XDiveLoader, XDiveResearchTool, XDiveScrapeTool, XDiveSearchTool

tools = [XDiveSearchTool(), XDiveScrapeTool(), XDiveResearchTool()]
docs = XDiveLoader("https://docs.example.com", mode="crawl", max_pages=100).load()

from xdive.llama_index import XDiveReader

documents = XDiveReader().load_data("https://docs.example.com", mode="crawl", max_pages=100)

XDiveSearchTool returns each result's query-relevant passages rather than whole pages, which keeps an agent's context small. See the LangChain and LlamaIndex guides.

Self-hosted

xdive = XDive(api_key="ck_live_…", base_url="https://your-host/api")

or set XDIVE_BASE_URL=https://your-host/api.

Development

pip install -e ".[dev]"
pytest

License

MIT. The license text is in the LICENSE file included in the package.