crawl4ai-client

Lightweight async Python client for Crawl4AI Docker server.

No browser dependencies required. Just httpx + pydantic (~2MB vs ~500MB for the full crawl4ai package).

Install

pip install crawl4ai-client

Quick Start

import asyncio
from crawl4ai_client import Crawl4aiDockerClient

async def main():
    async with Crawl4aiDockerClient(
        base_url="http://localhost:11235",
        api_token="your-token",  # optional
    ) as client:
        result = await client.crawl(["https://example.com"])
        print(result.raw_markdown)

asyncio.run(main())

Features

• Crawl single or multiple URLs (/crawl)
• Stream results as they complete (/crawl/stream)
• Markdown extraction with filters (/md)
• Screenshots as base64 PNG (/screenshot)
• PDF generation (/pdf)
• HTML preprocessing for schema extraction (/html)
• JavaScript execution on pages (/execute_js)
• LLM Q&A — ask questions about page content (/llm)
• Per-URL configs for batch crawling (crawler_configs list)
• Schema retrieval (/schema)
• Async context manager with automatic cleanup

Usage

Basic crawl

from crawl4ai_client import Crawl4aiDockerClient, CrawlerRunConfig, CacheMode

async with Crawl4aiDockerClient(base_url="http://localhost:11235") as client:
    result = await client.crawl(
        ["https://example.com"],
        crawler_config=CrawlerRunConfig(cache_mode=CacheMode.BYPASS),
    )
    print(result.raw_markdown)

Multiple URLs with per-URL configs

from crawl4ai_client import Crawl4aiDockerClient, CrawlerRunConfig

async with Crawl4aiDockerClient(base_url="http://localhost:11235") as client:
    results = await client.crawl(
        ["https://example.com", "https://httpbin.org/html"],
        crawler_configs=[
            CrawlerRunConfig(word_count_threshold=5),
            CrawlerRunConfig(word_count_threshold=50),
        ],
    )
    for r in results:
        print(f"{r.url}: {len(r.raw_markdown)} chars")

Deep crawl

from crawl4ai_client import Crawl4aiDockerClient, CrawlerRunConfig, BFSDeepCrawlStrategy

async with Crawl4aiDockerClient(base_url="http://localhost:11235") as client:
    results = await client.crawl(
        ["https://example.com"],
        crawler_config=CrawlerRunConfig(
            deep_crawl_strategy=BFSDeepCrawlStrategy(max_depth=2, max_pages=10),
        ),
    )
    for r in results:
        print(f"{r.url}: {r.success}")

Also available: DFSDeepCrawlStrategy, BestFirstCrawlingStrategy.

Streaming

async with Crawl4aiDockerClient(base_url="http://localhost:11235") as client:
    async for result in client.crawl_stream(["https://example.com", "https://httpbin.org/html"]):
        print(f"Got: {result.url}")

Markdown endpoint

async with Crawl4aiDockerClient(base_url="http://localhost:11235") as client:
    md = await client.get_markdown("https://example.com", content_filter="fit")
    print(md)

Screenshot

async with Crawl4aiDockerClient(base_url="http://localhost:11235") as client:
    screenshot_b64 = await client.screenshot("https://example.com")

PDF generation

async with Crawl4aiDockerClient(base_url="http://localhost:11235") as client:
    pdf_b64 = await client.get_pdf("https://example.com")

HTML preprocessing

async with Crawl4aiDockerClient(base_url="http://localhost:11235") as client:
    html = await client.get_html("https://example.com")

JavaScript execution

async with Crawl4aiDockerClient(base_url="http://localhost:11235") as client:
    result = await client.execute_js(
        "https://example.com",
        scripts=["document.title", "document.querySelectorAll('a').length"],
    )
    print(result.js_execution_result)

LLM Q&A

async with Crawl4aiDockerClient(base_url="http://localhost:11235") as client:
    answer = await client.llm_query(
        "https://example.com",
        query="What is this page about?",
    )
    print(answer)

Hooks (custom page interaction)

async def block_images(page, context, **kwargs):
    await context.route("**/*.{png,jpg,jpeg,gif}", lambda route: route.abort())
    return page

async def scroll_page(page, context, **kwargs):
    await page.evaluate("window.scrollTo(0, document.body.scrollHeight)")
    await page.wait_for_timeout(2000)
    return page

async with Crawl4aiDockerClient(base_url="http://localhost:11235") as client:
    result = await client.crawl(
        ["https://example.com"],
        hooks={
            "on_page_context_created": block_images,
            "before_retrieve_html": scroll_page,
        },
        hooks_timeout=30,
    )

Hooks can also be passed as pre-stringified source code:

hooks={"on_page_context_created": 'async def hook(page, context, **kwargs):\n    return page'}

Why this package?

The full crawl4ai package installs 34+ dependencies (~500MB) including Playwright, browsers, numpy, and litellm. If you're running Crawl4AI as a Docker service and only need the client, this package gives you the same Crawl4aiDockerClient with just 2 dependencies.

Compatibility

This client is compatible with Crawl4AI Docker server v0.8.x+. The config classes (BrowserConfig, CrawlerRunConfig) produce the same serialized format as the full library.

License

Apache 2.0 — based on crawl4ai by unclecode.