opencraw (Python)

Crawl with OpenCraw from Python, with no Node to install: Crawler downloads the driver's runtime for this platform once (a Node binary and the engine in one archive, checked against its SHA-256 and cached in ~/.cache/opencraw/driver/), starts it, and talks to it over its pipes. Standard library only; Python 3.12+.

from opencraw import Crawler

with Crawler(version="0.6.0") as crawler:

    @crawler.hook()
    def tidy(value, args, request):  # a hook the recipes call: plain Python
        return str(value).strip().lower()

    run = crawler.run(paths=["recipes/"])
    for record in run.records():  # as they are emitted
        print(record)
    print(run.report()["records"])

• crawler.run(...) takes the driver's run options (out, only, dryRun, parallel, access, ...); run.records() and run.events() yield as it crawls, run.report() waits for the end, run.cancel() stops it.
• crawler.work(next_item, ...) is worker mode: next_item() returns each window's item ({"id": ..., "vars": {...}}), or None to end.
• @crawler.hook(), @crawler.captcha_solver(), @crawler.access_plugin() register Python functions by name; raising fails that one call (CalloutError for a plain message), never the run.
• crawler.validate, probe, diff, export answer as the driver does.
• AsyncCrawler is the asyncio twin: async with AsyncCrawler(version=...) as crawler, await crawler.run(...), async for record in run.records(), and hooks, plugins and next_item may be async def.
• OPENCRAW_DRIVER=<folder> uses an unpacked runtime; OPENCRAW_DRIVER_OFFLINE=1 never downloads; command= starts any other driver (["node", "path/to/main.esm.js"]).

The rest of this page is the older way, which stays: build recipes in Python and run the opencraw CLI once per call (npm i -g @opencraw/cli, or OPENCRAW_CLI set to the command that starts it; with neither, npx @opencraw/cli is used).

A recipe stays a plain dict in the engine's own shape, so the recipe guide (docs/recipes/authoring.md) is the reference for every key. The constructors only save typing, and the engine, not this package, says what is valid.

from opencraw import input_recipe, output_recipe, run, step, validate

product = output_recipe(
    "product",
    {
        "url": {"type": "url", "required": True, "key": True},
        "title": {"type": "string", "required": True},
    },
)

shops = [
    input_recipe(
        f"shop-{name}",
        "product",
        "web",
        [url],
        [
            step("goto", url="{{start.url}}"),
            step("extract", id="title", selector=selector, kind="css", take="text"),
            step("emit"),
        ],
        {"url": {"from": "start.url"}, "title": {"from": "title"}},
    )
    for name, url, selector in [
        ("a", "https://a.example", "h1"),
        ("b", "https://b.example", ".name"),
    ]
]

result = validate(
    [product, *shops]
)  # Validation(ok, output): problems are data, not exceptions
records = run(
    [product, *shops], "--only", "shop-a"
)  # records as dicts; OpenCrawError if the crawl fails

Reserved words: pass a step's as or from through **{"as": "json"}.

write_recipes(folder, recipes) writes <id>.input.json and <id>.output.json for the CLI, the Studio and the MCP server.

A hook written in Python

A recipe's hook ({ "op": "hook", "name": "slug" }) can be a Python script. The hooks module registers it with commandHook (@opencraw/core), and the script is a few lines:

# hooks/slug.py
import re
from opencraw import serve_hook

serve_hook(
    lambda value, args, request: re.sub(r"[^a-z0-9]+", "-", str(value).lower()).strip(
        "-"
    )
)

serve_hook(handler) reads the request from stdin, calls handler(input, args, request) (input is None for a hook step; request carries context and idempotencyKey) and writes the answer to stdout. What the handler returns is the output; an exception becomes an error answer with its message, so the step fails with that reason. A handler whose work takes longer raises HookPending(retry_after_ms): OpenCraw runs the script again after that wait, with the same idempotencyKey, until it returns or the hook's maxWaitMs passes, so the script starts the work once and looks it up by that key on later runs. The request and answer shapes are in packages/core/schemas/callout-*.schema.json.

Not in this package: a Python engine and generated typed models. The crawl itself always runs in the OpenCraw CLI. It is published to PyPI as opencraw: mnci ci release versions and publishes what sits under python-packages/.

Development

smith run test -p opencraw (pytest; the CLI integration tests use this repository's built CLI and skip without it), smith run lint -p opencraw (Ruff), smith run typecheck -p opencraw (mypy strict).

Layout. One slice package per outcome: recipe_building/ (recipes in Python, written to files), hook_serving/ (hooks served as command callouts), cli_bridge/ (the CLI, run once per call), driver_runtime/ (the runtime's download and cache), driver_channel/ (JSON-RPC over the driver's pipes) and crawl_session/ (Crawler and Run). Its tests replay the driver's conformance transcripts against the repository's built driver.