outcrawl

The Python client for Outcrawl: scrape, crawl, search and run agent browser tasks, on one credit balance. Pure Python — one py3-none-any wheel for every operating system, and no compiler on any of them.

pip install outcrawl

Requires Python 3.10 or newer. One runtime dependency, httpx, which is pure Python, as is everything it pulls.

Usage

import asyncio
from outcrawl import Outcrawl

async def main() -> None:
    async with Outcrawl() as oc:                       # reads OUTCRAWL_API_KEY
        doc = await oc.scrape("https://example.com", formats=["markdown"])
        print(doc.markdown)                            # or doc["markdown"] — it is the wire object
        print(doc.usage)                               # every call returns what it cost

        job = oc.agent(
            task="Report the title of the top story on Hacker News.",
            caps={"budget": "2.00"},                   # the one ceiling a submit must carry
        )
        async for record in job:                       # the event cursor, while it runs
            print(record["kind"], record.get("text"))
        run = await job                                # and the settled run
        print(run["status"], run["data"])

asyncio.run(main())

await and async for on one job is one run and not two: they are separate requests against one durable row, so watching a run and having its answer is not a choice.

Configuration

The same two names @outcrawl/sdk, the outcrawl CLI and the Outcrawl MCP server read. Pass Outcrawl(api_key=..., base_url=...) to override them per client.

Surface

Every one of the 34 capabilities in the registry is reachable, and tests/test_registry.py fails when one is not.

• oc.scrape(url, **options) / oc.crawl(url, **options) / oc.search(query, **options) — crawl is async-iterable and single-pass.
• oc.crawl.job(id) — the same crawl by id, with status(), results(), cancel(), and oc.crawl.get(id) / .results(id) / .cancel(id) for an id on its own. A streaming crawl belongs to its connection, so hanging up cancels it: the row settles cancelled with every page it had already delivered, and these routes are how you read the pages you paid for. The id is on handle.job["id"] from the first progress frame — store it before you need it.
• oc.agent(task=..., caps=...) — caps is required at runtime and caps.budget is required inside it; caps.steps and caps.duration are optional and unset unless you name them, and all three are hard stops when present. The job is awaitable, async-iterable, and carries everything a handle does: status(), results(), events(), cancel(), control(), answer(), add_file(). oc.agent.get(id) and friends reach a run by id — an id from a webhook needs no submit. schema accepts a JSON Schema object and constrains run["data"] to it, the same typed submit @outcrawl/sdk documents.
• oc.profiles / oc.secrets / oc.rules / oc.integrations / oc.sessions / oc.monitors. oc.secrets has deliberately no get: there is no route to widen. A run reaches a value only by naming its HANDLE in the submit, and the substitution happens below the model.
• oc.usage(**query) and oc.credits() — every credit figure is a decimal string; use decimal.Decimal, never float().
• CAPABILITY_AVAILABILITY — whether anything is SERVED behind a declared route. A capability nothing serves is refused in your own process, with what is missing, rather than as a 503.

Errors

Identical to the TypeScript SDK's: same classes, same fields, same sentences. Branch on error.code, never on str(error).

from outcrawl import ProfileInUseError, QuotaExceededError

try:
    ...
except ProfileInUseError as leased:
    retry_at = leased.held_until          # a retry that knows when to retry
except QuotaExceededError as over:
    await asyncio.sleep(over.retry_after_seconds)

tests/error_parity.json is emitted from the TypeScript SDK's own error reconstruction and read by both test suites, so neither surface can drift from the other without turning its own suite red.

What is not here

oc.browser(). The live browser needs a CDP connection driven in the caller's own process, which is why it is deliberately not a registry capability and why the TypeScript SDK is the one place it exists. The same page capabilities are reachable through agent and scrape, which run the loop on our side. That is a difference in kind, not an omission.

The same reason takes Session.take_control with it: the TypeScript SDK's takeControl() opens a CDP connection to a running session's own page so a person can drive it directly, and that connection has to live in the caller's process for the same reason oc.browser() does. Everything else on a session — get, list, export, live() — is a row read or a held stream and is here.

Development

pip install -e '.[dev]'                  # pytest + pytest-asyncio, never runtime dependencies
python3 scripts/gen_registry.py          # regenerate registry.py from packages/core/src/registry.ts
python3 scripts/gen_registry.py --check  # or just fail if it is stale
bun ../sdk-python/scripts/gen_error_parity.ts   # regenerate the shared error contract
python3 -m pytest tests -q               # the drift, parity and ergonomics tests

From the repo root, npm run test:python runs the staleness check and the suite together on the SUPPORTED FLOOR — uv run --extra dev --python 3.10 pytest — so the gate exercises 3.10 rather than whatever python3 happens to be. Without uv: pip install -e '.[dev]' and python3 -m pytest tests -q.

The two registry-versus-TypeScript tests skip when run from an unpacked sdist, which has no TypeScript to compare against, and run whenever the monorepo is present. The reachability test — the one that fails when a registry row has no method — has no such dependency and always runs.