outcrawl

The Python client for Outcrawl: scrape, crawl, search and run agent browser tasks, on one credit balance. Pure Python — one py3-none-any wheel for every operating system, and no compiler on any of them.

pip install outcrawl

Requires Python 3.10 or newer. One runtime dependency, httpx, which is pure Python, as is everything it pulls.

Usage

import asyncio
from outcrawl import Outcrawl

async def main() -> None:
    async with Outcrawl() as oc:                       # reads OUTCRAWL_API_KEY
        doc = await oc.scrape("https://example.com", formats=["markdown"])
        print(doc.markdown)                            # or doc["markdown"] — it is the wire object
        print(doc.usage)                               # every call returns what it cost

        job = oc.agent(
            task="Report the title of the top story on Hacker News.",
            caps={"budget": "3.00"},                   # the one ceiling a submit must carry
        )
        async for record in job:                       # the event cursor, while it runs
            print(record["kind"], record.get("text"))
        run = await job                                # and the settled run
        print(run["status"], run.get("data"))          # no "data" when a cap stopped it first

asyncio.run(main())

await and async for on one job is one run and not two: they are separate requests against one durable row, so watching a run and having its answer is not a choice.

Configuration

The same two names @outcrawl/sdk, the outcrawl CLI and the Outcrawl MCP server read. Pass Outcrawl(api_key=..., base_url=...) to override them per client.

Surface

Every one of the 37 capabilities in the registry is reachable, and tests/test_registry.py fails when one is not.

• oc.scrape(url, **options) / oc.crawl(url, **options) / oc.search(query, **options) — crawl is async-iterable and single-pass. search returns SearchHits: when one search engine refuses the query and another answers, hits.warnings carries an engines_refused entry naming the engine that refused; when every engine refuses it raises SearchRefusedError (503, retryable after retry_after_seconds, a retry is served from another exit, nothing charged) rather than returning [].

• Options are keyword arguments with the API's own names, sent as given; help(oc.scrape) and help(oc.crawl) list every one with its default. maxAge is how old a cached answer may be, in milliseconds, default 172800000 (2 days); maxAge=0 is always live, neither reading nor writing the cache. A hit bills the same page_load credit a live fetch does.

• Every scrape result and crawl page says where it came from: doc.cacheState is "hit", "miss" or "bypass", doc.cachedAt is when a hit was captured, doc.usage["cache"]["status"] agrees with it (HIT / MISS / DISABLED), doc.metadata["servedVia"] is "direct" or "residential", the network path that fetched it, and doc.metadata["egressCountry"] is the country (ISO code) of the address it was fetched from.

• A degraded page still comes back, and says so in doc["warnings"]: absent on a clean result, otherwise a list of {"code", "format"?, "message"} whose message names the numbers. partial_document — the page was read before its main document finished loading (the parser was still in it, or the response was still arriving), so markdown, html and rawHtml are a prefix of the page; retry with a larger timeoutMs or a waitFor. empty_capture — the requested markdown or html came back empty. blocked_interstitial — a bot-protection page came back instead of the page. challenge_budget_exhausted — a challenge left too little time to finish reading. source_truncated — a JSON or text body arrived cut off. json_no_content, json_empty and json_required_empty describe a json extraction: no page text, no data, or empty values for fields your schema requires. egress_country_mismatch — this destination cannot be reached through our residential network, so the page was fetched from our own address in metadata["egressCountry"] instead of the country you asked for; prices and availability may differ from what a visitor there sees. search_challenge_unsolved — a search engine's results page came back as its "verify you are human" check rather than results; use oc.search(...) (POST /v1/search) for search results.

• oc.crawl.job(id) — the same crawl by id, with status(), results(), cancel(), and oc.crawl.get(id) / .results(id) / .cancel(id) for an id on its own. A streaming crawl belongs to its connection, so hanging up cancels it: the row settles cancelled with every page it had already delivered, and these routes are how you read the pages you paid for. The id is on handle.job["id"] from the first progress frame — store it before you need it.

• oc.agent(task=..., caps=...) — caps is required at runtime and caps.budget is required inside it; caps.steps and caps.duration are optional and unset unless you name them, and all three are hard stops when present. The job is awaitable, async-iterable, and carries everything a handle does: status(), results(), events(), cancel(), control(), answer(), add_file(). oc.agent.get(id) and friends reach a run by id — an id from a webhook needs no submit. schema accepts a JSON Schema object and constrains the run's data to it, the same typed submit @outcrawl/sdk documents.

• oc.profiles / oc.secrets / oc.rules / oc.integrations / oc.sessions / oc.monitors. oc.secrets has deliberately no get: there is no route to widen. A run reaches a value only by naming its HANDLE in the submit, and the substitution happens below the model. oc.secrets.intent(name=..., kind=...) is for a value a person types into YOUR web page, so it never passes through your server: your app server calls it and gets name, kind, url, token and expiresAt; hand url and token to the page, which sends PUT url with Authorization: Bearer <token> and a body {"value": "..."}. The token is single use, lasts five minutes and writes name and nothing else (creating it, or replacing the value already stored under it). It is a credential: give it to that page only and never log it; the result's repr masks it. A refused PUT answers secret_intent_invalid (401, the token does not open that URL for that name) or secret_intent_spent (410, details.reason used or expired), rebuilt here as SecretIntentInvalidError / SecretIntentSpentError; nothing is stored either way and the fix is a new intent. Minting is capped per account per minute (QuotaExceededError with ceiling == "secret_intents_per_minute"). A value your server already holds goes through oc.secrets.create.

• oc.integrations — MCP servers a run may call outside the browser. Connecting is not granting: a run reaches a connector only when its submit names it (oc.agent(integrations=["linear"])).

  verdict = await oc.integrations.test(
      url="https://mcp.linear.app/mcp", auth={"kind": "bearer", "secret": "linear_key"}
  )
  if not verdict["ok"]:
      print(verdict["reason"], verdict["message"])  # not_mcp | unauthorized | unreachable | missing_secret
  

  test(url=, auth=) checks a server and stores nothing; a check that ran always answers, and ok says how it went. connect(...) runs the same check first and raises IntegrationCheckFailedError (422, .reason from the same four) with nothing stored. update(id, tools=..., auth={"secret": ...}) changes the tool pin or the secret handle in place; tools=None means every tool the server offers, and a new secret is checked the same way before it replaces the old one. list() rows carry lastUsedAt, when a run last called the connector successfully (None if never). Each call a run makes appears in its event log as a record with kind: "integration" whose value is {id, name, tool, ok, ms}, or {id, name, status: "unreachable", reason, message} for a granted connector the run could not use; both are at the default step level. No tool arguments or results are logged.

• oc.usage(**query) and oc.credits() — every credit figure is a decimal string; use decimal.Decimal, never float().

• CAPABILITY_AVAILABILITY — whether anything is SERVED behind a declared route. A capability nothing serves is refused in your own process, with what is missing, rather than as a 503.

Errors

Identical to the TypeScript SDK's: same classes, same fields, same sentences. Branch on error.code, never on str(error).

from outcrawl import ProfileInUseError, QuotaExceededError

try:
    ...
except ProfileInUseError as leased:
    retry_at = leased.held_until          # a retry that knows when to retry
except QuotaExceededError as over:
    await asyncio.sleep(over.retry_after_seconds)

tests/error_parity.json is emitted from the TypeScript SDK's own error reconstruction and read by both test suites, so neither surface can drift from the other without turning its own suite red.

What is not here

oc.browser(). The live browser needs a CDP connection driven in the caller's own process, which is why it is deliberately not a registry capability and why the TypeScript SDK is the one place it exists. The same page capabilities are reachable through agent and scrape, which run the loop on our side. That is a difference in kind, not an omission.

The same reason takes Session.take_control with it: the TypeScript SDK's takeControl() opens a CDP connection to a running session's own page so a person can drive it directly, and that connection has to live in the caller's process for the same reason oc.browser() does. Everything else on a session — get, list, export, live() — is a row read or a held stream and is here.

Development

pip install -e '.[dev]'                  # pytest + pytest-asyncio, never runtime dependencies
python3 scripts/gen_registry.py          # regenerate registry.py from packages/core/src/registry.ts
python3 scripts/gen_registry.py --check  # or just fail if it is stale
bun ../sdk-python/scripts/gen_error_parity.ts   # regenerate the shared error contract
python3 -m pytest tests -q               # the drift, parity and ergonomics tests

From the repo root, npm run test:python runs the staleness check and the suite together on the SUPPORTED FLOOR — uv run --extra dev --python 3.10 pytest — so the gate exercises 3.10 rather than whatever python3 happens to be. Without uv: pip install -e '.[dev]' and python3 -m pytest tests -q.

The two registry-versus-TypeScript tests skip when run from an unpacked sdist, which has no TypeScript to compare against, and run whenever the monorepo is present. The reachability test — the one that fails when a registry row has no method — has no such dependency and always runs.