outcrawl

The Python client for Outcrawl: scrape, crawl, search and run agent browser tasks, on one credit balance. Pure Python — one py3-none-any wheel for every operating system, and no compiler on any of them.

pip install outcrawl

Requires Python 3.10 or newer. One runtime dependency, httpx, which is pure Python, as is everything it pulls.

Usage

import asyncio
from outcrawl import Outcrawl

async def main() -> None:
    async with Outcrawl() as oc:                       # reads OUTCRAWL_API_KEY
        doc = await oc.scrape("https://example.com", formats=["markdown"])
        print(doc.markdown)                            # or doc["markdown"] — it is the wire object
        print(doc.usage["credits"])                    # "1" — what this call was charged

        job = oc.agent(
            task="Report the title of the top story on Hacker News.",
            caps={"budget": "16.00"},                  # the one ceiling a submit must carry; this task
                                                       # cost 5.0–7.7 credits on 2026-10-03 (n=33)
        )
        async for record in job:                       # the event cursor, while it runs
            print(record["kind"], record["value"])     # payload shaped by kind: a step, a finding's text, ...
        run = await job                                # and the settled run
        print(run["status"], run.get("data"))          # a "capped" run keeps any data it found first

asyncio.run(main())

await and async for on one job is one run and not two: they are separate requests against one durable row, so watching a run and having its answer is not a choice.

Configuration

The same two names @outcrawl/sdk, the outcrawl CLI and the Outcrawl MCP server read. Pass Outcrawl(api_key=..., base_url=...) to override them per client.

Surface

Every capability in the registry (46 today) is reachable, and tests/test_registry.py fails when one is not.

• oc.scrape(url, **options) / oc.crawl(url, **options) / oc.search(query, **options) — handle = await oc.crawl(url) answers a handle that is async-iterable and single-pass (async for page in handle). search returns SearchHits: when one search engine refuses the query and another answers, hits.warnings carries an engines_refused entry naming the engine that refused; when every engine refuses it raises SearchRefusedError (503, retryable after retry_after_seconds, a retry is served from another exit, nothing charged) rather than returning [].

• Options are keyword arguments with the API's own names, sent as given; help(oc.scrape) and help(oc.crawl) list every one with its default. maxAge is how old a cached answer may be, in milliseconds, default 172800000 (2 days); maxAge=0 is always live, neither reading nor writing the cache. A hit bills the same page_load credit a live fetch does.

• doc.usage["credits"] is what the call was charged, as an exact decimal string ("1", "4.5") — the same figure the ledger bills; read it with decimal.Decimal, never float. A scrape states it (a cache hit included), a search states the results plus the hits it rendered, and a crawl states it on every page and for the whole crawl on its closing frame (handle.job["usage"]["credits"] once iteration ends). An agent run states its charge as the run's own credits.

• Every scrape result and crawl page says where it came from: doc.cacheState is "hit", "miss" or "bypass", doc.cachedAt is when a hit was captured, doc.usage["cache"]["status"] agrees with it (HIT / MISS / DISABLED), doc.metadata["servedVia"] is "direct" or "residential", the network path that fetched it, and doc.metadata["egressCountry"] is the country (ISO code) of the address it was fetched from. doc.metadata["exitRetries"] (absent unless it happened) is how many times the page was fetched again from a fresh exit inside your timeoutMs: 1 when the exit's connection never opened, the site answered it with a reduced page, or the exit answered nothing under the page; 2 at most, only when the first connection never opened. Still one page_load. doc.metadata["readFrom"] (absent unless it happened) is "served" when the content was extracted from the page's HTML as the server sent it, before its scripts ran: a markdown/html/json request whose page body already held its main content when it arrived. Set waitFor for the rendered page.

• A degraded page still comes back, and says so in doc["warnings"]: absent on a clean result, otherwise a list of {"code", "format"?, "message"} whose message names the numbers. partial_document — the page was read before its main document finished loading (the parser was still in it, or the response was still arriving or had stopped), so markdown, html and rawHtml are a prefix of the page; retry with a larger timeoutMs or a waitFor, or, when the message says the response stopped arriving (already retried once on a fresh exit when time allowed), retry as is. empty_capture — the requested markdown or html came back empty. blocked_interstitial — a bot-protection page came back instead of the page. challenge_budget_exhausted — a challenge left too little time to finish reading. source_truncated — a JSON or text body arrived cut off. json_no_content, json_empty and json_required_empty describe a json extraction: no page text, no data, or empty values for fields your schema requires. egress_country_mismatch — the page is not the one a visitor in the country you asked for sees: either this destination cannot be reached through our residential network, so the page was fetched from our own address in metadata["egressCountry"], or the site placed our exit in another country, which the message names, also after one fresh exit; prices and availability may differ. search_challenge_unsolved — a search engine's results page came back as its "verify you are human" check rather than results; use oc.search(...) (POST /v1/search) for search results. degraded_document — the URL names one item (today an Amazon product page) and the site answered with a reduced page without it (recommendations and footer only); it is fetched again, then once from a fresh exit, when timeoutMs allows, so a result still carrying it is a failed fetch. login_redirect — the site redirected the request to its sign-in page and that page came back instead (the message names where); treat it as a failed fetch. render_incomplete — the page was rendered and was not whole when read (its loads were still running when the render budget ran out, or its text changed on the confirming re-read); metadata.complete is false. robots_unchecked — a crawl page from a host whose robots.txt could not be read (5xx or timeout); the crawl went on as if the site had no rules.

• oc.crawl.job(id) — the same crawl by id, with status(), results(), cancel(), and oc.crawl.get(id) / .results(id) / .cancel(id) for an id on its own. A streaming crawl belongs to its connection, so hanging up cancels it: the row settles cancelled with every page it had already delivered, and these routes are how you read the pages you paid for. The id is on handle.job["id"] from the first progress frame — store it before you need it.

• Once a crawl's iteration ends, handle.failures lists the URLs it could not fetch (a message starting site_refused or site_rate_limited with the site's statusCode, robots_disallowed, or the network error) and handle.warnings carries crawl_incomplete when it settled completed short of its limit because pages failed (message names delivered/limit, counts the causes). Both are [] otherwise; oc.crawl.get(id) answers the same warnings.

• oc.agent(task=..., caps=...) — caps is required at runtime and caps.budget is required inside it; caps.steps and caps.duration are optional and unset unless you name them, and all three are hard stops when present. The job is awaitable, async-iterable, and carries everything a handle does: status(), results(), events(), cancel(), control(), answer(), add_file(). oc.agent.get(id) and friends reach a run by id — an id from a webhook needs no submit. schema accepts a JSON Schema object and constrains the run's data to it, the same typed submit @outcrawl/sdk documents.

• oc.profiles / oc.secrets / oc.rules / oc.integrations / oc.sessions / oc.monitors.

• Session reads, for a /connect session or an agent run (scrape, crawl and search record nothing): oc.sessions.events(id, kind=..., tab=..., after=..., follow=...) and oc.sessions.network(id, url=..., status="5xx", min_duration_ms=..., bodies=True, follow=...) are async iterators whose .source says schema (v1/v2) and the gaps an older recording never carried; oc.sessions.network.digest(id), oc.sessions.request(id, request_id), oc.sessions.console(id), oc.sessions.har(id), oc.sessions.videos(id) and oc.sessions.share(id, visibility="private" | "link" | "people", people=[...]) (private to your team by default; private revokes and rotates the link). oc.agent.list(status=..., since=...) lists your runs. A session read or share refused for the key raises ReplayAccessDeniedError (replay_access_denied, 403, details["reason"]): operator_key means the key was issued by Outcrawl staff, and such keys never open a customer's recordings; use a key made in the console. oc.secrets has deliberately no get: there is no route to widen. A run reaches a value only by naming its HANDLE in the submit, and the substitution happens below the model. oc.secrets.intent(name=..., kind=...) is for a value a person types into YOUR web page, so it never passes through your server: your app server calls it and gets name, kind, url, token and expiresAt; hand url and token to the page, which sends PUT url with Authorization: Bearer <token> and a body {"value": "..."}. The token is single use, lasts five minutes and writes name and nothing else (creating it, or replacing the value already stored under it). It is a credential: give it to that page only and never log it; the result's repr masks it. A refused PUT answers secret_intent_invalid (401, the token does not open that URL for that name) or secret_intent_spent (410, details.reason used or expired), rebuilt here as SecretIntentInvalidError / SecretIntentSpentError; nothing is stored either way and the fix is a new intent. Minting is capped per account per minute (QuotaExceededError with ceiling == "secret_intents_per_minute"). A value your server already holds goes through oc.secrets.create.

• oc.integrations — MCP servers a run may call outside the browser. Connecting is not granting: a run reaches a connector only when its submit names it (oc.agent(integrations=["linear"])).

  verdict = await oc.integrations.test(
      url="https://mcp.linear.app/mcp", auth={"kind": "bearer", "secret": "linear_key"}
  )
  if not verdict["ok"]:
      print(verdict["reason"], verdict["message"])  # not_mcp | unauthorized | unreachable | missing_secret
  

  test(url=, auth=) checks a server and stores nothing; a check that ran always answers, and ok says how it went. connect(...) runs the same check first and raises IntegrationCheckFailedError (422, .reason from the same four) with nothing stored. update(id, tools=..., auth={"secret": ...}) changes the tool pin or the secret handle in place; tools=None means every tool the server offers, and a new secret is checked the same way before it replaces the old one. list() rows carry lastUsedAt, when a run last called the connector successfully (None if never). Each call a run makes appears in its event log as a record with kind: "integration" whose value is {id, name, tool, ok, ms}, or {id, name, status: "unreachable", reason, message} for a granted connector the run could not use; both are at the default step level. No tool arguments or results are logged.

• oc.usage(**query) and oc.credits() — every credit figure is a decimal string; use decimal.Decimal, never float().

• CAPABILITY_AVAILABILITY — whether anything is SERVED behind a declared route. A capability nothing serves is refused in your own process, with what is missing, rather than as a 503.

Errors

Identical to the TypeScript SDK's: same classes, same fields, same sentences. Branch on error.code, never on str(error).

from outcrawl import ProfileInUseError, QuotaExceededError

try:
    ...
except ProfileInUseError as leased:
    retry_at = leased.held_until          # a retry that knows when to retry
except QuotaExceededError as over:
    await asyncio.sleep(over.retry_after_seconds)

tests/error_parity.json is emitted from the TypeScript SDK's own error reconstruction and read by both test suites, so neither surface can drift from the other without turning its own suite red.

What is not here

oc.browser(). The live browser needs a CDP connection driven in the caller's own process, which is why it is deliberately not a registry capability and why the TypeScript SDK is the one place it exists. The same page capabilities are reachable through agent and scrape, which run the loop on our side. That is a difference in kind, not an omission.

The same reason takes Session.take_control with it: the TypeScript SDK's takeControl() opens a CDP connection to a running session's own page so a person can drive it directly, and that connection has to live in the caller's process for the same reason oc.browser() does. Everything else on a session — get, list, export, live() — is a row read or a held stream and is here.

Development

pip install -e '.[dev]'                  # pytest + pytest-asyncio, never runtime dependencies
python3 scripts/gen_registry.py          # regenerate registry.py from packages/core/src/registry.ts
python3 scripts/gen_registry.py --check  # or just fail if it is stale
bun ../sdk-python/scripts/gen_error_parity.ts   # regenerate the shared error contract
python3 -m pytest tests -q               # the drift, parity and ergonomics tests

From the repo root, npm run test:python runs the staleness check and the suite together on the SUPPORTED FLOOR — uv run --extra dev --python 3.10 pytest — so the gate exercises 3.10 rather than whatever python3 happens to be. Without uv: pip install -e '.[dev]' and python3 -m pytest tests -q.

The two registry-versus-TypeScript tests skip when run from an unpacked sdist, which has no TypeScript to compare against, and run whenever the monorepo is present. The reachability test — the one that fails when a registry row has no method — has no such dependency and always runs.