outcrawl

The Python client for Outcrawl: scrape, crawl, search and run agent browser tasks, on one credit balance. Pure Python — one py3-none-any wheel for every operating system, and no compiler on any of them.

pip install outcrawl

Requires Python 3.10 or newer. One runtime dependency, httpx, which is pure Python, as is everything it pulls.

Usage

import asyncio
from outcrawl import Outcrawl

async def main() -> None:
    async with Outcrawl() as oc:                       # reads OUTCRAWL_API_KEY
        doc = await oc.scrape("https://example.com", formats=["markdown"])
        print(doc.markdown)                            # or doc["markdown"] — it is the wire object
        print(doc.usage["credits"])                    # "1" — what this call was charged

        job = oc.agent(
            task="Report the title of the top story on Hacker News.",
            caps={"budget": "16.00"},                  # the one ceiling a submit must carry; this task
                                                       # cost 5.0–7.7 credits on 2026-10-03 (n=33)
        )
        async for record in job:                       # the event cursor, while it runs
            print(record["kind"], record["value"])     # payload shaped by kind: a step, a finding's text, ...
        run = await job                                # and the settled run
        print(run["status"], run.get("data"))          # a "capped" run keeps any data it found first

asyncio.run(main())

await and async for on one job is one run and not two: they are separate requests against one durable row, so watching a run and having its answer is not a choice.

Configuration

The same two names @outcrawl/sdk, the outcrawl CLI and the Outcrawl MCP server read. Pass Outcrawl(api_key=..., base_url=...) to override them per client.

Surface

Every capability in the registry (50 today) is reachable, and tests/test_registry.py fails when one is not.

• oc.scrape(url, **options) / oc.crawl(url, **options) / oc.search(query, **options) — handle = await oc.crawl(url) answers a handle that is async-iterable and single-pass (async for page in handle). search returns SearchHits: when one search engine refuses the query and another answers, hits.warnings carries an engines_refused entry naming the engine that refused; when every engine refuses it raises SearchRefusedError (503, retryable after retry_after_seconds, a retry is served from another exit, nothing charged) rather than returning [].

• Options are keyword arguments with the API's own names, sent as given; help(oc.scrape) and help(oc.crawl) list every one with its default. maxAge is how old a cached answer may be, in milliseconds, default 172800000 (2 days); maxAge=0 is always live, neither reading nor writing the cache. A hit bills the same page_load credit a live fetch does.

• doc.usage["credits"] is what the call was charged, as an exact decimal string ("1", "4.5") — the same figure the ledger bills; read it with decimal.Decimal, never float. A scrape states it (a cache hit included), a search states the results plus the hits it rendered, and a crawl states it on every page and for the whole crawl on its closing frame (handle.job["usage"]["credits"] once iteration ends). An agent run states its charge as the run's own credits.

• Every scrape result and crawl page says where it came from: doc.cacheState is "hit", "miss" or "bypass", doc.cachedAt is when a hit was captured, doc.usage["cache"]["status"] agrees with it (HIT / MISS / DISABLED), doc.metadata["servedVia"] is "direct" or "residential", the network path that fetched it, and doc.metadata["egressCountry"] is the country (ISO code) of the address it was fetched from. doc.metadata["exitRetries"] (absent unless it happened) is how many times the page was fetched again from a fresh exit inside your timeoutMs: 1 when the exit's connection never opened, the site answered it with a reduced page, or the exit answered nothing under the page; 2 at most, only when the first connection never opened. Still one page_load. doc.metadata["readFrom"] (absent unless it happened) is "served" when the content was extracted from the page's HTML as the server sent it, before its scripts ran: a markdown/html/json request whose page body already held its main content when it arrived. Set waitFor for the rendered page.

• A degraded page still comes back, and says so in doc["warnings"]: absent on a clean result, otherwise a list of {"code", "format"?, "message"} whose message names the numbers. partial_document — the page was read before its main document finished loading (the parser was still in it, or the response was still arriving or had stopped), so markdown, html and rawHtml are a prefix of the page; retry with a larger timeoutMs or a waitFor, or, when the message says the response stopped arriving (already retried once on a fresh exit when time allowed), retry as is. empty_capture — the requested markdown or html came back empty. blocked_interstitial — a bot-protection page came back instead of the page. challenge_budget_exhausted — a challenge left too little time to finish reading. source_truncated — a JSON or text body arrived cut off. json_no_content, json_empty and json_required_empty describe a json extraction: no page text, no data, or empty values for fields your schema requires. egress_country_mismatch — the page is not the one a visitor in the country you asked for sees: either this destination cannot be reached through our residential network, so the page was fetched from our own address in metadata["egressCountry"], or the site placed our exit in another country, which the message names, also after one fresh exit; prices and availability may differ. search_challenge_unsolved — a search engine's results page came back as its "verify you are human" check rather than results; use oc.search(...) (POST /v1/search) for search results. degraded_document — the URL names one item (today an Amazon product page) and the site answered with a reduced page without it (recommendations and footer only); it is fetched again, then once from a fresh exit, when timeoutMs allows, so a result still carrying it is a failed fetch. login_redirect — the site redirected the request to its sign-in page and that page came back instead (the message names where); treat it as a failed fetch. render_incomplete — the page was rendered and was not whole when read (its loads were still running when the render budget ran out, or its text changed on the confirming re-read); metadata.complete is false. robots_unchecked — a crawl page from a host whose robots.txt could not be read (5xx or timeout); the crawl went on as if the site had no rules.

• oc.crawl.job(id) — the same crawl by id, with status(), results(), cancel(), and oc.crawl.get(id) / .results(id) / .cancel(id) for an id on its own. A streaming crawl belongs to its connection, so hanging up cancels it: the row settles cancelled with every page it had already delivered, and these routes are how you read the pages you paid for. The id is on handle.job["id"] from the first progress frame — store it before you need it.

• Once a crawl's iteration ends, handle.failures lists the URLs it could not fetch (a message starting site_refused or site_rate_limited with the site's statusCode, robots_disallowed, or the network error) and handle.warnings carries crawl_incomplete when it settled completed short of its limit because pages failed (message names delivered/limit, counts the causes). Both are [] otherwise; oc.crawl.get(id) answers the same warnings.

• oc.agent(task=..., caps=...) — caps is required at runtime and caps.budget is required inside it; caps.steps and caps.duration are optional and unset unless you name them, and all three are hard stops when present. The job is awaitable, async-iterable, and carries everything a handle does: status(), results(), events(), cancel(), control(), answer(), add_file(). oc.agent.get(id) and friends reach a run by id — an id from a webhook needs no submit. schema accepts a JSON Schema object and constrains the run's data to it, the same typed submit @outcrawl/sdk documents.

• Downloads: every file the browser downloaded, for an agent run or a /connect session, is kept 7 days, up to 200 MB each. await job.downloads() (or oc.agent.downloads(run_id)) and oc.sessions.downloads(session_id) list them a page at a time (cursor=, limit= 1 to 1000), each a Download with name, contentType, bytes, sourceUrl, createdAt, expiresAt and url. The url needs no API key and works for one hour; list again for a fresh one. await save_download(download, directory) streams it into a directory that exists and returns the path. It never overwrites: a taken name is saved as <name>-<first 8 of id><ext>. A transfer that ends short is deleted and raises OSError.

  from outcrawl import save_download
  
  for download in await job.downloads():
      path = await save_download(download, "./downloads")
  

• oc.profiles / oc.secrets / oc.rules / oc.integrations / oc.sessions / oc.monitors, and oc.purchases / oc.wallet (see Wallet and purchases).

• Session reads, for a /connect session or an agent run (scrape, crawl and search record nothing): oc.sessions.events(id, kind=..., tab=..., after=..., follow=...) and oc.sessions.network(id, url=..., status="5xx", min_duration_ms=..., bodies=True, follow=...) are async iterators whose .source says schema (v1/v2) and the gaps an older recording never carried; oc.sessions.network.digest(id), oc.sessions.request(id, request_id), oc.sessions.console(id), oc.sessions.har(id), oc.sessions.videos(id) and oc.sessions.share(id, visibility="private" | "link" | "people", people=[...]) (private to your team by default; private revokes and rotates the link). oc.agent.list(status=..., since=...) lists your runs. A session read or share refused for the key raises ReplayAccessDeniedError (replay_access_denied, 403, details["reason"]): operator_key means the key was issued by Outcrawl staff, and such keys never open a customer's recordings; use a key made in the console. oc.secrets has deliberately no get: there is no route to widen. A run reaches a value only by naming its HANDLE in the submit, and the substitution happens below the model. A card secret holds the number, expiry and optional name and billing address, as one CardValue object:

  row = await oc.secrets.create(handle="card", kind="card", value={
      "number": "4242 4242 4242 4242", "expMonth": 12, "expYear": 2028,
      "name": "Jane Doe", "billing": {"postalCode": "94105", "country": "US"},  # both optional
  })
  row["hint"], row["brand"], row["fields"], row["missing"]  # "•••• 4242", "visa", [...], []
  

  billing is inline (CardBillingAddress: postalCode, and optionally country, line1, line2, city, state) or {"ref": "<details handle>"} (CardBillingRef). The security code is never stored: there is no field for it, and a create carrying one raises CvcNotStorableError (cvc_not_storable, 400, details["field"] naming the key) with nothing stored. Supply it for one run instead, as a temporary secret (see Wallet and purchases). The answer, and every oc.secrets.list() row, carries hint (last four only), brand (the card network), fields (the parts stored: number, exp, name, billing) and missing (the required parts that are not, [] when complete). A plain string is still accepted and stored as the number alone; it answers missing == ["exp"]. Any other refused part raises BadRequestError with details["field"] naming it (value.expMonth, ...) and never the value. A run fills each part with {{secret:card}} (the number, as before), {{secret:card.number}}, {{secret:card.exp_month}} (MM), {{secret:card.exp_year}} (YYYY), {{secret:card.exp}} (MM/YY), {{secret:card.name}}, {{secret:card.postal_code}} or {{secret:card.country}}.

  A details secret holds what a checkout asks for besides the card, as one DetailsValue object (every part optional, at least one): name, email, phone (E.164), company, taxId, billingAddress and shippingAddress (each a DetailsAddress: line1, city, postalCode, country as ISO 3166-1 alpha-2, and optionally line2, state). Any create takes optional tags (at most 16, a-z0-9_-), answered back on the row:

  await oc.secrets.create(handle="details_hq", kind="details", value={
      "email": "ops@example.com", "phone": "+14155550100",
      "billingAddress": {"line1": "1 Market St", "city": "San Francisco",
                         "postalCode": "94105", "country": "US"},
  }, tags=["team-ops"])
  

  Its row's fields lists the parts stored (hint, brand and missing are None); a run fills them one at a time, never whole: {{secret:details_hq.email}}, .name, .phone, .company, .tax_id, .billing.postal_code, .shipping.country and so on (line1, line2, city, state, postal_code, country). A card can bill to it with "billing": {"ref": "details_hq"}.

  oc.secrets.list(kind="details", tag="team-ops", handle="details_*", limit=..., cursor=...) answers one page; async for row in oc.secrets.each(kind="card") walks every page by cursor. oc.secrets.batch(items=[...], on_conflict="skip" | "replace") stores up to 500 create bodies in one call and answers {"results": [...]}, one per item in request order: {"handle", "ok": True, "id", "outcome"} (created, replaced or skipped) or {"handle", "ok": False, "code", "field", "message"} (never a value). One bad item never fails the batch, and re-sending with skip (the default) is safe: stored handles answer skipped with their id.

  oc.secrets.intent(name=..., kind=...) is for a value a person types into YOUR web page, so it never passes through your server: your app server calls it and gets name, kind, url, token and expiresAt; hand url and token to the page, which sends PUT url with Authorization: Bearer <token> and a body {"value": "..."}. The token is single use, lasts five minutes and writes name and nothing else (creating it, or replacing the value already stored under it). It is a credential: give it to that page only and never log it; the result's repr masks it. A refused PUT answers secret_intent_invalid (401, the token does not open that URL for that name) or secret_intent_spent (410, details.reason used or expired), rebuilt here as SecretIntentInvalidError / SecretIntentSpentError; nothing is stored either way and the fix is a new intent. Minting is capped per account per minute (QuotaExceededError with ceiling == "secret_intents_per_minute"). A value your server already holds goes through oc.secrets.create.

• oc.integrations — MCP servers a run may call outside the browser. Connecting is not granting: a run reaches a connector only when its submit names it (oc.agent(integrations=["linear"])).

  verdict = await oc.integrations.test(
      url="https://mcp.linear.app/mcp", auth={"kind": "bearer", "secret": "linear_key"}
  )
  if not verdict["ok"]:
      print(verdict["reason"], verdict["message"])  # not_mcp | unauthorized | unreachable | missing_secret
  

  test(url=, auth=) checks a server and stores nothing; a check that ran always answers, and ok says how it went. connect(...) runs the same check first and raises IntegrationCheckFailedError (422, .reason from the same four) with nothing stored. update(id, tools=..., auth={"secret": ...}) changes the tool pin or the secret handle in place; tools=None means every tool the server offers, and a new secret is checked the same way before it replaces the old one. list() rows carry lastUsedAt, when a run last called the connector successfully (None if never). Each call a run makes appears in its event log as a record with kind: "integration" whose value is {id, name, tool, ok, ms}, or {id, name, status: "unreachable", reason, message} for a granted connector the run could not use; both are at the default step level. No tool arguments or results are logged.

• oc.usage(**query) and oc.credits() — every credit figure is a decimal string; use decimal.Decimal, never float().

• CAPABILITY_AVAILABILITY — whether anything is SERVED behind a declared route. A capability nothing serves is refused in your own process, with what is missing, rather than as a 503.

Wallet and purchases

A run can pay for something with a card you stored, inside limits it cannot raise. Four fields on oc.agent(...) do it, all checked here in the API's own words before anything is sent:

job = oc.agent(
    task="Buy one box of A4 copy paper from staples.com, delivered to the office.",
    caps={
        "budget": "40",                                   # credits, as always
        "spend": {"limit": "60.00", "currency": "USD", "perPurchase": "60.00"},
    },
    merchants=["staples.com"],                            # where it may pay
    secrets=["card_7731", "details_hq"],                  # the grant
    wallet={"card": "card_7731", "details": "details_hq"},  # which granted handles to pay with
)

• caps.spend is purchase money, separate from the credits budget: limit for the whole run, an ISO 4217 currency in capitals, and an optional perPurchase no larger than limit. Amounts are exact decimal strings at the currency's precision ("60.00", "1500" in JPY); a float or a Decimal is refused. Raising a cap means a new run.
• merchants is required with caps.spend and refused without it: up to 100 registrable domains ("staples.com", "amazon.co.uk"), and a subdomain of one matches. The merchant of a purchase is the site of the tab's top-level page, read by the runtime, never a name the model gives.
• wallet.card and wallet.details name the default card and payment-details handles. Each must also be granted in secrets; naming a handle the run is not granted is an InvalidAgentRequestError. The card itself is a kind="card" secret stored once with oc.secrets.create (above); a run only ever holds its handle.
• A card is revealed only into an open purchase: before filling a card the run opens one with the checkout's amount, and it is refused unless the merchant is on the list, the amount is within perPurchase, and spent plus reserved plus the amount stays within limit. Outside those the step fails with merchant_not_allowed, spend_cap_exceeded or per_purchase_exceeded (the codes SpendRefusedError carries) and no card detail is revealed.
• Honest limit: for a stored card the purchase amount is read from the checkout page by the run. Issuer-side limits, the hard backstop beneath it, arrive with connector cards; until then caps.spend bounds the amounts the run read and declared, not what a merchant finally charges.

Two refusals happen in your own process, with no request sent:

• CardNumberInTaskError (card_number_in_task, details["field"] is task, rules or answer): a Luhn-valid card number in the task, a rule or an answer. The message is fixed and never echoes a digit; put the card in a secret and name its handle.
• SpendCapRequiredError (spend_cap_required, details["handle"]): wallet.card names a card but there is no caps.spend. The API refuses a granted stored card with no cap the same way, as only it knows each handle's kind.

What a run bought, and what it is holding:

run = await job.status()
run["spend"]                                   # {"limit", "spent", "reserved", "currency"}
page = await job.purchases()                   # this run's, also oc.agent.purchases(run_id)
page = await oc.purchases.list(                # every run's, newest first
    card="card_7731", merchant="staples.com", status="completed", limit=100,
)
for purchase in page:
    print(purchase["merchant"], purchase["amount"], purchase["currency"], purchase["status"])

A purchase carries merchant, amount and currency (parse with decimal.Decimal, never float), card as {source, handle, provider, brand, last4, providerCardId} (source is stored, run or provider; provider and providerCardId are set only for a provider's card), status (reserved, completed, declined, abandoned, unknown), orderRef, createdAt and settledAt. oc.purchases.list also takes since= and until= on createdAt, and pages with cursor= (limit 1 to 500, default 50).

Your own card provider. Connect a Privacy.com, Brex, Slash, Stripe Issuing or Airwallex account once, and a run that names the connection gets ONE card minted from it at its first purchase, limited to caps.spend and closed when the run ends (Privacy: one merchant-locked card per site the run buys at). The provider's read-only test runs before anything is stored, and no route ever returns the credential.

providers = await oc.wallet.providers.list()   # fields (secret ones marked), currencies, available
await oc.wallet.connections.create(
    provider="privacy", handle="ops_privacy", credential={"apiKey": os.environ["PRIVACY_API_KEY"]},
)
job = oc.agent(
    task="Buy one box of A4 copy paper from staples.com.",
    caps={"budget": "40", "spend": {"limit": "60.00", "currency": "USD"}},
    merchants=["staples.com"],
    wallet={"provider": "ops_privacy"},
)

A refused credential raises ProviderTestFailedError (provider_test_failed, 422, details["provider"]), with nothing stored. oc.wallet.connections.list(provider=..., cursor=..., limit=...), .each() and .get(id) read connections with their status (pending, active, failing, revoked) and warnings; .test(id) runs the test again and answers {ok, account, warnings, reason}; .delete(id) closes every card it still has open, then deletes it. .intent(provider=..., handle=..., origin=..., ttl_s=...) lets a browser send the credential instead (its repr masks token; after five failed tests the token is spent with SecretIntentSpentError.reason == "attempts").

Approvals. oc.wallet.settings holds one workspace setting, approvalAbove:

await oc.wallet.settings.set(approval_above={"amount": "100.00", "currency": "USD"})
await oc.wallet.settings.set(approval_above=None)          # off, the default
(await oc.wallet.settings.get())["approvalAbove"]

With it set, a purchase within every limit and above the threshold (same currency) pauses the run with run["pause"]["ask"]["kind"] == "approval", carrying purchase, merchant, amount and currency. Answer it with the decision, never with text:

run = await job.status()
if run["status"] == "paused" and run["pause"].get("ask", {}).get("kind") == "approval":
    ask = run["pause"]["ask"]
    print(ask["merchant"], ask["amount"], ask["currency"])  # show the person, take their decision
    await job.answer(approve=True)                     # or approve=False; oc.agent.answer(run_id, approve=...)

Send exactly one of answer and approve. approve=False fails the step with approval_denied and no answer by the run's ask timeout with approval_timed_out; both record the purchase as abandoned. An approval never lifts a purchase past caps.spend or merchants. A run waiting on a secret ask is never answered here: the API refuses with SecretInAnswerError (secret_in_answer) and the value is neither stored nor echoed.

Temporary secrets. A run can also hold a secret only for itself: held in the memory of the machine running it, never stored (even encrypted), dropped when the run ends; at most 16 per run, 4 KB each. One carries a security code, a whole card, a billing profile or a text value, on three paths: runSecrets= on the submit, oc.agent.secrets.add on a live run, and oc.agent.secrets.intent for a browser. Kinds are text; card ({number, expMonth, expYear, name?, billing?, cvc?}, the one place a card may carry its security code); details (a DetailsValue, checked as a stored details item is and filled by part, such as {{secret:billing_tmp.billing.line1}}); and, on a live add or an intent, card_field with field="cvc", which adds the code to a card the run already holds. wallet may name a run secret, a run secret shadows a stored handle of the same name, and a run card is revealed only into an open purchase, as a stored one is.

job = oc.agent(
    task="Buy one box of A4 copy paper from staples.com.",
    caps={"budget": "40", "spend": {"limit": "60.00", "currency": "USD"}},
    merchants=["staples.com"],
    runSecrets={
        "card_tmp": {"kind": "card", "singleUse": True, "value": {
            "number": "4242424242424242", "expMonth": 12, "expYear": 2030, "cvc": "123"}},
        "billing_tmp": {"kind": "details", "value": {
            "email": "ops@example.com",
            "billingAddress": {"line1": "1 Market St", "city": "San Francisco",
                               "postalCode": "94105", "country": "US"}}},
    },
    wallet={"card": "card_tmp", "details": "billing_tmp"},
)
run_id = (await job.submitted)["id"]

# Live add: a billing profile, or a stored card's security code, for a run already going.
await oc.agent.secrets.add(run_id, name="ship_tmp", kind="details", value={
    "name": "Jane Doe", "shippingAddress": {"line1": "1 Market St", "city": "San Francisco",
                                            "postalCode": "94105", "country": "US"}})
await oc.agent.secrets.add(run_id, name="card_7731", kind="card_field", field="cvc", value="123")

# Browser intent: mint it here for your checkout page's origin; that page then sends
# PUT uploadUrl with Authorization: Bearer <token> and {"value": {"number": "4242424242424242",
# "expMonth": 12, "expYear": 2030, "cvc": "123"}}.
intent = await oc.agent.secrets.intent(
    run_id, name="card_typed", kind="card", origin="https://checkout.example.com"
)

origin is the exact origin of the page that sends the value: https://host[:port], or http://localhost:<port> / http://127.0.0.1:<port> while developing. Only that page may send it; without origin, only the Outcrawl console may. The PUT body's value is a string for text and card_field and an object for card and details. A refused value answers 400 naming the field, never the value, and leaves the token unspent. The token is single use, lasts at most five minutes and dies with the run. A value your server already holds goes through oc.agent.secrets.add.

Errors

Identical to the TypeScript SDK's: same classes, same fields, same sentences. Branch on error.code, never on str(error).

from outcrawl import ProfileInUseError, QuotaExceededError

try:
    ...
except ProfileInUseError as leased:
    retry_at = leased.held_until          # a retry that knows when to retry
except QuotaExceededError as over:
    await asyncio.sleep(over.retry_after_seconds)

tests/error_parity.json is emitted from the TypeScript SDK's own error reconstruction and read by both test suites, so neither surface can drift from the other without turning its own suite red.

What is not here

oc.browser(). The live browser needs a CDP connection driven in the caller's own process, which is why it is deliberately not a registry capability and why the TypeScript SDK is the one place it exists. The same page capabilities are reachable through agent and scrape, which run the loop on our side. That is a difference in kind, not an omission.

The same reason takes Session.take_control with it: the TypeScript SDK's takeControl() opens a CDP connection to a running session's own page so a person can drive it directly, and that connection has to live in the caller's process for the same reason oc.browser() does. Everything else on a session — get, list, export, live() — is a row read or a held stream and is here.

Development

pip install -e '.[dev]'                  # pytest + pytest-asyncio, never runtime dependencies
python3 scripts/gen_registry.py          # regenerate registry.py from packages/core/src/registry.ts
python3 scripts/gen_registry.py --check  # or just fail if it is stale
bun ../sdk-python/scripts/gen_error_parity.ts   # regenerate the shared error contract
python3 -m pytest tests -q               # the drift, parity and ergonomics tests

From the repo root, npm run test:python runs the staleness check and the suite together on the SUPPORTED FLOOR — uv run --extra dev --python 3.10 pytest — so the gate exercises 3.10 rather than whatever python3 happens to be. Without uv: pip install -e '.[dev]' and python3 -m pytest tests -q.

The two registry-versus-TypeScript tests skip when run from an unpacked sdist, which has no TypeScript to compare against, and run whenever the monorepo is present. The reachability test — the one that fails when a registry row has no method — has no such dependency and always runs.