azcrawlerpy

A framework for navigating and filling multi-step web forms programmatically. Supports Camoufox anti-detect browser (built on Firefox with C++ level fingerprint spoofing) and standard Chromium via Playwright. Uses JSON instruction files to define form navigation workflows, making it ideal for automated form submission, web scraping, and AI agent-driven web interactions.

Table of Contents

• Installation
• Quick Start
• Logging & Telemetry
• Core Concepts
• Instructions Schema
  • Top-Level Structure
  • Browser Configuration
  • URL Rewrites
  • Cookie Consent Handling
  • Step Definitions
  • Field Types
  • Optional Fields
  • Action Types
  • Final Page Configuration
  • Post Final Page Steps
  • Per-Step Page Extraction
  • Data Extraction Configuration
• Data Points (input_data)
• Element Discovery
• AI Agent Guidance
• Retry and Resilience
• Error Handling and Diagnostics
• Browser Profile Building (Profiler)
• Examples

Installation

Requires Python 3.11–3.13.

uv add azcrawlerpy

Or install from source:

uv pip install -e .

Quick Start

FormCrawler requires a logger. In production, pass an AzureLogger from azpaddypy so crawl telemetry (spans, correlation IDs, structured records) flows into Application Insights. For local scripts and notebooks, any stdlib-shaped logger works — anything that implements debug/info/warning/error/exception/critical.

import asyncio

from azcrawlerpy import CrawlerBrowserConfig, DebugMode, FormCrawler, HumanizeConfig
from azpaddypy.mgmt.logging import AzureLogger, bootstrap_azure_monitor


async def main():
    # 1. Wire telemetry once per process (idempotent; no-op without a
    #    connection string so it's safe in local runs).
    bootstrap_azure_monitor(service_name="crawler-demo")
    logger = AzureLogger(__name__)

    # 2. CrawlerBrowserConfig controls runtime browser settings
    #    (proxy, stealth, humanize).
    browser_config = CrawlerBrowserConfig(humanize=HumanizeConfig(enabled=True))

    # 3. FormCrawler now requires a logger -- pass AzureLogger for App Insights
    #    integration, or a stdlib `logging.getLogger(__name__)` for local work.
    crawler = FormCrawler(
        headless=True,
        browser_config=browser_config,
        logger=logger,
    )

    instructions = {
        "url": "https://example.com/form",
        "browser_config": {
            "browser_type": "camoufox",
            "viewport_width": 1920,
            "viewport_height": 1080
        },
        "steps": [
            {
                "name": "step_1",
                "wait_for": "input[name='email']",
                "timeout_ms": 15000,
                "fields": [
                    {
                        "type": "text",
                        "selector": "input[name='email']",
                        "data_key": "email"
                    }
                ],
                "next_action": {
                    "type": "click",
                    "selector": "button[type='submit']"
                }
            }
        ],
        "final_page": {
            "wait_for": ".success-message",
            "timeout_ms": 60000
        }
    }

    input_data = {
        "email": "user@example.com"
    }

    result = await crawler.crawl(
        url=instructions["url"],
        input_data=input_data,
        instructions=instructions,
        debug_mode=DebugMode.ALL,
        # Optional wall-clock cap on the entire crawl. Exceeding it raises
        # CrawlerTimeoutError with a partial result attached. Omit or set
        # to None to disable.
        global_timeout_ms=300_000,
    )

    # The crawler returns every artifact in memory -- the caller owns the
    # sink (disk, Azure Blob, stdout, ...). Nothing is written to disk on
    # your behalf.
    print(f"Final URL: {result.final_url}")
    print(f"Steps completed: {result.steps_completed}")
    print(f"Screenshots captured: {len(result.screenshots)}")
    print(f"HTML bytes: {len(result.html)}")
    print(f"Extracted data: {result.extracted_data}")

asyncio.run(main())

Logging & Telemetry

FormCrawler accepts any logger that matches the stdlib logging.Logger shape (structural LoggerLike protocol in crawling/utils.py): debug, info, warning, error, exception, critical.

Production: AzureLogger

Pairing the crawler with AzureLogger from azpaddypy gives you, with no code in the crawler itself:

• Application Insights export of all crawl log records via the OTel handler that azpaddypy attaches to the root logger; the crawler's module-level loggers propagate to root automatically.
• Trace context (operation_Id / operation_ParentId) is set by the Azure Monitor exporter from the active OTel span — no per-record duplication.
• Built-in OpenTelemetry spans on the crawler's hot paths so the App Insights Application Map and Performance views show structure inside the crawl rather than one opaque blob:
  • FormCrawler.crawl — root span; carries crawl.url, crawl.headless, crawl.debug_mode, crawl.strict, crawl.steps_planned, plus post-success crawl.steps_completed and crawl.profiler_visited_count. On timeout / error it carries outcome=timeout|error with Status.ERROR and the captured exception.
  • FormCrawler.attempt — one span per restart-loop iteration (attempt.index, browser.type); restart paths add attempt.outcome=restart and attempt.restart_step.
  • profiler.run — wraps the profiler phase (profiler.visit_count, profiler.in_memory, profiler.browser_type).
  • ElementDiscovery.discover — wraps page element discovery (discovery.url, discovery.browser_type, discovery.explore_iframes, plus post-run discovery.total_elements and discovery.iframes_count).
• Correlation IDs if the caller sets one via AzureLogger.set_correlation_id(...) — it propagates through the contextvars-backed OTel context into all crawler spans automatically.

from azpaddypy.mgmt.logging import AzureLogger, bootstrap_azure_monitor, trace_function

bootstrap_azure_monitor(service_name="crawler-api", service_version="0.5.1")
logger = AzureLogger(__name__)

# Optional: generate a correlation ID per request + emit a parent span
@trace_function(name="crawl_request")
async def handle_crawl(url: str, input_data: dict, instructions: dict):
    crawler = FormCrawler(headless=True, browser_config=None, logger=logger)
    return await crawler.crawl(
        url=url, input_data=input_data, instructions=instructions,
    )

The crawler does not import opentelemetry directly. Spans are produced through a soft-import shim: when OTel is installed (the typical case in a function-app that pulls in azpaddypy) spans flow to Application Insights; when not, the shim is a zero-cost no-op so standalone CLI runs and minimal test environments keep working unchanged.

Local / scripts: stdlib logger

If you don't need App Insights, a standard library logger is enough:

import logging

logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(name)s: %(message)s")
crawler = FormCrawler(
    headless=True,
    browser_config=None,
    logger=logging.getLogger("my_crawler"),
)

Any logger that satisfies the LoggerLike Protocol is accepted — there is no hard runtime dependency on azpaddypy.

Core Concepts

The framework operates on two primary inputs:

1. Instructions (instructions.json): Defines the form structure, selectors, navigation flow, and field types
2. Data Points (input_data): Contains the actual values to fill into form fields

The crawler processes each step sequentially:

1. Wait for the step's wait_for selector to become visible
2. Fill all fields defined in the step using values from input_data
3. Execute the next_action to navigate to the next step
4. Repeat until all steps are complete
5. Wait for and capture the final page

Instructions Schema

Top-Level Structure

{
  "url": "https://example.com/form",
  "browser_config": { ... },
  "cookie_consent": { ... },
  "steps": [ ... ],
  "final_page": { ... },
  "post_final_page_steps": [ ... ],
  "data_extraction": { ... }
}

Browser Configuration

{
  "browser_config": {
    "browser_type": "camoufox",
    "viewport_width": 1920,
    "viewport_height": 1080
  }
}

Both engines render the configured width exactly. The height differs by engine: chromium sets it as the page's inner height, while camoufox sizes the browser window, leaving the page shorter by the height of the browser chrome. Camoufox ignores a Playwright viewport, so the size goes in through the fingerprint: the generated screen is pinned to the viewport so camoufox cannot shrink the window to a smaller rolled screen, and every camoufox crawl pins a fingerprint preset whose available screen area (the screen less its taskbar) holds the window, and sets the window explicitly.

URL Rewrites

Redirects matching requests to a different host. The main use case is a proxy that cannot reach a third-party host the page depends on, but which has a reachable mirror.

{
  "browser_config": {
    "browser_type": "chromium",
    "viewport_width": 1920,
    "viewport_height": 1080,
    "url_rewrites": {
      "://www.google.com/recaptcha/": "://www.recaptcha.net/recaptcha/"
    }
  }
}

Each key is matched as a substring of the request URL and replaced with its value. Requests are routed through page.route(), so the page's own loader keeps working — including any onload callback it registered on the original script tag.

url_rewrite_mode controls how the replacement is reached:

Prefer redirect — it is a single request. Reach for mirror when a rewritten resource loads same-origin sub-resources: firefox refuses to start a web worker from a different origin than its document, so a redirected script that spawns one fails with a Security Error and never finishes initialising. mirror keeps the origin intact and avoids the check entirely.

Notes:

• With redirect the replacement must keep the same protocol; cross-origin redirects are allowed.
• Rewrites apply for the whole page lifetime, so a rewritten script's own follow-up requests are rewritten too when they match.
• Prefer either mode over an init script that patches URLs in JavaScript. Camoufox runs injected scripts in an isolated world that cannot touch page globals, so a JS-based rewrite silently does nothing there; routing works on both engines and leaves no page-visible trace.

Example — reCAPTCHA behind a proxy whose zone blocks google.com. Google serves reCAPTCHA from recaptcha.net for exactly this situation, so rewriting the host lets the widget load and render normally. On camoufox this needs "url_rewrite_mode": "mirror", because the reCAPTCHA frame spawns a worker from a hardcoded google.com URL. Note this only restores reachability; whether the challenge then passes still depends on the exit IP's reputation.

Cookie Consent Handling

The framework supports two modes for handling cookie consent banners:

Standard Mode (regular DOM elements):

{
  "cookie_consent": {
    "banner_selector": "dialog:has-text('cookies')",
    "accept_selector": "button:has-text('Accept')"
  }
}

Shadow DOM Mode (for Usercentrics, OneTrust, etc.):

{
  "cookie_consent": {
    "banner_selector": "#usercentrics-cmp-ui",
    "shadow_host_selector": "#usercentrics-cmp-ui",
    "accept_button_texts": ["Accept All", "Alle akzeptieren"]
  }
}

The JS fallback matches button text against: klar, akzept, accept, agree, ok, verstanden, einverstanden, zustimm. Set js_fallback_texts to override this list with site-specific patterns. Both js_fallback_texts and accept_button_texts match case-insensitively at the start of a word, so ok matches "OK" and "Okay" but not "Cookie-Einstellungen", and akzept matches "Alle akzeptieren". A text that only occurs inside a word ("verstanden" in "Einverstanden") needs its own entry.

Shadow DOM accept buttons get a real click. The JS fallback, and removing a banner that no accept button dismissed, only run when browser_config.allow_synthetic_events is on.

Step Definitions

Each step represents a form page or section:

{
  "name": "personal_info",
  "wait_for": "input[name='firstName']",
  "timeout_ms": 15000,
  "fields": [ ... ],
  "next_action": { ... }
}

Restart Crawl

Steps marked with "restart_crawl": true raise an internal CrawlRestartError when the step fails: its wait_for selector times out, its timeout_ms budget runs out, or one of its extractions fails (the error message names which). The crawler then closes the browser, recreates the context (reusing any profiler storage state so cookie consent does not re-run), and replays the entire crawl from the start URL. Each step tracks its own restart budget against max_restarts; once exhausted, the original timeout error is raised.

{
  "name": "results_page",
  "wait_for": "[data-cy='quote-result']",
  "timeout_ms": 15000,
  "restart_crawl": true,
  "max_restarts": 2,
  "fields": [],
  "next_action": { "type": "delay", "delay_ms": 0 }
}

restart_crawl takes priority over optional and strict. Use it for steps where the page may render broken or never finalize due to flaky SPA/backend behavior, where a fresh browser context is the most reliable recovery. Prefer retry_config on individual fields/actions for transient interaction failures; reserve restart_crawl for full-page load failures.

Optional Steps

Steps marked with "optional": true are skipped gracefully when the wait_for selector is not found within timeout_ms. The entire step (fields, data extraction, next_action) is skipped with an info log. This is useful for conditional workflow pages that only appear depending on prior input values or dynamic form behavior.

{
  "name": "additional_driver_details",
  "wait_for": "#additional-driver-form",
  "timeout_ms": 5000,
  "optional": true,
  "fields": [ ... ],
  "next_action": { "type": "click", "selector": "#next" }
}

The optional check takes precedence over the strict parameter. When a step is optional and the wait_for selector times out, the step is always skipped regardless of strict mode. Non-optional steps follow the existing behavior: raise CrawlerTimeoutError when strict=True, or log a warning and continue into the step when strict=False.

Tip: Use a shorter timeout_ms (e.g., 3000-5000ms) on optional steps to avoid waiting the full timeout when the step is absent.

Field Types

TEXT

For text inputs, email fields, phone numbers, and similar:

{
  "type": "text",
  "selector": "input[name='email']",
  "data_key": "email"
}

TEXTAREA

For multi-line text areas:

{
  "type": "textarea",
  "selector": "textarea[name='message']",
  "data_key": "message"
}

DROPDOWN / SELECT

For native <select> elements:

{
  "type": "dropdown",
  "selector": "select[name='country']",
  "data_key": "country",
  "type_config": {
    "select_by": "text"
  }
}

RADIO

For radio button groups:

{
  "type": "radio",
  "selector": "input[type='radio'][value='${value}']",
  "data_key": "payment_method"
}

Pattern A - Value-driven selector: Use ${value} placeholder in selector, data provides the value:

{
  "type": "radio",
  "selector": "input[type='radio'][value='${value}']",
  "data_key": "gender"
}
// data: { "gender": "male" }

Pattern B - Boolean flags: Use explicit selectors with boolean data values:

{
  "type": "radio",
  "selector": "[role='radio']:has-text('Yes')",
  "data_key": "accept_terms",
  "force_click": true
}
// data: { "accept_terms": true }  // clicks if truthy, skips if null

force_click is supported on radio, click_only, and click_select field types. When set, the click fallback chain tries force click before normal click.

CHECKBOX

For checkbox inputs:

{
  "type": "checkbox",
  "selector": "input[type='checkbox'][name='newsletter']",
  "data_key": "subscribe_newsletter"
}

Data value true checks the box, false unchecks it, and null skips the field entirely (no interaction). Only JSON booleans are accepted: any other value, such as the string "false", raises FieldInteractionError (or skips an optional field) instead of being read as truthy.

DATE

For date inputs with format conversion:

{
  "type": "date",
  "selector": "input[name='birthdate']",
  "data_key": "birthdate",
  "type_config": {
    "format": "DD.MM.YYYY"
  }
}

Supported formats (mapped to strftime internally):

Data must be provided in ISO format (YYYY-MM-DD) in input_data. The type_config.format specifies the output format for typing into the field; it takes the patterns above, not strftime directives. A value that is not ISO 8601 raises FieldInteractionError before the page is touched (an optional field is skipped instead), and an unsupported format raises InvalidInstructionError. Native <input type="date"> fields are auto-detected and use the value as-is, no type_config needed.

SLIDER

For range inputs:

{
  "type": "slider",
  "selector": "input[type='range'][name='coverage']",
  "data_key": "coverage_amount"
}

FILE

For file upload fields:

{
  "type": "file",
  "selector": "input[type='file']",
  "data_key": "document_path"
}

Data value should be the absolute file path.

COMBOBOX

For autocomplete/typeahead inputs:

{
  "type": "combobox",
  "selector": "input[aria-label='City']",
  "data_key": "city",
  "press_enter": true,
  "type_config": {
    "option_selector": ".autocomplete-option",
    "type_delay_ms": 50,
    "wait_after_type_ms": 500
  }
}

Breaking change in 1.1.6: press_enter moved from the combobox type_config to the field itself, where it now also works for text, date, iframe_field and click_select. Move the key up one level; leaving it inside type_config fails validation.

CLICK_SELECT

For custom dropdowns requiring click-then-select:

{
  "type": "click_select",
  "selector": ".custom-dropdown-trigger",
  "data_key": "option_value",
  "post_click_delay_ms": 300,
  "type_config": {
    "option_selector": ".dropdown-item:has-text('${value}')"
  }
}

CLICK_ONLY

For elements that only need clicking (no data input):

{
  "type": "click_only",
  "selector": "button.expand-section"
}

With conditional clicking based on data:

{
  "type": "click_only",
  "selector": "button:has-text('${value}')",
  "data_key": "selected_option"
}

IFRAME_FIELD

For fields inside iframes (alternative to iframe_selector):

{
  "type": "iframe_field",
  "selector": "input[name='card_number']",
  "iframe_selector": "iframe#payment-frame",
  "data_key": "card_number"
}

Common Field Parameters

Pressing Keys

press_keys sends keys to the page once the field's own interaction has finished, in the order given. Its main use is dismissing a widget that swallows the next click — a date picker left open by a force_click, or a custom select with no close control:

{
  "type": "click_select",
  "selector": "#vehicle-fuel",
  "data_key": "fuel",
  "press_keys": ["Escape"],
  "type_config": {
    "option_selector": "ul.dropdown-content li"
  }
}

The keys go to whatever the page currently focuses, as a real key press would, so the field's own element does not need to still exist. This is what separates it from press_enter, which presses on the field itself to commit a value; the two are independent and can be combined.

An overlay that outlives the last field of a step blocks the step's next_action. Attach the keys to that last field rather than adding a field for them.

Optional Fields

Fields marked with "optional": true are skipped gracefully when the element is not found on the page or when interaction fails. This is useful for elements that may or may not appear depending on dynamic page behavior, A/B tests, or conditional rendering that cannot be predicted by data alone.

{
  "type": "click_only",
  "selector": "#promotional-banner button.dismiss",
  "data_key": "dismiss_promo",
  "optional": true,
  "field_visible_timeout_ms": 2000
}

How optional differs from null values:

When optional is set and the element is not found or interaction fails (including a value that does not stick after filling, or an unusable input value), the handler logs an info message and continues to the next field. No error is raised regardless of the strict mode. The message says why: Optional field not found when nothing matches the selector, Optional field present but not visible (N matching) when it matches only hidden elements, and Optional field interaction failed with Playwright's reason (for example a disabled element) when the interaction itself failed. A required field that matches only hidden elements raises FieldNotFoundError with Field not visible: ... instead of Field not found: ....

Tip: Set a low field_visible_timeout_ms (e.g., 1000-2000ms) on optional fields to avoid waiting the full default timeout when the element is absent.

Note: For skipping entire steps (all fields + next_action), use Optional Steps instead.

Action Types

CLICK

Click a button or link:

{
  "type": "click",
  "selector": "button[type='submit']"
}

With iframe support:

{
  "type": "click",
  "selector": "button:has-text('Next')",
  "iframe_selector": "iframe#form-frame"
}

WAIT

Wait for an element to appear:

{
  "type": "wait",
  "selector": ".loading-complete"
}

WAIT_HIDDEN

Wait for an element to disappear:

{
  "type": "wait_hidden",
  "selector": ".loading-spinner"
}

SCROLL

Scroll to an element:

{
  "type": "scroll",
  "selector": "#section-bottom"
}

DELAY

Wait for a fixed time:

{
  "type": "delay",
  "delay_ms": 2000
}

CONDITIONAL

Execute actions based on conditions:

{
  "type": "conditional",
  "condition": {
    "type": "selector_visible",
    "selector": ".error-message"
  },
  "actions": [
    {
      "type": "click",
      "selector": "button.dismiss-error"
    }
  ]
}

Condition types:

• selector_visible: True if any element matching the selector is visible on the page
• selector_hidden: True if no element matching the selector is visible on the page
• data_equals: True if input_data[key] equals value
• data_exists: True if input_data[key] is truthy

SOLVE_CAPTCHA

Clear a captcha that appears mid-flow, typically after a submit. Place it where the challenge appears, usually as a step's next_action once the submit button has been clicked as a click_only field. captcha.kind names the challenge and picks the solver; each kind has its own settings and its own bundled model.

azcrawlerpy[captcha] installs every solver. Both share these fields:

Every kind's default model ships in its own package, installed by the kind's extra: azcrawlerpy-recaptcha-google for recaptcha_v2 and azcrawlerpy-recaptcha-text for text. They are published apart from azcrawlerpy, so a code release does not upload the weights again. Each extra pins an exact version of its weights package, so every install of one azcrawlerpy release runs the same models, and a retrained model reaches installs through an azcrawlerpy release that bumps the pin. There is no download at run time. A provenance card sits next to each weights file (same name, .json): which model variant, which training run, which commit, and its score. The solver logs that line when it loads the model, so a deployment's model is identifiable from its log. The models are trained in the captcha-training project, which scores candidates against the pinned release and writes a winner into a weights package checkout it is given. Override per action with weights, or per machine with AZCRAWLERPY_CAPTCHA_RECAPTCHA_V2 and AZCRAWLERPY_CAPTCHA_TEXT.

A challenge that does not clear raises CaptchaNotSolvedError naming the kind, the rounds spent and the reason. It is never reported as cleared, so a page guarded by something the solver cannot handle fails loudly instead of failing later for an unrelated-looking reason. If no challenge appears at all the action succeeds, which is the normal case for invisible reCAPTCHA that passes silently.

Kind recaptcha_v2

{ "type": "solve_captcha", "captcha": { "kind": "recaptcha_v2" } }

The bundled model is TesseraNet, one network for both grid kinds: a 3x3 round's tiles and a 4x4 round's photo, read whole, go through a graph per grid kind (tesseranet_3x3.onnx, tesseranet_4x4.onnx) that takes pixels to every category's probability per square, and cell_head.json beside them names each channel's category and threshold. It runs on CPU through onnxruntime with no torch, in tens of milliseconds per round, over the reCAPTCHA categories listed below. It has no zero-shot fallback: a category it was not trained on is skipped. captcha-training trains, exports and promotes it.

A weights path (or AZCRAWLERPY_CAPTCHA_RECAPTCHA_V2) must name a TesseraNet 3x3 graph with its tesseranet_4x4.onnx and cell_head.json beside it. Category labels are matched to the head's channel labels ignoring case, spaces and underscores.

How the category is read, in any language

reCAPTCHA renders in the page's language (the browser locale, or the hl parameter the site passes). Each round resolves its category in this order, and its log line says which step decided: reCAPTCHA round: prompt='…' target='Bridge' via=challenge-id id=/m/015kr tiles=9.

1. challenge-id: the widget's own challenge response names the category by an id that is the same in every language. The crawler listens for it from the moment a page opens, and a round only uses the response that delivered the grid on screen, never an earlier one.
2. challenge-name: the English name Google sends with 3x3 challenges, for an id not in the table yet.
3. prompt-noun: the word the widget sets in bold ("buses", "Brücken", "biciclette"), matched against built-in English, German and Italian stems plus prompt_labels. Used when the challenge response was not seen.
4. unresolved: the English name, or else the noun itself, is passed on as is; a label the weights do not know is skipped.

The labels are Bicycle, Boat, Bridge, Bus, Car, Chimney, Crosswalk, Hydrant, Motorcycle, Mountain, Palm, Parking Meter, Stairs, Taxi, Tractor, Traffic Light and Truck. The weights' cell_head.json lists the categories they know; a round asking for any other is skipped.

Another widget language

prompt_labels only matters for rounds without a challenge id. Take the bold words from the round log and add a stem per category, short enough to cover inflections and long enough not to appear inside another category's word; longer stems are tried first, so "autobus" wins over "auto":

{
  "type": "solve_captcha",
  "captcha": {"kind": "recaptcha_v2", "prompt_labels": {"feux": "Traffic Light", "vélo": "Bicycle"}}
}

What determines the success rate

The approach follows the ETH Zurich Breaking reCAPTCHAv2 paper (COMPSAC 2024), which reports a 100% solve rate on the Google demo widget with a YOLOv8 tile classifier; the bundled model is TesseraNet, a different network trained on the same public tile sets plus photo datasets. Their measurements show that the model is the smallest part of that result; what reCAPTCHA thinks of the session decides how many rounds it serves before accepting an answer:

Humans in the same study needed a median of 2 challenges, so a session that keeps being served rounds after a handful of correct answers is being judged on its reputation, not its answers. Practical consequences for a crawl:

• Keep humanize enabled in the browser config; Camoufox then moves the cursor along a curve for every tile click, which is the mouse-movement condition above.
• A warmed profile (see Browser Profile Building) is the cookie/history condition. Dropping it costs rounds even with a perfect classifier.
• A datacenter exit IP is the dominant negative signal. The paper's bot passed every run on a residential-looking IP and was flagged within ~20 runs without one.
• The paper's threshold on the target-class probability is 0.2; the bundled weights carry their own per-category cuts for each grid kind in cell_head.json. A refused empty Verify means the grid holds a match no tile cleared its cut for, so the single likeliest tile is clicked.
• For 4x4 "select all squares" grids the paper segments the whole image and maps the mask onto cells. This action likewise reads the whole photo, with TesseraNet's 4x4 graph scoring every square.

Notes and limitations (recaptcha_v2)

• Only the visible challenge frame is solved. Pages rendering several widgets have several hidden frames holding blank tiles; those are ignored.
• Tiles are cropped from one screenshot of the grid per pass, so solving never scrolls the page.
• Clicks are paced like a hand: a pause to look at the grid, a varying gap between tiles, a pause before Verify, and each click lands at a random point in the middle of the tile rather than its exact centre. Camoufox's humanize supplies the cursor path; this supplies where and when it ends. The ETH paper measured natural pointer behaviour as the second-largest factor after the IP.
• Unknown categories are skipped (targets are sparse), which keeps the round loop alive rather than aborting on the first unfamiliar prompt.
• An empty answer is submitted at most once in a row. If no tile clears the confidence threshold and the previous round already submitted nothing, or the widget answers "Please select all matching images", the likeliest tile is clicked instead. Repeating an empty answer cannot clear a challenge that is still on screen, and it burns the round budget.
• Dynamic grids are recognised by the widget marking clicked tiles while replacements load; the replacements are classified again until none match, then Verify is pressed.
• Solving is not the same as passing. A session reCAPTCHA distrusts receives visually degraded tiles and effectively unlimited rounds; no classifier reliably beats that. The action gives up after max_rounds rather than looping forever, so the failure stays visible. Proxy reputation and session history matter at least as much as the model.
• 4x4 grids are one photo cut into squares. The squares are cropped out of the widget's gaps, reassembled into the photo and read whole by the 4x4 graph, which scores every square at once.
• Classifiers are cached per configuration, so repeated challenges in one process reuse loaded weights.
• The bundled weights are TesseraNet, trained from scratch in captcha-training with no pretrained backbone. Its training data includes public reCAPTCHA tile sets (MIT, CC BY 4.0, CC0, and one with no stated licence) and COCO, LVIS, ADE20K and Open Images photos, and some of its labels were produced by OWLv2 (Apache-2.0) and YOLOv8-seg (AGPL-3.0); review those terms before redistributing, or point weights at your own model.

Kind text

{
  "type": "solve_captcha",
  "captcha": {
    "kind": "text",
    "image_selector": "img.captcha",
    "input_selector": "#captcha-code",
    "submit_selector": "button[type=submit]",
    "refresh_selector": "a.captcha-reload",
    "min_confidence": 0.6
  }
}

The image is screenshotted, read by the bundled SVTRv2 recogniser (ONNX, runs on CPU through onnxruntime, no torch), and the reading is typed into the input one character at a time with a human pace. The alphabet is digits, ASCII letters in both cases and ASCII punctuation, up to the length the recogniser was trained for (see azcrawlerpy.captcha.text.charset). Before reading, a small chooser (selector.onnx) looks at a thumbnail and the crop's aspect ratio and names one of four input sizes; the recogniser then reads the crop once at that size. The two are one pair, so a weights path (or AZCRAWLERPY_CAPTCHA_TEXT) must name a recogniser with its selector.onnx beside it.

Rounds only exist with submit_selector: a page that shows a new image after a wrong answer gets another reading until max_rounds. Without it the action ends after typing, so a wrong reading surfaces wherever the flow submits.

Common Action Parameters

Final Page Configuration

{
  "final_page": {
    "wait_for": ".result-container, .confirmation",
    "timeout_ms": 60000,
    "post_wait_delay_ms": 2000,
    "screenshot_selector": ".result-panel"
  }
}

Only one of wait_for or wait_for_data_key can be set (not both).

Post Final Page Steps

post_final_page_steps is an optional list of regular step definitions that run on the result page after its wait_for/post_wait_delay_ms settle and before data_extraction, the end screenshot, and the HTML capture. Use it to interact with the results (load more entries, switch coverage tabs) that a normal step can never reach because the result page only exists after all steps and CAPTCHA handling.

Each entry supports the full step semantics (wait_for, optional, restart_crawl, extraction, retries). Use {"type": "delay", "delay_ms": 0} as next_action when a step only extracts. Step extractions additionally support a repeat-click loop for load-more buttons:

{
  "post_final_page_steps": [
    {
      "name": "load_all_results",
      "wait_for": "div[data-test^='tarifkachel/']",
      "timeout_ms": 60000,
      "fields": [],
      "post_field_extraction": [
        {
          "name": "tariff_count",
          "selector": "div[data-test^='tarifkachel/']",
          "click_before": "button:has-text('weitere Ergebnisse laden')",
          "repeat_click_until_gone": true,
          "max_repeat_clicks": 20,
          "wait_after_click_ms": 1500
        }
      ],
      "next_action": { "type": "delay", "delay_ms": 0 }
    }
  ]
}

The repeat-click loop is available on any step extraction, but is most useful here. The whole loop runs inside the step's timeout_ms budget.

Per-Step Page Extraction

instructions.data_extraction runs once, after every step, so it only ever captures the UI state the last step left behind. When a results page changes in response to clicks — a coverage tab, a filter, a sort order — each state needs its own list snapshot. page_extraction puts a full DataExtractionConfig on a step and runs it at the end of that step, after post_field_extraction and before next_action, so it sees the DOM the step's own clicks produced.

{
  "name": "screen_vk_sb500",
  "wait_for": "[data-test='deckung/VK']",
  "timeout_ms": 120000,
  "optional": true,
  "data_extraction": [
    { "name": "_switch_vk", "selector": "[data-test='deckung/VK']",
      "click_before": "[data-test='deckung/VK']", "wait_after_click_ms": 6000 },
    { "name": "_load_more", "selector": "div[data-test^='tarifkachel/']",
      "click_before": "button:has-text('weitere Ergebnisse laden')",
      "repeat_click_until_gone": true, "max_repeat_clicks": 20 }
  ],
  "page_extraction": {
    "fields": {
      "insurer_names": { "selector": "div[data-test^='tarifkachel/'] img[alt]", "attribute": "alt", "multiple": true }
    }
  },
  "fields": [],
  "next_action": { "type": "delay", "delay_ms": 2000 }
}

Results are namespaced under the step name, so several steps can extract the same field names without overwriting each other:

result.extracted_data["screen_vk_sb500"]["insurer_names"]  # [... 98 items ...]

Because every extraction shares one flat result dict and instructions.data_extraction is merged over it last, Instructions rejects configs where a page_extraction step name would collide with another snapshot step, a step extraction name, or a final-page field name.

Failures follow the same restart_crawl > optional > strict chain as the other extraction slots, so a snapshot step marked optional: true cannot fail the crawl. Extraction cost scales with fields × matched elements and counts against the step's timeout_ms, so budget list-heavy snapshots accordingly — an exhausted budget on an optional step silently drops the snapshot.

Data Extraction Configuration

Extract structured data from the final page using CSS selectors:

{
  "data_extraction": {
    "fields": {
      "tier_prices": {
        "selector": ".price-value",
        "attribute": null,
        "regex": "([0-9]+[.,][0-9]{2})",
        "multiple": true,
        "iframe_selector": "iframe#form-frame"
      },
      "selected_price": {
        "selector": "#total-amount",
        "attribute": "data-value",
        "regex": null,
        "multiple": false
      }
    }
  }
}

The data_extraction config also supports nested_fields to combine flat extracted arrays into structured outputs (paired_dict or object_list). See docs/README.md for details.

Extracted data is available in the crawl result:

result = await crawler.crawl(...)
print(result.extracted_data)
# {'tier_prices': ['32,28', '35,26', '50,34'], 'selected_price': '35,26'}

Data Points (input_data)

The input_data dictionary provides values for form fields. Keys must match data_key values in the instructions.

Structure

{
  "email": "user@example.com",
  "first_name": "John",
  "last_name": "Doe",
  "birthdate": "1985-06-15",
  "country": "Germany",
  "accept_terms": true,
  "newsletter": false,
  "premium_option": null
}

Value Types

Note: Setting a value to null skips the field without any DOM interaction. This is different from "optional": true on the field definition, which attempts the interaction but tolerates failure. See Optional Fields.

Every data_key a non-optional field references must be present in the data row: an absent key raises MissingDataError, so a misspelt key cannot silently leave a field unfilled. Use null to skip a field.

Radio Button Patterns

Pattern A - Mutually exclusive options with value selector:

{
  "gender": "male"
}

Selector uses ${value} placeholder: input[value='${value}']

Pattern B - Boolean flags for each option:

{
  "option_a": true,
  "option_b": null,
  "option_c": null
}

Only the option with true gets clicked.

Date Handling

Dates in input_data should use ISO format (YYYY-MM-DD):

{
  "birthdate": "1985-06-15",
  "start_date": "2024-01-01"
}

The framework converts to the format specified in the field definition.

Element Discovery

The ElementDiscovery class scans web pages to identify interactive elements, helping build instructions.json files.

from pathlib import Path
from azcrawlerpy import ElementDiscovery

async def discover_elements():
    discovery = ElementDiscovery(headless=False)

    report = await discovery.discover(
        url="https://example.com/form",
        output_dir=Path("./discovery_output"),
        cookie_consent={
            "banner_selector": "#cookie-banner",
            "accept_selector": "button.accept"
        },
        explore_iframes=True,
        screenshot=True,
    )

    print(f"Found {report.total_elements} elements")

    for text_input in report.text_inputs:
        print(f"Text input: {text_input.selector}")
        print(f"  Suggested type: {text_input.suggested_field_type}")

    for dropdown in report.selects:
        print(f"Dropdown: {dropdown.selector}")
        print(f"  Options: {dropdown.options}")

    for radio_group in report.radio_groups:
        print(f"Radio group: {radio_group.name}")
        for option in radio_group.options:
            print(f"  - {option.label}: {option.selector}")

Discovery Report Contents

• text_inputs: Text, email, phone, password fields
• textareas: Multi-line text areas
• selects: Native dropdown elements with options
• radio_groups: Grouped radio buttons
• checkboxes: Checkbox inputs
• buttons: Clickable buttons
• links: Anchor elements
• date_inputs: Date picker fields
• file_inputs: File upload fields
• sliders: Range inputs
• custom_components: Non-standard interactive elements
• iframes: Discovered iframes with their elements

AI Agent Guidance

This section provides instructions for AI agents tasked with creating instructions.json and input_data files.

Workflow for Creating Instructions

1. Discovery Phase: Use ElementDiscovery to scan each page/step of the form
2. Mapping Phase: Map discovered elements to field definitions
3. Flow Definition: Define step transitions and actions
4. Data Schema: Create the input_data structure

Step-by-Step Process

1. Analyze the Form Structure

• Identify how many pages/steps the form has
• Note the URL pattern changes (if any)
• Identify what element appears when each step loads

2. For Each Step, Define:

{
  "name": "<descriptive_step_name>",
  "wait_for": "<selector_that_confirms_step_loaded>",
  "timeout_ms": 15000,
  "fields": [...],
  "next_action": {...}
}

Naming conventions:

• Use snake_case for step names: personal_info, payment_details
• Use descriptive data_keys: first_name, email_address, accepts_terms

3. Selector Priority

When choosing selectors, prefer in order:

1. [data-testid='...'] or [data-cy='...'] - Most stable
2. [aria-label='...'] or [aria-labelledby='...'] - Accessible and stable
3. input[name='...'] - Form field names
4. :has-text('...') - Text content (use for buttons/labels)
5. CSS class selectors - Least stable, avoid if possible

4. Handle Dynamic Content

For AJAX-loaded content:

• Use wait action before interacting
• Add field_visible_timeout_ms to field definitions
• Use post_click_delay_ms for fields that trigger updates

5. Radio Button Strategy

Option A - When radio values are meaningful:

{
  "type": "radio",
  "selector": "input[type='radio'][value='${value}']",
  "data_key": "payment_type"
}
// data: { "payment_type": "credit_card" }

Option B - When you need individual control:

{
  "type": "radio",
  "selector": "[role='radio']:has-text('Credit Card')",
  "data_key": "payment_credit_card",
  "force_click": true
},
{
  "type": "radio",
  "selector": "[role='radio']:has-text('PayPal')",
  "data_key": "payment_paypal",
  "force_click": true
}
// data: { "payment_credit_card": true, "payment_paypal": null }

6. Iframe Handling

When elements are inside iframes:

{
  "type": "text",
  "selector": "input[name='card_number']",
  "iframe_selector": "iframe#payment-iframe",
  "data_key": "card_number"
}

Creating input_data

1. Analyze Required Fields

From the instructions, extract all unique data_key values:

data_keys = set()
for step in instructions["steps"]:
    for field in step["fields"]:
        if field.get("data_key"):
            data_keys.add(field["data_key"])

2. Determine Value Types

3. Handle Mutually Exclusive Options

For radio groups with boolean flags, only ONE should be true:

{
  "employment_fulltime": true,
  "employment_parttime": null,
  "employment_selfemployed": null,
  "employment_unemployed": null
}

4. Date Format

Always provide dates in ISO format in input_data:

{
  "birthdate": "1985-06-15",
  "policy_start": "2024-01-01"
}

The instructions specify the output format for the specific form.

Common Patterns

Multi-Step Wizard

{
  "steps": [
    {
      "name": "step_1_personal",
      "wait_for": "input[name='firstName']",
      "fields": [...],
      "next_action": { "type": "click", "selector": "button:has-text('Next')" }
    },
    {
      "name": "step_2_address",
      "wait_for": "input[name='street']",
      "fields": [...],
      "next_action": { "type": "click", "selector": "button:has-text('Next')" }
    }
  ]
}

Form with Loading States

{
  "next_action": {
    "type": "click",
    "selector": "button[type='submit']",
    "post_action_delay_ms": 1000
  }
}

Conditional Fields

{
  "type": "conditional",
  "condition": {
    "type": "data_equals",
    "key": "has_additional_driver",
    "value": true
  },
  "actions": [
    {
      "type": "click",
      "selector": "button:has-text('Add Driver')"
    }
  ]
}

Retry and Resilience

The framework provides built-in retry and fallback mechanisms for handling transient failures during web interactions.

Retry Configuration

Add retry_config to any field, action, step extraction, or data extraction to enable automatic retry with exponential backoff:

{
  "type": "text",
  "selector": "input[name='email']",
  "data_key": "email",
  "retry_config": {
    "max_attempts": 3,
    "base_delay_ms": 500,
    "backoff_multiplier": 1.5
  }
}

Retryable errors include Playwright TimeoutError and Error (element not found, intercepted clicks, etc.), and on fields a value that does not stick after filling.

Click Fallback Chain

All click interactions (click actions, click_only, radio, click_select fields) use an escalating fallback strategy when a click fails:

1. Normal click (or force click if force_first=True)
2. Force click (or normal click if force_first=True) -- bypasses actionability checks
3. JS dispatchEvent -- fires a MouseEvent via element.dispatchEvent()
4. JS element.click() -- calls element.click() via document.querySelector()

This handles common real-world issues like overlay interception, sticky headers, and elements that pass Playwright's visibility checks but fail to receive pointer events.

Steps 3 and 4 only run when browser_config.allow_synthetic_events is on: a click dispatched from JS reaches the page with isTrusted=false, which bot detection reads as automation.

Value Verification

text and textarea fields verify the actual input value after filling. If the value written to the field does not match the expected value, a FieldInteractionError is raised once retry_config attempts are exhausted (an optional field is skipped instead). This catches silent data corruption from autofill interference, input masks, or JavaScript reformatting.

Set skip_verification: true on the field to disable this check for fields where the site intentionally transforms the input (e.g., phone number formatting).

ComboBox Degraded Retry

When retry_config is set on a combobox field, retries use degraded parameters to handle slow autocomplete responses:

• Typing delay scales by 1.5^(attempt-1) (slower typing gives autocomplete more time)
• The field is always cleared before retyping on retry attempts

Error Handling and Diagnostics

The framework provides detailed error information when failures occur.

Exception Types

Every CrawlerError that crawl() raises carries a partial_result: a CrawlResult holding whatever had been collected before the failure (screenshots, extracted_data, steps_completed, profiler_visited_urls, platform_logs, crawler_log, stdout_log, stderr_log, playwright_debug_log, error_diagnostics, and the debug recording: run_info, page_events, network_log, route_hits). That holds for failures before the page loop too, such as a profiler error. Catch the exception and read e.partial_result to recover that data — see In-Memory Error Diagnostics below.

Timeout Budgets

Every timeout is used exactly as configured (or as its built-in default); only delays are randomized, between 1x and 2x, to pace interactions like a person.

The crawler enforces two independent wall-clock budgets. Both translate their exhaustion into CrawlerTimeoutError (or CrawlRestartError when restart is configured) and attach a partial result.

1. Per-step budget — step.timeout_ms

Hard upper bound on the total wall-clock duration of one step. Covers wait_for, data_extraction, field handlers (including retries), next_action (including retries and delays), post_field_extraction and page_extraction. The same field is also used as the timeout for the initial wait_for_selector call, so a step that uses its full budget waiting for the selector leaves no time for the rest of the body. On exhaustion, the priority chain restart_crawl > optional > strict decides the outcome:

• restart_crawl=True → CrawlRestartError → full browser restart (up to max_restarts).
• optional=True → step is skipped silently.
• strict=True → CrawlerTimeoutError.
• strict=False → warning logged, step returns.

When the budget runs out, the crawler reports where it went, phase by phase: the log line carries time_spent=[...] and the CrawlerTimeoutError message ends with, for example, Time spent: wait_for 812ms, field 'birth_date' 1203ms, field 'brand' 6120ms (did not finish). Retries and debug screenshots count against the budget and show up in that breakdown. Every completed step logs the same breakdown in Step completed: name='...' time_spent=[...], so a failing run can be compared with a successful one.

2. Crawl-level budget — crawler.crawl(..., global_timeout_ms=...)

Optional wall-clock cap on the entire crawl (profiler, navigation, all steps and the final-page capture). None (the default) disables it. When the deadline fires, the crawl task is cancelled and CrawlerTimeoutError(step_name="<global_timeout>", selector="<global>", timeout_ms=...) is raised with a partial CrawlResult attached. The partial carries screenshots, extracted_data, steps_completed, profiler_visited_urls, platform_logs, crawler_log, stdout_log, stderr_log, playwright_debug_log and the debug recording collected up to the moment of cancellation. html is empty and final_url falls back to the start URL because the live page is being torn down by the cancellation and cannot be read.

from azcrawlerpy.crawling.exceptions import CrawlerTimeoutError

try:
    result = await crawler.crawl(
        url=url,
        input_data=input_data,
        instructions=instructions,
        debug_mode=DebugMode.ALL,
        global_timeout_ms=300_000,  # 5 minutes
    )
except CrawlerTimeoutError as e:
    if e.step_name == "<global_timeout>":
        # Global budget hit -- inspect partial_result for what was collected.
        partial = e.partial_result
        if partial is not None:
            print(f"Timed out after {partial.steps_completed} step(s)")
            print(f"Screenshots: {len(partial.screenshots)}")
    else:
        # Per-step budget or restart-budget exhaustion.
        raise

Debug Mode

Enable debug mode to capture screenshots at various stages:

from azcrawlerpy import DebugMode

result = await crawler.crawl(
    ...,
    debug_mode=DebugMode.ALL,  # Capture all screenshots
)

Every mode other than NONE also returns the captured logs (crawler_log, stdout_log, stderr_log, playwright_debug_log), error diagnostics with an error screenshot on failure, and the debug recording. Each screenshot pairs the JPEG with the page HTML and the form state at the same moment. A screenshot that cannot be taken, for example because a click is navigating the page, is skipped and logged; debug capture never fails a crawl, and never replaces the error of one that failed.

Debug Recording

With any debug_mode other than NONE, the result, or the partial_result of a failed crawl, holds a record of the whole run across every browser attempt, so restarts keep what the earlier attempts saw:

The recording is bounded, so a chatty page cannot exhaust memory: each list keeps its newest entries, texts and bodies are cut to a fixed length, and captured bodies share a size budget that drops the oldest bodies first. Body reads still running when an attempt ends get a few seconds before the browser closes.

The recording is held in memory and returned on success, on every failure and when global_timeout_ms runs out. A task cancelled from outside, or a process killed before crawl() returns, has nowhere to return it, so set global_timeout_ms below the host's own time limit: the crawl then ends itself and hands back a partial result.

Captured headers and bodies contain cookies and the submitted form data; store them like the input data itself.

Seeing the network exit. Set egress_check_url on CrawlerBrowserConfig to an IP echo service, for example https://ipinfo.io/json, which also returns the location and network owner. Before each attempt starts, the crawler loads that URL in a separate tab of the same browser, so it leaves through the same proxy exit as the crawl. The response is logged and, when debugging, recorded in run_info.egress_checks.

With geoip enabled on camoufox, the crawler looks up the exit IP once per crawl and passes it to every launch, so the profiler warmup and every restart claim the same location; run_info.geoip_ip records it. Comparing it with the egress check shows whether the timezone and locale the fingerprint claims match the exit the site actually sees. A proxy that rotates the exit per connection makes them differ.

The locale follows CrawlerBrowserConfig.locale (e.g. "de-DE"). Left unset with geoip on, it is the exit country's most spoken language, where camoufox would otherwise draw one at random by population share and present rare locales such as bar-DE. Each camoufox crawl also fixes one device for all its launches: a fingerprint preset plus the canvas, audio and font noise seeds, font list and voices that camoufox would redraw on every launch.

from azcrawlerpy import CrawlerBrowserConfig, ProxyConfig

browser_config = CrawlerBrowserConfig(
    proxy=ProxyConfig(server="http://proxy.example.com:8080", username="user", password="pass"),
    geoip=True,
    egress_check_url="https://ipinfo.io/json",
)

AI Diagnostics

When errors occur with debug mode enabled, the framework captures:

• Current page URL and title
• Available data-cy selectors
• Visible buttons and input fields
• Similar selectors (fuzzy matching suggestions)
• Error screenshot (captured in-memory as bytes), with the page HTML and form state

Each of these captures is independent: when the page is navigating or gone, the ones that fail are left empty and the rest are kept. Console output and network traffic are in page_events and network_log, not in the diagnostics.

This information is included in the exception message and attached to the returned/partial CrawlResult as error_diagnostics (a dict[str, Any] equivalent to the content of the previous error_diagnostics.json file). The caller persists it wherever they like.

In-Memory Error Diagnostics

Error diagnostics are fully available in-memory. The error screenshot is captured as bytes and accessible in two ways:

1. On the exception's diagnostics: exception.diagnostics.screenshot_bytes
2. As the last entry in the partial result: exception.partial_result.screenshots[-1] (a Screenshot model with label, image bytes, and the page html at capture time)

try:
    result = await crawler.crawl(
        url=url,
        input_data=input_data,
        instructions=instructions,
        debug_mode=DebugMode.ALL,
    )
except CrawlerError as e:
    # Access the error screenshot bytes directly from diagnostics
    if e.diagnostics and e.diagnostics.screenshot_bytes:
        error_screenshot = e.diagnostics.screenshot_bytes

    # Or from the partial result (includes all screenshots taken during the crawl,
    # each a Screenshot model pairing label, image bytes and page HTML)
    if e.partial_result:
        all_screenshots = e.partial_result.screenshots  # includes error screenshot as last item
        extracted_so_far = e.partial_result.extracted_data
        steps_done = e.partial_result.steps_completed

Azure App Service Log Capture

When the crawler runs on an Azure App Service Plan target (Web App, Function App, or Web App / Function App in a container), debug_mode=DebugMode.ALL automatically captures the platform log tree so it can be attached to the run's output artifacts. It holds the host-side view of the process, including the worker's stdout and stderr, so it is often the only way to see low-level browser-launch failures in post-mortem. Playwright's own debug stream is captured separately into playwright_debug_log.

When it activates (two gating conditions):

• debug_mode=DebugMode.ALL, AND
• $HOME is set and $HOME/LogFiles/ is a directory.

If either gating check fails, the crawl is otherwise untouched and a clear reason is logged (see "Log output" below).

Sink:

The full $HOME/LogFiles/ tree is read into a dict[str, bytes] and returned on CrawlResult.platform_logs. Keys are forward-slash relative paths (e.g. "Application/Functions/Worker/worker.log"); values are the raw file bytes. On graceful failure the same dict is attached to CrawlerError.partial_result.platform_logs, so harnesses that catch CrawlerError can still recover them. The crawler never writes to disk — the caller owns the sink.

What gets collected:

The entire $HOME/LogFiles/ tree, recursively. No filtering. Files include:

• Application/Functions/Worker/*.log (Python worker stdout/stderr on Functions)
• Application/Functions/Host/ (Functions host logs)
• <instance>_default_docker.log (Web App container stdout/stderr)
• kudu/, Nginx/, docker.log, etc.

On Linux App Service that path is /home/LogFiles/; on Windows App Service it is typically C:\home\LogFiles\ or D:\home\LogFiles\. The detection uses $HOME so both work without configuration.

Usage example (harness uploads to its own storage):

result = await crawler.crawl(
    url=url,
    input_data=input_data,
    instructions=instructions,
    debug_mode=DebugMode.ALL,  # activates platform log capture
)

# result.platform_logs is a dict[str, bytes]
for rel_path, content in result.platform_logs.items():
    blob_client.upload_blob(
        name=f"{run_id}/platform_logs/{rel_path}",
        data=content,
        overwrite=True,
    )

Crash-safe:

The collection is wrapped in try / finally around the entire crawl body, so it runs on:

• Successful return
• CrawlerTimeoutError from an exhausted restart budget
• Any CrawlerError wrapping a browser crash
• Profiler-phase failures (the profiler is inside the try)
• KeyboardInterrupt / task cancellation

Per-file errors (Windows log rotation, permission issues, races) are logged as warnings and counted; they never mask the crawl's original exception.

Log output:

Every invocation emits one start line plus one outcome line via the configured logger (flows to Application Insights through azpaddypy):

Deployment requirements per target:

Delivery:

The collected bytes land on CrawlResult.platform_logs on success. On graceful failure (any CrawlerError, including CrawlerTimeoutError from an exhausted restart budget or the global_timeout_ms wall-clock cap), they land on exception.partial_result.platform_logs.

Shared-worker noise:

/home/LogFiles/Application/Functions/Worker/*.log is shared across all concurrent function invocations on the same worker instance. The copied snapshot therefore includes any overlapping activity — there is no per-invocation filtering. This is an intentional "full copy always" trade for forensic completeness.

Browser Profile Building (Profiler)

The profiler visits random sites before the main crawl to accumulate cookies and storage, making the browser appear more natural. This is configured via the profiler field in instructions.json.

Profiler Modes

The profiler supports two storage modes:

Disk mode (storage_path set): Persists the browser profile to disk.

• Camoufox: saves as a Firefox user data directory ({storage_path}.camoufox/)
• Chromium: saves as a JSON file ({storage_path}.chromium.json)

In-memory mode (storage_path omitted or null): No permanent files written.

• Chromium: returns storage state as a dict, passed directly to browser.new_context(storage_state=dict)
• Camoufox: uses a temporary directory, passed as user_data_dir during the crawl and deleted when the crawl ends, whether it succeeds or fails

Profiler Configuration

{
  "profiler": {
    "sites": [
      {
        "url": "https://www.google.de",
        "cookie_consent": {
          "banner_selector": "[role='dialog'], .cookie-banner",
          "accept_selector": "button[id*='accept'], button[id*='agree']",
          "js_fallback_texts": ["akzept", "accept", "zustimm"]
        },
        "browse_delay_ms": 2000
      },
      {
        "url": "https://en.wikipedia.org",
        "browse_delay_ms": 3000
      }
    ],
    "visit_count": 3,
    "ignore_errors": true,
    "storage_path": null,
    "inter_site_delay_ms": 1000
  }
}

ProfilerConfig Parameters

Bootstrap Profiles

A profile with no history is a signal in itself: reputation systems score a browser that has never existed before far lower than one with aged cookies. storage_path solves that by persisting the profile, but it makes runs stateful — concurrent crawls then share and mutate one directory.

bootstrap_path keeps runs stateless. The seed is treated as read-only: it is copied into the run's throw-away profile before the warmup, and every change the crawl makes stays in the copy.

{
  "profiler": {
    "visit_count": 3,
    "ignore_errors": true,
    "bootstrap_path": "./browser_profiles/it-seed",
    "sites": [{ "url": "https://www.youtube.com/", "browse_delay_ms": 15000 }]
  }
}

• Camoufox: point it at a user_data_dir directory; it is copied with copytree.
• Chromium: point it at a storage state JSON file; it is passed as the context's storage_state.
• A missing or wrong-typed path logs a warning and is skipped, so a bad seed never fails a crawl.

Create a seed once, by hand, then reuse it: launch a browser with persistent_context=True and a user_data_dir, browse normally for a while (signing in to accounts whose cookies you want the seed to carry), and close it. Point bootstrap_path at that directory. Refresh it periodically — cookies expire, and a stale seed loses its value.

Note the seed is shared state, not shared identity: each run still gets its own device, so a seed used across many concurrent runs presents the same cookies from different machines. Keep seeds per-target, and prefer several seeds over one when running at volume.

Fingerprint Consistency

Camoufox picks a new fingerprint on every launch. The profiler and the crawl are two launches, so without intervention the crawl would present the profiler's cookies from a subtly different device — navigator.hardwareConcurrency changing between the warmup and the crawl, for example.

When a profiler is configured on camoufox, the crawler pins one fingerprint_preset and passes it to both launches, so warmup and crawl are the same machine. Presets are real captured devices and are the mechanism camoufox documents for this; hand-built fingerprints trigger a leak warning and are avoided deliberately. The preset follows browser_config.os, so it stays consistent with the spoofed platform.

Nothing to configure — it applies automatically whenever profiler is set and browser_type is camoufox. Without a profiler there is only one launch, so camoufox's per-launch fingerprint is left alone.

ProfilerSiteConfig Parameters

Each site in the sites array supports per-site cookie consent handling:

ProfilerResult

The profiler returns a ProfilerResult with:

The crawler automatically passes the profiler result to the browser context creation, so no manual wiring is needed.

Examples

Insurance Quote Form

instructions.json:

{
  "url": "https://insurance.example.com/quote",
  "browser_config": {
    "viewport_width": 1920,
    "viewport_height": 1080
  },
  "cookie_consent": {
    "banner_selector": "#cookie-banner",
    "accept_selector": "button:has-text('Accept')"
  },
  "steps": [
    {
      "name": "vehicle_info",
      "wait_for": "input[name='hsn']",
      "timeout_ms": 15000,
      "fields": [
        {
          "type": "text",
          "selector": "input[name='hsn']",
          "data_key": "vehicle_hsn"
        },
        {
          "type": "text",
          "selector": "input[name='tsn']",
          "data_key": "vehicle_tsn"
        },
        {
          "type": "date",
          "selector": "input[name='registration_date']",
          "data_key": "first_registration",
          "type_config": {
            "format": "MM.YYYY"
          }
        }
      ],
      "next_action": {
        "type": "click",
        "selector": "button:has-text('Continue')"
      }
    },
    {
      "name": "personal_info",
      "wait_for": "input[name='birthdate']",
      "timeout_ms": 15000,
      "fields": [
        {
          "type": "date",
          "selector": "input[name='birthdate']",
          "data_key": "birthdate",
          "type_config": {
            "format": "DD.MM.YYYY"
          }
        },
        {
          "type": "text",
          "selector": "input[name='zipcode']",
          "data_key": "postal_code"
        }
      ],
      "next_action": {
        "type": "click",
        "selector": "button:has-text('Get Quote')"
      }
    }
  ],
  "final_page": {
    "wait_for": ".quote-result",
    "timeout_ms": 60000,
    "screenshot_selector": ".quote-panel"
  }
}

data_row.json:

{
  "vehicle_hsn": "0603",
  "vehicle_tsn": "AKZ",
  "first_registration": "2020-03-15",
  "birthdate": "1985-06-20",
  "postal_code": "80331"
}

Form with Iframes

{
  "steps": [
    {
      "name": "embedded_form",
      "wait_for": "iframe#form-frame",
      "timeout_ms": 15000,
      "fields": [
        {
          "type": "text",
          "selector": "input[name='email']",
          "iframe_selector": "iframe#form-frame",
          "data_key": "email"
        },
        {
          "type": "dropdown",
          "selector": "select[name='plan']",
          "iframe_selector": "iframe#form-frame",
          "data_key": "selected_plan",
          "type_config": {
            "select_by": "text"
          }
        }
      ],
      "next_action": {
        "type": "click",
        "selector": "button:has-text('Submit')",
        "iframe_selector": "iframe#form-frame"
      }
    }
  ]
}

License

MIT