azcrawlerpy

A framework for navigating and filling multi-step web forms programmatically. Supports Camoufox anti-detect browser (built on Firefox with C++ level fingerprint spoofing) and standard Chromium via Playwright. Uses JSON instruction files to define form navigation workflows, making it ideal for automated form submission, web scraping, and AI agent-driven web interactions.

Table of Contents

• Installation
• Quick Start
• Logging & Telemetry
• Core Concepts
• Instructions Schema
  • Top-Level Structure
  • Browser Configuration
  • URL Rewrites
  • Cookie Consent Handling
  • Step Definitions
  • Field Types
  • Optional Fields
  • Action Types
  • Final Page Configuration
  • Post Final Page Steps
  • Per-Step Page Extraction
  • Data Extraction Configuration
• Data Points (input_data)
• Element Discovery
• AI Agent Guidance
• Retry and Resilience
• Error Handling and Diagnostics
• Browser Profile Building (Profiler)
• Examples

Installation

Requires Python 3.11–3.13.

uv add azcrawlerpy

Or install from source:

uv pip install -e .

Quick Start

FormCrawler requires a logger. In production, pass an AzureLogger from azpaddypy so crawl telemetry (spans, correlation IDs, structured records) flows into Application Insights. For local scripts and notebooks, any stdlib-shaped logger works — anything that implements debug/info/warning/error/exception/critical.

import asyncio

from azcrawlerpy import CrawlerBrowserConfig, DebugMode, FormCrawler, HumanizeConfig
from azpaddypy.mgmt.logging import AzureLogger, bootstrap_azure_monitor


async def main():
    # 1. Wire telemetry once per process (idempotent; no-op without a
    #    connection string so it's safe in local runs).
    bootstrap_azure_monitor(service_name="crawler-demo")
    logger = AzureLogger(__name__)

    # 2. CrawlerBrowserConfig controls runtime browser settings
    #    (proxy, stealth, humanize).
    browser_config = CrawlerBrowserConfig(humanize=HumanizeConfig(enabled=True))

    # 3. FormCrawler now requires a logger -- pass AzureLogger for App Insights
    #    integration, or a stdlib `logging.getLogger(__name__)` for local work.
    crawler = FormCrawler(
        headless=True,
        browser_config=browser_config,
        logger=logger,
    )

    instructions = {
        "url": "https://example.com/form",
        "browser_config": {
            "browser_type": "camoufox",
            "viewport_width": 1920,
            "viewport_height": 1080
        },
        "steps": [
            {
                "name": "step_1",
                "wait_for": "input[name='email']",
                "timeout_ms": 15000,
                "fields": [
                    {
                        "type": "text",
                        "selector": "input[name='email']",
                        "data_key": "email"
                    }
                ],
                "next_action": {
                    "type": "click",
                    "selector": "button[type='submit']"
                }
            }
        ],
        "final_page": {
            "wait_for": ".success-message",
            "timeout_ms": 60000
        }
    }

    input_data = {
        "email": "user@example.com"
    }

    result = await crawler.crawl(
        url=instructions["url"],
        input_data=input_data,
        instructions=instructions,
        debug_mode=DebugMode.ALL,
        # Optional wall-clock cap on the entire crawl. Exceeding it raises
        # CrawlerTimeoutError with a partial result attached. Omit or set
        # to None to disable.
        global_timeout_ms=300_000,
    )

    # The crawler returns every artifact in memory -- the caller owns the
    # sink (disk, Azure Blob, stdout, ...). Nothing is written to disk on
    # your behalf.
    print(f"Final URL: {result.final_url}")
    print(f"Steps completed: {result.steps_completed}")
    print(f"Screenshots captured: {len(result.screenshots)}")
    print(f"HTML bytes: {len(result.html)}")
    print(f"Extracted data: {result.extracted_data}")

asyncio.run(main())

Logging & Telemetry

FormCrawler accepts any logger that matches the stdlib logging.Logger shape (structural LoggerLike protocol in crawling/utils.py): debug, info, warning, error, exception, critical.

Production: AzureLogger

Pairing the crawler with AzureLogger from azpaddypy gives you, with no code in the crawler itself:

• Application Insights export of all crawl log records via the OTel handler that azpaddypy attaches to the root logger; the crawler's module-level loggers propagate to root automatically.
• Trace context (operation_Id / operation_ParentId) is set by the Azure Monitor exporter from the active OTel span — no per-record duplication.
• Built-in OpenTelemetry spans on the crawler's hot paths so the App Insights Application Map and Performance views show structure inside the crawl rather than one opaque blob:
  • FormCrawler.crawl — root span; carries crawl.url, crawl.headless, crawl.debug_mode, crawl.strict, crawl.steps_planned, plus post-success crawl.steps_completed and crawl.profiler_visited_count. On timeout / error it carries outcome=timeout|error with Status.ERROR and the captured exception.
  • FormCrawler.attempt — one span per restart-loop iteration (attempt.index, browser.type); restart paths add attempt.outcome=restart and attempt.restart_step.
  • profiler.run — wraps the profiler phase (profiler.visit_count, profiler.in_memory, profiler.browser_type).
  • ElementDiscovery.discover — wraps page element discovery (discovery.url, discovery.browser_type, discovery.explore_iframes, plus post-run discovery.total_elements and discovery.iframes_count).
• Correlation IDs if the caller sets one via AzureLogger.set_correlation_id(...) — it propagates through the contextvars-backed OTel context into all crawler spans automatically.

from azpaddypy.mgmt.logging import AzureLogger, bootstrap_azure_monitor, trace_function

bootstrap_azure_monitor(service_name="crawler-api", service_version="0.5.1")
logger = AzureLogger(__name__)

# Optional: generate a correlation ID per request + emit a parent span
@trace_function(name="crawl_request")
async def handle_crawl(url: str, input_data: dict, instructions: dict):
    crawler = FormCrawler(headless=True, browser_config=None, logger=logger)
    return await crawler.crawl(
        url=url, input_data=input_data, instructions=instructions,
    )

The crawler does not import opentelemetry directly. Spans are produced through a soft-import shim: when OTel is installed (the typical case in a function-app that pulls in azpaddypy) spans flow to Application Insights; when not, the shim is a zero-cost no-op so standalone CLI runs and minimal test environments keep working unchanged.

Local / scripts: stdlib logger

If you don't need App Insights, a standard library logger is enough:

import logging

logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(name)s: %(message)s")
crawler = FormCrawler(
    headless=True,
    browser_config=None,
    logger=logging.getLogger("my_crawler"),
)

Any logger that satisfies the LoggerLike Protocol is accepted — there is no hard runtime dependency on azpaddypy.

Core Concepts

The framework operates on two primary inputs:

1. Instructions (instructions.json): Defines the form structure, selectors, navigation flow, and field types
2. Data Points (input_data): Contains the actual values to fill into form fields

The crawler processes each step sequentially:

1. Wait for the step's wait_for selector to become visible
2. Fill all fields defined in the step using values from input_data
3. Execute the next_action to navigate to the next step
4. Repeat until all steps are complete
5. Wait for and capture the final page

Instructions Schema

Top-Level Structure

{
  "url": "https://example.com/form",
  "browser_config": { ... },
  "cookie_consent": { ... },
  "steps": [ ... ],
  "final_page": { ... },
  "post_final_page_steps": [ ... ],
  "data_extraction": { ... }
}

Browser Configuration

{
  "browser_config": {
    "browser_type": "camoufox",
    "viewport_width": 1920,
    "viewport_height": 1080
  }
}

Both engines render the configured width exactly. The height differs by engine: chromium sets it as the page's inner height, while camoufox sizes the browser window, leaving the page shorter by the height of the browser chrome. Camoufox ignores a Playwright viewport, so this is the only lever it offers.

URL Rewrites

Redirects matching requests to a different host. The main use case is a proxy that cannot reach a third-party host the page depends on, but which has a reachable mirror.

{
  "browser_config": {
    "browser_type": "chromium",
    "viewport_width": 1920,
    "viewport_height": 1080,
    "url_rewrites": {
      "://www.google.com/recaptcha/": "://www.recaptcha.net/recaptcha/"
    }
  }
}

Each key is matched as a substring of the request URL and replaced with its value. Requests are routed through page.route() and redirected with route.continue_(url=...), so the page's own loader keeps working — including any onload callback it registered on the original script tag.

Notes:

• The replacement must keep the same protocol; cross-origin redirects are allowed.
• Rewrites apply to the whole page lifetime, so a mirrored script's own follow-up requests are rewritten too when they match.
• Prefer this over an init script that patches URLs in JavaScript. Camoufox runs injected scripts in an isolated world that cannot touch page globals, so a JS-based rewrite silently does nothing there; routing works identically on both engines and leaves no page-visible trace.

Example — reCAPTCHA behind a proxy whose zone blocks google.com. Google serves reCAPTCHA from recaptcha.net for exactly this situation, so rewriting the host lets the widget load and render normally. Note this only restores reachability; whether the challenge then passes still depends on the exit IP's reputation.

Cookie Consent Handling

The framework supports two modes for handling cookie consent banners:

Standard Mode (regular DOM elements):

{
  "cookie_consent": {
    "banner_selector": "dialog:has-text('cookies')",
    "accept_selector": "button:has-text('Accept')"
  }
}

Shadow DOM Mode (for Usercentrics, OneTrust, etc.):

{
  "cookie_consent": {
    "banner_selector": "#usercentrics-cmp-ui",
    "shadow_host_selector": "#usercentrics-cmp-ui",
    "accept_button_texts": ["Accept All", "Alle akzeptieren"]
  }
}

The JS fallback matches button text against: klar, akzept, accept, agree, ok, verstanden, zustimm. Set js_fallback_texts to override this list with site-specific patterns.

Step Definitions

Each step represents a form page or section:

{
  "name": "personal_info",
  "wait_for": "input[name='firstName']",
  "timeout_ms": 15000,
  "fields": [ ... ],
  "next_action": { ... }
}

Restart Crawl

Steps marked with "restart_crawl": true raise an internal CrawlRestartError when the wait_for selector times out. The crawler then closes the browser, recreates the context (reusing any profiler storage state so cookie consent does not re-run), and replays the entire crawl from the start URL. Each step tracks its own restart budget against max_restarts; once exhausted, the original timeout error is raised.

{
  "name": "results_page",
  "wait_for": "[data-cy='quote-result']",
  "timeout_ms": 15000,
  "restart_crawl": true,
  "max_restarts": 2,
  "fields": [],
  "next_action": { "type": "delay", "delay_ms": 0 }
}

restart_crawl takes priority over optional and strict. Use it for steps where the page may render broken or never finalize due to flaky SPA/backend behavior, where a fresh browser context is the most reliable recovery. Prefer retry_config on individual fields/actions for transient interaction failures; reserve restart_crawl for full-page load failures.

Optional Steps

Steps marked with "optional": true are skipped gracefully when the wait_for selector is not found within timeout_ms. The entire step (fields, data extraction, next_action) is skipped with an info log. This is useful for conditional workflow pages that only appear depending on prior input values or dynamic form behavior.

{
  "name": "additional_driver_details",
  "wait_for": "#additional-driver-form",
  "timeout_ms": 5000,
  "optional": true,
  "fields": [ ... ],
  "next_action": { "type": "click", "selector": "#next" }
}

The optional check takes precedence over the strict parameter. When a step is optional and the wait_for selector times out, the step is always skipped regardless of strict mode. Non-optional steps follow the existing behavior: raise CrawlerTimeoutError when strict=True, or log a warning and continue into the step when strict=False.

Tip: Use a shorter timeout_ms (e.g., 3000-5000ms) on optional steps to avoid waiting the full timeout when the step is absent.

Field Types

TEXT

For text inputs, email fields, phone numbers, and similar:

{
  "type": "text",
  "selector": "input[name='email']",
  "data_key": "email"
}

TEXTAREA

For multi-line text areas:

{
  "type": "textarea",
  "selector": "textarea[name='message']",
  "data_key": "message"
}

DROPDOWN / SELECT

For native <select> elements:

{
  "type": "dropdown",
  "selector": "select[name='country']",
  "data_key": "country",
  "type_config": {
    "select_by": "text"
  }
}

RADIO

For radio button groups:

{
  "type": "radio",
  "selector": "input[type='radio'][value='${value}']",
  "data_key": "payment_method"
}

Pattern A - Value-driven selector: Use ${value} placeholder in selector, data provides the value:

{
  "type": "radio",
  "selector": "input[type='radio'][value='${value}']",
  "data_key": "gender"
}
// data: { "gender": "male" }

Pattern B - Boolean flags: Use explicit selectors with boolean data values:

{
  "type": "radio",
  "selector": "[role='radio']:has-text('Yes')",
  "data_key": "accept_terms",
  "force_click": true
}
// data: { "accept_terms": true }  // clicks if truthy, skips if null

force_click is supported on radio, click_only, and click_select field types. When set, the click fallback chain tries force click before normal click.

CHECKBOX

For checkbox inputs:

{
  "type": "checkbox",
  "selector": "input[type='checkbox'][name='newsletter']",
  "data_key": "subscribe_newsletter"
}

Data value true checks the box, false unchecks it, and null skips the field entirely (no interaction).

DATE

For date inputs with format conversion:

{
  "type": "date",
  "selector": "input[name='birthdate']",
  "data_key": "birthdate",
  "type_config": {
    "format": "DD.MM.YYYY"
  }
}

Supported formats (mapped to strftime internally):

Data must be provided in ISO format (YYYY-MM-DD) in input_data. The type_config.format specifies the output format for typing into the field. Native <input type="date"> fields are auto-detected and use the value as-is, no type_config needed.

SLIDER

For range inputs:

{
  "type": "slider",
  "selector": "input[type='range'][name='coverage']",
  "data_key": "coverage_amount"
}

FILE

For file upload fields:

{
  "type": "file",
  "selector": "input[type='file']",
  "data_key": "document_path"
}

Data value should be the absolute file path.

COMBOBOX

For autocomplete/typeahead inputs:

{
  "type": "combobox",
  "selector": "input[aria-label='City']",
  "data_key": "city",
  "press_enter": true,
  "type_config": {
    "option_selector": ".autocomplete-option",
    "type_delay_ms": 50,
    "wait_after_type_ms": 500
  }
}

Breaking change in 1.1.6: press_enter moved from the combobox type_config to the field itself, where it now also works for text, date, iframe_field and click_select. Move the key up one level; leaving it inside type_config fails validation.

CLICK_SELECT

For custom dropdowns requiring click-then-select:

{
  "type": "click_select",
  "selector": ".custom-dropdown-trigger",
  "data_key": "option_value",
  "post_click_delay_ms": 300,
  "type_config": {
    "option_selector": ".dropdown-item:has-text('${value}')"
  }
}

CLICK_ONLY

For elements that only need clicking (no data input):

{
  "type": "click_only",
  "selector": "button.expand-section"
}

With conditional clicking based on data:

{
  "type": "click_only",
  "selector": "button:has-text('${value}')",
  "data_key": "selected_option"
}

IFRAME_FIELD

For fields inside iframes (alternative to iframe_selector):

{
  "type": "iframe_field",
  "selector": "input[name='card_number']",
  "iframe_selector": "iframe#payment-frame",
  "data_key": "card_number"
}

Common Field Parameters

Pressing Keys

press_keys sends keys to the page once the field's own interaction has finished, in the order given. Its main use is dismissing a widget that swallows the next click — a date picker left open by a force_click, or a custom select with no close control:

{
  "type": "click_select",
  "selector": "#vehicle-fuel",
  "data_key": "fuel",
  "press_keys": ["Escape"],
  "type_config": {
    "option_selector": "ul.dropdown-content li"
  }
}

The keys go to whatever the page currently focuses, as a real key press would, so the field's own element does not need to still exist. This is what separates it from press_enter, which presses on the field itself to commit a value; the two are independent and can be combined.

An overlay that outlives the last field of a step blocks the step's next_action. Attach the keys to that last field rather than adding a field for them.

Optional Fields

Fields marked with "optional": true are skipped gracefully when the element is not found on the page or when interaction fails. This is useful for elements that may or may not appear depending on dynamic page behavior, A/B tests, or conditional rendering that cannot be predicted by data alone.

{
  "type": "click_only",
  "selector": "#promotional-banner button.dismiss",
  "data_key": "dismiss_promo",
  "optional": true,
  "field_visible_timeout_ms": 2000
}

How optional differs from null values:

When optional is set and the element is not found or interaction fails, the handler logs an info message and continues to the next field. No error is raised regardless of the strict mode.

Tip: Set a low field_visible_timeout_ms (e.g., 1000-2000ms) on optional fields to avoid waiting the full default timeout when the element is absent.

Note: For skipping entire steps (all fields + next_action), use Optional Steps instead.

Action Types

CLICK

Click a button or link:

{
  "type": "click",
  "selector": "button[type='submit']"
}

With iframe support:

{
  "type": "click",
  "selector": "button:has-text('Next')",
  "iframe_selector": "iframe#form-frame"
}

WAIT

Wait for an element to appear:

{
  "type": "wait",
  "selector": ".loading-complete"
}

WAIT_HIDDEN

Wait for an element to disappear:

{
  "type": "wait_hidden",
  "selector": ".loading-spinner"
}

SCROLL

Scroll to an element:

{
  "type": "scroll",
  "selector": "#section-bottom"
}

DELAY

Wait for a fixed time:

{
  "type": "delay",
  "delay_ms": 2000
}

CONDITIONAL

Execute actions based on conditions:

{
  "type": "conditional",
  "condition": {
    "type": "selector_visible",
    "selector": ".error-message"
  },
  "actions": [
    {
      "type": "click",
      "selector": "button.dismiss-error"
    }
  ]
}

Condition types:

• selector_visible: True if selector is visible on the page
• selector_hidden: True if selector is NOT visible on the page
• data_equals: True if input_data[key] equals value
• data_exists: True if input_data[key] is truthy

Common Action Parameters

Final Page Configuration

{
  "final_page": {
    "wait_for": ".result-container, .confirmation",
    "timeout_ms": 60000,
    "post_wait_delay_ms": 2000,
    "screenshot_selector": ".result-panel"
  }
}

Only one of wait_for or wait_for_data_key can be set (not both).

Post Final Page Steps

post_final_page_steps is an optional list of regular step definitions that run on the result page after its wait_for/post_wait_delay_ms settle and before data_extraction, the end screenshot, and the HTML capture. Use it to interact with the results (load more entries, switch coverage tabs) that a normal step can never reach because the result page only exists after all steps and CAPTCHA handling.

Each entry supports the full step semantics (wait_for, optional, restart_crawl, extraction, retries). Use {"type": "delay", "delay_ms": 0} as next_action when a step only extracts. Step extractions additionally support a repeat-click loop for load-more buttons:

{
  "post_final_page_steps": [
    {
      "name": "load_all_results",
      "wait_for": "div[data-test^='tarifkachel/']",
      "timeout_ms": 60000,
      "fields": [],
      "post_field_extraction": [
        {
          "name": "tariff_count",
          "selector": "div[data-test^='tarifkachel/']",
          "click_before": "button:has-text('weitere Ergebnisse laden')",
          "repeat_click_until_gone": true,
          "max_repeat_clicks": 20,
          "wait_after_click_ms": 1500
        }
      ],
      "next_action": { "type": "delay", "delay_ms": 0 }
    }
  ]
}

The repeat-click loop is available on any step extraction, but is most useful here. The whole loop runs inside the step's timeout_ms budget.

Per-Step Page Extraction

instructions.data_extraction runs once, after every step, so it only ever captures the UI state the last step left behind. When a results page changes in response to clicks — a coverage tab, a filter, a sort order — each state needs its own list snapshot. page_extraction puts a full DataExtractionConfig on a step and runs it at the end of that step, after post_field_extraction and before next_action, so it sees the DOM the step's own clicks produced.

{
  "name": "screen_vk_sb500",
  "wait_for": "[data-test='deckung/VK']",
  "timeout_ms": 120000,
  "optional": true,
  "data_extraction": [
    { "name": "_switch_vk", "selector": "[data-test='deckung/VK']",
      "click_before": "[data-test='deckung/VK']", "wait_after_click_ms": 6000 },
    { "name": "_load_more", "selector": "div[data-test^='tarifkachel/']",
      "click_before": "button:has-text('weitere Ergebnisse laden')",
      "repeat_click_until_gone": true, "max_repeat_clicks": 20 }
  ],
  "page_extraction": {
    "fields": {
      "insurer_names": { "selector": "div[data-test^='tarifkachel/'] img[alt]", "attribute": "alt", "multiple": true }
    }
  },
  "fields": [],
  "next_action": { "type": "delay", "delay_ms": 2000 }
}

Results are namespaced under the step name, so several steps can extract the same field names without overwriting each other:

result.extracted_data["screen_vk_sb500"]["insurer_names"]  # [... 98 items ...]

Because every extraction shares one flat result dict and instructions.data_extraction is merged over it last, Instructions rejects configs where a page_extraction step name would collide with another snapshot step, a step extraction name, or a final-page field name.

Failures follow the same restart_crawl > optional > strict chain as the other extraction slots, so a snapshot step marked optional: true cannot fail the crawl. Extraction cost scales with fields × matched elements and counts against the step's timeout_ms, so budget list-heavy snapshots accordingly — an exhausted budget on an optional step silently drops the snapshot.

Data Extraction Configuration

Extract structured data from the final page using CSS selectors:

{
  "data_extraction": {
    "fields": {
      "tier_prices": {
        "selector": ".price-value",
        "attribute": null,
        "regex": "([0-9]+[.,][0-9]{2})",
        "multiple": true,
        "iframe_selector": "iframe#form-frame"
      },
      "selected_price": {
        "selector": "#total-amount",
        "attribute": "data-value",
        "regex": null,
        "multiple": false
      }
    }
  }
}

The data_extraction config also supports nested_fields to combine flat extracted arrays into structured outputs (paired_dict or object_list). See docs/README.md for details.

Extracted data is available in the crawl result:

result = await crawler.crawl(...)
print(result.extracted_data)
# {'tier_prices': ['32,28', '35,26', '50,34'], 'selected_price': '35,26'}

Data Points (input_data)

The input_data dictionary provides values for form fields. Keys must match data_key values in the instructions.

Structure

{
  "email": "user@example.com",
  "first_name": "John",
  "last_name": "Doe",
  "birthdate": "1985-06-15",
  "country": "Germany",
  "accept_terms": true,
  "newsletter": false,
  "premium_option": null
}

Value Types

Note: Setting a value to null skips the field without any DOM interaction. This is different from "optional": true on the field definition, which attempts the interaction but tolerates failure. See Optional Fields.

Radio Button Patterns

Pattern A - Mutually exclusive options with value selector:

{
  "gender": "male"
}

Selector uses ${value} placeholder: input[value='${value}']

Pattern B - Boolean flags for each option:

{
  "option_a": true,
  "option_b": null,
  "option_c": null
}

Only the option with true gets clicked.

Date Handling

Dates in input_data should use ISO format (YYYY-MM-DD):

{
  "birthdate": "1985-06-15",
  "start_date": "2024-01-01"
}

The framework converts to the format specified in the field definition.

Element Discovery

The ElementDiscovery class scans web pages to identify interactive elements, helping build instructions.json files.

from pathlib import Path
from azcrawlerpy import ElementDiscovery

async def discover_elements():
    discovery = ElementDiscovery(headless=False)

    report = await discovery.discover(
        url="https://example.com/form",
        output_dir=Path("./discovery_output"),
        cookie_consent={
            "banner_selector": "#cookie-banner",
            "accept_selector": "button.accept"
        },
        explore_iframes=True,
        screenshot=True,
    )

    print(f"Found {report.total_elements} elements")

    for text_input in report.text_inputs:
        print(f"Text input: {text_input.selector}")
        print(f"  Suggested type: {text_input.suggested_field_type}")

    for dropdown in report.selects:
        print(f"Dropdown: {dropdown.selector}")
        print(f"  Options: {dropdown.options}")

    for radio_group in report.radio_groups:
        print(f"Radio group: {radio_group.name}")
        for option in radio_group.options:
            print(f"  - {option.label}: {option.selector}")

Discovery Report Contents

• text_inputs: Text, email, phone, password fields
• textareas: Multi-line text areas
• selects: Native dropdown elements with options
• radio_groups: Grouped radio buttons
• checkboxes: Checkbox inputs
• buttons: Clickable buttons
• links: Anchor elements
• date_inputs: Date picker fields
• file_inputs: File upload fields
• sliders: Range inputs
• custom_components: Non-standard interactive elements
• iframes: Discovered iframes with their elements

AI Agent Guidance

This section provides instructions for AI agents tasked with creating instructions.json and input_data files.

Workflow for Creating Instructions

1. Discovery Phase: Use ElementDiscovery to scan each page/step of the form
2. Mapping Phase: Map discovered elements to field definitions
3. Flow Definition: Define step transitions and actions
4. Data Schema: Create the input_data structure

Step-by-Step Process

1. Analyze the Form Structure

• Identify how many pages/steps the form has
• Note the URL pattern changes (if any)
• Identify what element appears when each step loads

2. For Each Step, Define:

{
  "name": "<descriptive_step_name>",
  "wait_for": "<selector_that_confirms_step_loaded>",
  "timeout_ms": 15000,
  "fields": [...],
  "next_action": {...}
}

Naming conventions:

• Use snake_case for step names: personal_info, payment_details
• Use descriptive data_keys: first_name, email_address, accepts_terms

3. Selector Priority

When choosing selectors, prefer in order:

1. [data-testid='...'] or [data-cy='...'] - Most stable
2. [aria-label='...'] or [aria-labelledby='...'] - Accessible and stable
3. input[name='...'] - Form field names
4. :has-text('...') - Text content (use for buttons/labels)
5. CSS class selectors - Least stable, avoid if possible

4. Handle Dynamic Content

For AJAX-loaded content:

• Use wait action before interacting
• Add field_visible_timeout_ms to field definitions
• Use post_click_delay_ms for fields that trigger updates

5. Radio Button Strategy

Option A - When radio values are meaningful:

{
  "type": "radio",
  "selector": "input[type='radio'][value='${value}']",
  "data_key": "payment_type"
}
// data: { "payment_type": "credit_card" }

Option B - When you need individual control:

{
  "type": "radio",
  "selector": "[role='radio']:has-text('Credit Card')",
  "data_key": "payment_credit_card",
  "force_click": true
},
{
  "type": "radio",
  "selector": "[role='radio']:has-text('PayPal')",
  "data_key": "payment_paypal",
  "force_click": true
}
// data: { "payment_credit_card": true, "payment_paypal": null }

6. Iframe Handling

When elements are inside iframes:

{
  "type": "text",
  "selector": "input[name='card_number']",
  "iframe_selector": "iframe#payment-iframe",
  "data_key": "card_number"
}

Creating input_data

1. Analyze Required Fields

From the instructions, extract all unique data_key values:

data_keys = set()
for step in instructions["steps"]:
    for field in step["fields"]:
        if field.get("data_key"):
            data_keys.add(field["data_key"])

2. Determine Value Types

3. Handle Mutually Exclusive Options

For radio groups with boolean flags, only ONE should be true:

{
  "employment_fulltime": true,
  "employment_parttime": null,
  "employment_selfemployed": null,
  "employment_unemployed": null
}

4. Date Format

Always provide dates in ISO format in input_data:

{
  "birthdate": "1985-06-15",
  "policy_start": "2024-01-01"
}

The instructions specify the output format for the specific form.

Common Patterns

Multi-Step Wizard

{
  "steps": [
    {
      "name": "step_1_personal",
      "wait_for": "input[name='firstName']",
      "fields": [...],
      "next_action": { "type": "click", "selector": "button:has-text('Next')" }
    },
    {
      "name": "step_2_address",
      "wait_for": "input[name='street']",
      "fields": [...],
      "next_action": { "type": "click", "selector": "button:has-text('Next')" }
    }
  ]
}

Form with Loading States

{
  "next_action": {
    "type": "click",
    "selector": "button[type='submit']",
    "post_action_delay_ms": 1000
  }
}

Conditional Fields

{
  "type": "conditional",
  "condition": {
    "type": "data_equals",
    "key": "has_additional_driver",
    "value": true
  },
  "actions": [
    {
      "type": "click",
      "selector": "button:has-text('Add Driver')"
    }
  ]
}

Retry and Resilience

The framework provides built-in retry and fallback mechanisms for handling transient failures during web interactions.

Retry Configuration

Add retry_config to any field, action, step extraction, or data extraction to enable automatic retry with exponential backoff:

{
  "type": "text",
  "selector": "input[name='email']",
  "data_key": "email",
  "retry_config": {
    "max_attempts": 3,
    "base_delay_ms": 500,
    "backoff_multiplier": 1.5
  }
}

Retryable errors include Playwright TimeoutError and Error (element not found, intercepted clicks, etc.).

Click Fallback Chain

All click interactions (click actions, click_only, radio, click_select fields) use an escalating fallback strategy when a click fails:

1. Normal click (or force click if force_first=True)
2. Force click (or normal click if force_first=True) -- bypasses actionability checks
3. JS dispatchEvent -- fires a MouseEvent via element.dispatchEvent()
4. JS element.click() -- calls element.click() via document.querySelector()

This handles common real-world issues like overlay interception, sticky headers, and elements that pass Playwright's visibility checks but fail to receive pointer events.

Value Verification

text and textarea fields verify the actual input value after filling. If the value written to the field does not match the expected value, a FieldInteractionError is raised. This catches silent data corruption from autofill interference, input masks, or JavaScript reformatting.

Set skip_verification: true on the field to disable this check for fields where the site intentionally transforms the input (e.g., phone number formatting).

ComboBox Degraded Retry

When retry_config is set on a combobox field, retries use degraded parameters to handle slow autocomplete responses:

• Typing delay scales by 1.5^(attempt-1) (slower typing gives autocomplete more time)
• The field is always cleared before retyping on retry attempts

Error Handling and Diagnostics

The framework provides detailed error information when failures occur.

Exception Types

All CrawlerError subclasses can carry a partial_result attribute populated via with_partial_result(). This is a CrawlResult holding whatever state had been collected before the failure (screenshots, extracted_data, steps_completed, profiler_visited_urls, platform_logs, crawler_log, stdout_log, stderr_log, playwright_debug_log, error_diagnostics). Catch the exception and read e.partial_result to recover that data — see In-Memory Error Diagnostics below.

Timeout Budgets

The crawler enforces two independent wall-clock budgets. Both translate their exhaustion into CrawlerTimeoutError (or CrawlRestartError when restart is configured) and attach a partial result.

1. Per-step budget — step.timeout_ms

Hard upper bound on the total wall-clock duration of one step. Covers wait_for, data_extraction, field handlers (including retries), next_action (including retries and delays), post_field_extraction and page_extraction. The same field is also used as the timeout for the initial wait_for_selector call, so a step that uses its full budget waiting for the selector leaves no time for the rest of the body. On exhaustion, the priority chain restart_crawl > optional > strict decides the outcome:

• restart_crawl=True → CrawlRestartError → full browser restart (up to max_restarts).
• optional=True → step is skipped silently.
• strict=True → CrawlerTimeoutError.
• strict=False → warning logged, step returns.

2. Crawl-level budget — crawler.crawl(..., global_timeout_ms=...)

Optional wall-clock cap on the entire crawl (profiler, navigation, all steps and the final-page capture). None (the default) disables it. When the deadline fires, the crawl task is cancelled and CrawlerTimeoutError(step_name="<global_timeout>", selector="<global>", timeout_ms=...) is raised with a partial CrawlResult attached. The partial carries screenshots, extracted_data, steps_completed, profiler_visited_urls, platform_logs, crawler_log, stdout_log, stderr_log and playwright_debug_log collected up to the moment of cancellation. html is empty and final_url falls back to the start URL because the live page is being torn down by the cancellation and cannot be read.

from azcrawlerpy.crawling.exceptions import CrawlerTimeoutError

try:
    result = await crawler.crawl(
        url=url,
        input_data=input_data,
        instructions=instructions,
        debug_mode=DebugMode.ALL,
        global_timeout_ms=300_000,  # 5 minutes
    )
except CrawlerTimeoutError as e:
    if e.step_name == "<global_timeout>":
        # Global budget hit -- inspect partial_result for what was collected.
        partial = e.partial_result
        if partial is not None:
            print(f"Timed out after {partial.steps_completed} step(s)")
            print(f"Screenshots: {len(partial.screenshots)}")
    else:
        # Per-step budget or restart-budget exhaustion.
        raise

Debug Mode

Enable debug mode to capture screenshots at various stages:

from azcrawlerpy import DebugMode

result = await crawler.crawl(
    ...,
    debug_mode=DebugMode.ALL,  # Capture all screenshots
)

AI Diagnostics

When errors occur with debug mode enabled, the framework captures:

• Current page URL and title
• Available data-cy and data-testid selectors
• Visible buttons and input fields
• Similar selectors (fuzzy matching suggestions)
• Console errors and warnings
• Failed network requests
• XHR/fetch API responses (URL, status, method) for debugging SPA extraction failures
• HTML snippet of the form area
• Error screenshot (captured in-memory as bytes)

This information is included in the exception message and attached to the returned/partial CrawlResult as error_diagnostics (a dict[str, Any] equivalent to the content of the previous error_diagnostics.json file). The caller persists it wherever they like.

In-Memory Error Diagnostics

Error diagnostics are fully available in-memory. The error screenshot is captured as bytes and accessible in two ways:

1. On the exception's diagnostics: exception.diagnostics.screenshot_bytes
2. As the last entry in the partial result: exception.partial_result.screenshots[-1] (a Screenshot model with label, image bytes, and the page html at capture time)

try:
    result = await crawler.crawl(
        url=url,
        input_data=input_data,
        instructions=instructions,
        debug_mode=DebugMode.ALL,
    )
except CrawlerError as e:
    # Access the error screenshot bytes directly from diagnostics
    if e.diagnostics and e.diagnostics.screenshot_bytes:
        error_screenshot = e.diagnostics.screenshot_bytes

    # Or from the partial result (includes all screenshots taken during the crawl,
    # each a Screenshot model pairing label, image bytes and page HTML)
    if e.partial_result:
        all_screenshots = e.partial_result.screenshots  # includes error screenshot as last item
        extracted_so_far = e.partial_result.extracted_data
        steps_done = e.partial_result.steps_completed

Azure App Service Log Capture

When the crawler runs on an Azure App Service Plan target (Web App, Function App, or Web App / Function App in a container), debug_mode=DebugMode.ALL automatically captures the platform log tree so it can be attached to the run's output artifacts. This is where Playwright's own DEBUG=pw:api,pw:browser* stderr ends up in Azure, so it is often the only way to see low-level browser-launch failures in post-mortem.

When it activates (two gating conditions):

• debug_mode=DebugMode.ALL, AND
• $HOME is set and $HOME/LogFiles/ is a directory.

If either gating check fails, the crawl is otherwise untouched and a clear reason is logged (see "Log output" below).

Sink:

The full $HOME/LogFiles/ tree is read into a dict[str, bytes] and returned on CrawlResult.platform_logs. Keys are forward-slash relative paths (e.g. "Application/Functions/Worker/worker.log"); values are the raw file bytes. On graceful failure the same dict is attached to CrawlerError.partial_result.platform_logs, so harnesses that catch CrawlerError can still recover them. The crawler never writes to disk — the caller owns the sink.

What gets collected:

The entire $HOME/LogFiles/ tree, recursively. No filtering. Files include:

• Application/Functions/Worker/*.log (Python worker stdout/stderr on Functions)
• Application/Functions/Host/ (Functions host logs)
• <instance>_default_docker.log (Web App container stdout/stderr)
• kudu/, Nginx/, docker.log, etc.

On Linux App Service that path is /home/LogFiles/; on Windows App Service it is typically C:\home\LogFiles\ or D:\home\LogFiles\. The detection uses $HOME so both work without configuration.

Usage example (harness uploads to its own storage):

result = await crawler.crawl(
    url=url,
    input_data=input_data,
    instructions=instructions,
    debug_mode=DebugMode.ALL,  # activates platform log capture
)

# result.platform_logs is a dict[str, bytes]
for rel_path, content in result.platform_logs.items():
    blob_client.upload_blob(
        name=f"{run_id}/platform_logs/{rel_path}",
        data=content,
        overwrite=True,
    )

Crash-safe:

The collection is wrapped in try / finally around the entire crawl body, so it runs on:

• Successful return
• CrawlerTimeoutError from an exhausted restart budget
• Any CrawlerError wrapping a browser crash
• Profiler-phase failures (the profiler is inside the try)
• KeyboardInterrupt / task cancellation

Per-file errors (Windows log rotation, permission issues, races) are logged as warnings and counted; they never mask the crawl's original exception.

Log output:

Every invocation emits one start line plus one outcome line via the configured logger (flows to Application Insights through azpaddypy):

Deployment requirements per target:

Delivery:

The collected bytes land on CrawlResult.platform_logs on success. On graceful failure (any CrawlerError, including CrawlerTimeoutError from an exhausted restart budget or the global_timeout_ms wall-clock cap), they land on exception.partial_result.platform_logs.

Shared-worker noise:

/home/LogFiles/Application/Functions/Worker/*.log is shared across all concurrent function invocations on the same worker instance. The copied snapshot therefore includes any overlapping activity — there is no per-invocation filtering. This is an intentional "full copy always" trade for forensic completeness.

Browser Profile Building (Profiler)

The profiler visits random sites before the main crawl to accumulate cookies and storage, making the browser appear more natural. This is configured via the profiler field in instructions.json.

Profiler Modes

The profiler supports two storage modes:

Disk mode (storage_path set): Persists the browser profile to disk.

• Camoufox: saves as a Firefox user data directory ({storage_path}.camoufox/)
• Chromium: saves as a JSON file ({storage_path}.chromium.json)

In-memory mode (storage_path omitted or null): No permanent files written.

• Chromium: returns storage state as a dict, passed directly to browser.new_context(storage_state=dict)
• Camoufox: uses a temporary directory (cleaned up on error, passed as user_data_dir during crawl)

Profiler Configuration

{
  "profiler": {
    "sites": [
      {
        "url": "https://www.google.de",
        "cookie_consent": {
          "banner_selector": "[role='dialog'], .cookie-banner",
          "accept_selector": "button[id*='accept'], button[id*='agree']",
          "js_fallback_texts": ["akzept", "accept", "zustimm"]
        },
        "browse_delay_ms": 2000
      },
      {
        "url": "https://en.wikipedia.org",
        "browse_delay_ms": 3000
      }
    ],
    "visit_count": 3,
    "ignore_errors": true,
    "storage_path": null,
    "inter_site_delay_ms": 1000
  }
}

ProfilerConfig Parameters

ProfilerSiteConfig Parameters

Each site in the sites array supports per-site cookie consent handling:

ProfilerResult

The profiler returns a ProfilerResult with:

The crawler automatically passes the profiler result to the browser context creation, so no manual wiring is needed.

Examples

Insurance Quote Form

instructions.json:

{
  "url": "https://insurance.example.com/quote",
  "browser_config": {
    "viewport_width": 1920,
    "viewport_height": 1080
  },
  "cookie_consent": {
    "banner_selector": "#cookie-banner",
    "accept_selector": "button:has-text('Accept')"
  },
  "steps": [
    {
      "name": "vehicle_info",
      "wait_for": "input[name='hsn']",
      "timeout_ms": 15000,
      "fields": [
        {
          "type": "text",
          "selector": "input[name='hsn']",
          "data_key": "vehicle_hsn"
        },
        {
          "type": "text",
          "selector": "input[name='tsn']",
          "data_key": "vehicle_tsn"
        },
        {
          "type": "date",
          "selector": "input[name='registration_date']",
          "data_key": "first_registration",
          "type_config": {
            "format": "MM.YYYY"
          }
        }
      ],
      "next_action": {
        "type": "click",
        "selector": "button:has-text('Continue')"
      }
    },
    {
      "name": "personal_info",
      "wait_for": "input[name='birthdate']",
      "timeout_ms": 15000,
      "fields": [
        {
          "type": "date",
          "selector": "input[name='birthdate']",
          "data_key": "birthdate",
          "type_config": {
            "format": "DD.MM.YYYY"
          }
        },
        {
          "type": "text",
          "selector": "input[name='zipcode']",
          "data_key": "postal_code"
        }
      ],
      "next_action": {
        "type": "click",
        "selector": "button:has-text('Get Quote')"
      }
    }
  ],
  "final_page": {
    "wait_for": ".quote-result",
    "timeout_ms": 60000,
    "screenshot_selector": ".quote-panel"
  }
}

data_row.json:

{
  "vehicle_hsn": "0603",
  "vehicle_tsn": "AKZ",
  "first_registration": "2020-03-15",
  "birthdate": "1985-06-20",
  "postal_code": "80331"
}

Form with Iframes

{
  "steps": [
    {
      "name": "embedded_form",
      "wait_for": "iframe#form-frame",
      "timeout_ms": 15000,
      "fields": [
        {
          "type": "text",
          "selector": "input[name='email']",
          "iframe_selector": "iframe#form-frame",
          "data_key": "email"
        },
        {
          "type": "dropdown",
          "selector": "select[name='plan']",
          "iframe_selector": "iframe#form-frame",
          "data_key": "selected_plan",
          "type_config": {
            "select_by": "text"
          }
        }
      ],
      "next_action": {
        "type": "click",
        "selector": "button:has-text('Submit')",
        "iframe_selector": "iframe#form-frame"
      }
    }
  ]
}

License

MIT