selenium-execution-engine 1.0.0

A flexible, JSON-driven execution engine for Selenium with BeautifulSoup-powered extraction.

Selenium Execution Engine

A flexible, JSON-driven execution engine for Selenium. This engine allows you to define navigation steps, interactions, and data extraction rules using a simple JSON configuration. It can be used as a standalone Command Line Interface (CLI) tool or imported as a Python library.

Installation

1. Clone this repository.
2. Install the package in editable mode (which also installs all dependencies):

   pip install -e .
   

3. Make sure you have the appropriate WebDriver installed (e.g., ChromeDriver for Google Chrome) and available in your system's PATH.

Usage as CLI

Once installed, the CLI tool selenium-engine will be available in your PATH. You can run it directly:

selenium-engine config.json

To save the extracted results to a file, use the -o or --output flag:

selenium-engine config.json -o results.json

To override the number of iterations when using loop mode, use the -i or --iterations flag:

selenium-engine config.json -i 5

Usage as a Library

You can integrate the engine into your Python scripts.

import json
from selenium_engine import execute_scraping

config_json = """
{
  "target_url": "https://example.com",
  "actions": [
    {"type": "wait", "selector": "h1", "timeout": 5}
  ],
  "extraction_schema": {
    "items": {
      "fields": {
        "title": {"selector": "h1", "type": "text"}
      }
    }
  }
}
"""

# You can pass the JSON string directly
results = execute_scraping(config_json)
print(json.dumps(results, indent=2))

# Or you can pass a parsed dictionary
config_dict = json.loads(config_json)
results_dict = execute_scraping(config_dict)
print(json.dumps(results_dict, indent=2))

JSON Configuration Schema

The engine uses a JSON schema to validate your configuration. Here is an overview of the options:

Basic Structure

{
  "target_url": "https://surtido.app/productos",
  "selenium_config": {
    "driver": "chrome",
    "headless": true
  },
  "actions": [
    ...
  ],
  "extraction_schema": {
    ...
  }
}

Selenium Configuration (selenium_config)

• driver: Browser to use (chrome, firefox, edge, safari). Defaults to chrome.
• headless: Run in headless mode (true/false). Defaults to true.
• arguments: Array of string arguments to pass to the driver.
• page_load_timeout: Timeout in seconds.

Execution Mode (execution_mode)

Allows running the engine in a loop instead of a single linear execution.

• type: Either linear or loop.
• iterations: Number of times to loop (requires target_url at the root).
• loop_urls: Array of URLs to loop through.

Note: If you use execution_mode, you can omit target_url at the root if you provide loop_urls.

Actions (actions)

An array of steps to execute sequentially on the page.

Wait Action

Waits for an element to be present in the DOM.

{"type": "wait", "selector": ".product-list", "timeout": 10}

Click Action

Clicks on an element.

{"type": "click", "selector": "#submit-btn"}

Scroll Action

Scrolls the page.

{"type": "scroll", "target": "bottom"}
// OR
{"type": "scroll", "amount": 500}

Input Action

Types text into an input field.

{"type": "input", "selector": "#search-box", "value": "laptops"}

Extraction Schema (extraction_schema)

Defines how to extract data from the page after actions are completed.

Extracting a Single Item

"extraction_schema": {
  "items": {
    "fields": {
      "title": {"selector": "h1.title", "type": "text"}
    }
  }
}

Extracting a List of Items

Specify a container selector to loop through multiple elements.

"extraction_schema": {
  "items": {
    "container": ".product-card",
    "fields": {
      "title": {"selector": "h2.title", "type": "text"},
      "price": {"selector": ".price-tag", "type": "text", "transform": "float"},
      "stock": {"selector": ".stock-status", "type": "attribute", "attr": "data-qty"}
    }
  }
}

Field Configurations:

• selector: CSS selector. Set to "self" (or leave empty "") to select the container element itself.
• type: text (inner text) or attribute (element attribute).
• attr: Required if type is attribute. Name of the attribute.
• transform: Optional. Cast the value to float, int, string, or bool.

BeautifulSoup Extraction

Under the hood, once actions are completed, the engine retrieves the page source and uses BeautifulSoup to parse and extract the configured fields. This yields high-performance queries and allows flexible scraping features like referencing the container element itself.

Modularity

To add new actions, simply create a new .py file in the selenium_engine/actions/ directory. It should inherit from BaseAction and implement the execute(self, driver, action_config) method. Then, register it in selenium_engine/actions/__init__.py and update the schema.json to allow the new action type.